Re: [squid-users] Squid for Windows Repeatedly Crashing
It is confirmed that the problems experienced under Windows Server 2016 do not occur under Linux. All traffic in the VIP is going to the Linux Squid (using 4K file descriptors), but I've requisitioned another for redundancy. Thanks to everyone for their advice! -Original Message- From: Van Order, Drew (US - Hermitage) Sent: Wednesday, February 27, 2019 4:47 PM To: 'elie...@ngtech.co.il' Cc: squid-users@lists.squid-cache.org Subject: RE: [EXT] RE: [squid-users] Squid for Windows Repeatedly Crashing Wow. This is very generous Eliezer. I am humbled by your generosity! Before I try your proxy for Windows, I've decided it's time to eliminate the OS as a variable. I quickly tacked up Squid running on a sandbox RHEL, and submitted a ticket to have it added to the F5 VIP. If the issue vanishes, bye bye Windows, and some folks are going to have to teach themselves Linux quickly :-) Stay tuned... -Original Message- From: elie...@ngtech.co.il Sent: Wednesday, February 27, 2019 4:28 PM To: Van Order, Drew (US - Hermitage) Cc: squid-users@lists.squid-cache.org Subject: [EXT] RE: [squid-users] Squid for Windows Repeatedly Crashing Forgot to mention that this simple proxy: http://secure-web.cisco.com/11Ju3FBMy81J840cSoCXGXdubwW80knGlevOiEwmFux7MQTjMwodNApLbJYerezA5dSOY7bJJChXO2aVi80fseEIMDaEj12mh4Ig4yNYzxviiWXlGyk_IUiyoo4tIFc-tnaWbefXsQ49afvPY1yTX-B3H7BK3voG5Dfw2WmyZJ1N8lEwnCwquwbLcdnYnYw8zp5qIMe-Rq4fl-399jML9snz7QIUgE4jK46s-OgXDOPlHDlMfqgp66UhJL7cw-AkWDYfQV_uIGnUEWpvvmS1qEfhOLC89KnTzH3WCIRGR-Zh3LgWUo5yr4vW_nmyO0deNOGfNP4t2D-JjK85rZEahU_JLuFgzQLJC95M-uzoATapIbxqkCdSJ9ibyDaLhZWNdCyV6H64olDKlBBonUSnOTeu2C-RaoCUoOPhOL4I2zX_vyKrB5zGX2qWpo4TVQxRWd1z-WVIOJb0AS9J9m86mpQ-Op-Govz_L9XwqaMOHngH2bb1UB9JWMHbW8fcZny9nFZR2VeG6N9X87shN9Ek1dQ/http%3A%2F%2Fgogs.ngtech.co.il%2Felicro%2Fgolang-http-proxy Is a simple forward proxy I wrote. A binary packaged for any OS that GoLang supports including Windows 2k16 is there: http://secure-web.cisco.com/1ySd839vtqkoCLOWAs5SXi2Fzc8RNRQd0Vk53qQWH0XChRYXvX7qhbT1_QhocdaqgeVsDhkDZscU9PQNRd-4mhsOlnZHRKyqrSW4zlw4x-BaRogwP4jInaTbDEhCTTt4wUSiKS9VaahRIdiCoI81Sy46jhpq4i14fB5KSHtSywhD1SzmqDQfokkEr0vUFP0x2RdYtkY9axCTbSljyVgdDMk0QQfIPQ8nmFs5FULbfd4Xrts9UPlcmoNleo0YXHCWlrizaT2JCuRqW23kq9baAB8VOk06MtwBkmdFLY7AMT49HqRhTwgHHPTuL2jyL7IA4FYG-RAlo3JU0GyLgZWeX2ruEk66ZhadtuwLNkyucJwAoyoQMIyhM5ps0lC2DdHWEamYLT8M7NoW4TZju03jD76ixc8xMPzbnN0IBFznWcnZPYIooUHeAxyAaYEBp47vR-pAMV4kur_zcuU_Exv7B2jY9lfXLPAnPUW-c20el5ZGTosbPeV6bF9D7XofMf6FvnbsqTkf_VgUDynE-tnLDsw/http%3A%2F%2Fgogs.ngtech.co.il%2FNgTech-LTD%2Fgolang-http-proxy%2Fsrc%2Fmaster%2Fgolang-http-proxy.tar.xz You will need some software to make it a service but these are easy to find. If you need a recommendation for one I will try to find. All The Bests, Eliezer Eliezer Croitoru Linux System Administrator Mobile: +972-5-28704261 Email: elie...@ngtech.co.il -Original Message- From: squid-users On Behalf Of elie...@ngtech.co.il Sent: Wednesday, February 27, 2019 22:44 To: 'Van Order, Drew (US - Hermitage)' Cc: squid-users@lists.squid-cache.org Subject: Re: [squid-users] Squid for Windows Repeatedly Crashing So just to be on the clear. You need a basic forward proxy that does CONNECT requests for any LAN client to safe ports? Do you have any other requirements then being Windows 2k16 compatbile and the mentioned ACL's? If you Insist on a Windows proxy there are couple I collected... http://secure-web.cisco.com/17a99t4PIGRmHeQmTLY5KRhYDoGTUPwyYGXT0fAV0DVh9MsSteT4Pi-0sb_DM-mY6nLb-NLB1ftORaQ0bC7KstwyrnAci2lsLoKWzNgOiKHwBGQSVL7MMHSGJ1zHRTGIcyEuDlGdldzgihQb6_79nG9yppR2yvpbWX2uvTAEr-qZB46PVCd_d3YtLah9RzDxyJymPdDeyaAw66X6Agmqs512eb5uI1oCN3auT9qbjI11NDr8edlo3R04C-tHHqBAka4hQXEs9LavQUNcBcHFWhME6PEBNCdLVBeitC3d9ZF2rtYKIP5iFYZs1w72GL_-Xh0zawz7uiX9GcwN60Tx0m8MJQQoEMPp4v3cfSiM-pFHI7YAJRvcCfFENvZcgNXFQrzX4ZVaLIxPkV5q2fN8uGObAZKRTWkqAXwo2LE40s9waGLpTDmiXy76gE6sGFerW5m1mImQElzPWjajbWJfqSi8aD7W1TC0w42AGOqQJ60VnXhZw4CuupXzmylyd8E6D_GPtTtAdp5VPrXTEQCT20w/http%3A%2F%2Fwww1.ngtech.co.il%2Fwpe%2F2016%2F05%2F02%2Fproxy-per-internet-user-is-it-realis tic/ I know that RedWood might be good for your needs to compare...: http://secure-web.cisco.com/1ETToAy7lpIzQDnVWVx4VN460yeA4V0c0irlIMN9P5wE50B-0kLELWsz8usawuOgy7IZPOJ6iV2FTApPTEaJH1nHFLR-pnUkuG7C5E6f_fZdUofEV5UX__yh2g0MRKI7XB3x1uEdiMhlhtodTsSeJYzqWK_5Zij6_rNjYlWlY8573ATbmhIZNgkfwcaoRJl4FwO50zEAueB_tlGyikPc8FfJGgKZbcRYa7frdZcwsn9JKeQh_GISsi-_BpAETFQf6ZeZ5SGXQ5TB4z9GoPlncAf0vxingBSktcPgqF-jRLgxwHQ9nv6a-Ses-94UmCser4hIzsd6pTOHLVYY8u0OWUjlB5rRqROlH_IMkslBcFtEXaRYl1Fy2LEgL9RWaSuNFG-wRWmFw4BRtweHIpRnypuF24a2vKjwnN929-EHwHv-t-rCk8FSTQ5OfkHkP78sN0ErpcZv9GFBXOPA_7y0MTp3evd2SWbD4YBfXSZ5a3BCL4iM2Jx4KfG9SBK5KIz2TkdGzjQXIWujLFeWtOd5KIw/http%3A%2F%2Fngtech.co.il%2Fstatic%2Fredwood%2Fredwood-0.2.0.tar.xz https://secure-web.cisco.com/14Y9nKqTcPsVkJBCkyIkxFkE_XO9jqzDvqIf1yzNdAIfhtJDiPPfe1HdfUmXxsFWMojRBh
Re: [squid-users] Squid for Windows Repeatedly Crashing
Wow. This is very generous Eliezer. I am humbled by your generosity! Before I try your proxy for Windows, I've decided it's time to eliminate the OS as a variable. I quickly tacked up Squid running on a sandbox RHEL, and submitted a ticket to have it added to the F5 VIP. If the issue vanishes, bye bye Windows, and some folks are going to have to teach themselves Linux quickly :-) Stay tuned... -Original Message- From: elie...@ngtech.co.il Sent: Wednesday, February 27, 2019 4:28 PM To: Van Order, Drew (US - Hermitage) Cc: squid-users@lists.squid-cache.org Subject: [EXT] RE: [squid-users] Squid for Windows Repeatedly Crashing Forgot to mention that this simple proxy: http://secure-web.cisco.com/11Ju3FBMy81J840cSoCXGXdubwW80knGlevOiEwmFux7MQTjMwodNApLbJYerezA5dSOY7bJJChXO2aVi80fseEIMDaEj12mh4Ig4yNYzxviiWXlGyk_IUiyoo4tIFc-tnaWbefXsQ49afvPY1yTX-B3H7BK3voG5Dfw2WmyZJ1N8lEwnCwquwbLcdnYnYw8zp5qIMe-Rq4fl-399jML9snz7QIUgE4jK46s-OgXDOPlHDlMfqgp66UhJL7cw-AkWDYfQV_uIGnUEWpvvmS1qEfhOLC89KnTzH3WCIRGR-Zh3LgWUo5yr4vW_nmyO0deNOGfNP4t2D-JjK85rZEahU_JLuFgzQLJC95M-uzoATapIbxqkCdSJ9ibyDaLhZWNdCyV6H64olDKlBBonUSnOTeu2C-RaoCUoOPhOL4I2zX_vyKrB5zGX2qWpo4TVQxRWd1z-WVIOJb0AS9J9m86mpQ-Op-Govz_L9XwqaMOHngH2bb1UB9JWMHbW8fcZny9nFZR2VeG6N9X87shN9Ek1dQ/http%3A%2F%2Fgogs.ngtech.co.il%2Felicro%2Fgolang-http-proxy Is a simple forward proxy I wrote. A binary packaged for any OS that GoLang supports including Windows 2k16 is there: http://secure-web.cisco.com/1ySd839vtqkoCLOWAs5SXi2Fzc8RNRQd0Vk53qQWH0XChRYXvX7qhbT1_QhocdaqgeVsDhkDZscU9PQNRd-4mhsOlnZHRKyqrSW4zlw4x-BaRogwP4jInaTbDEhCTTt4wUSiKS9VaahRIdiCoI81Sy46jhpq4i14fB5KSHtSywhD1SzmqDQfokkEr0vUFP0x2RdYtkY9axCTbSljyVgdDMk0QQfIPQ8nmFs5FULbfd4Xrts9UPlcmoNleo0YXHCWlrizaT2JCuRqW23kq9baAB8VOk06MtwBkmdFLY7AMT49HqRhTwgHHPTuL2jyL7IA4FYG-RAlo3JU0GyLgZWeX2ruEk66ZhadtuwLNkyucJwAoyoQMIyhM5ps0lC2DdHWEamYLT8M7NoW4TZju03jD76ixc8xMPzbnN0IBFznWcnZPYIooUHeAxyAaYEBp47vR-pAMV4kur_zcuU_Exv7B2jY9lfXLPAnPUW-c20el5ZGTosbPeV6bF9D7XofMf6FvnbsqTkf_VgUDynE-tnLDsw/http%3A%2F%2Fgogs.ngtech.co.il%2FNgTech-LTD%2Fgolang-http-proxy%2Fsrc%2Fmaster%2Fgolang-http-proxy.tar.xz You will need some software to make it a service but these are easy to find. If you need a recommendation for one I will try to find. All The Bests, Eliezer Eliezer Croitoru Linux System Administrator Mobile: +972-5-28704261 Email: elie...@ngtech.co.il -Original Message- From: squid-users On Behalf Of elie...@ngtech.co.il Sent: Wednesday, February 27, 2019 22:44 To: 'Van Order, Drew (US - Hermitage)' Cc: squid-users@lists.squid-cache.org Subject: Re: [squid-users] Squid for Windows Repeatedly Crashing So just to be on the clear. You need a basic forward proxy that does CONNECT requests for any LAN client to safe ports? Do you have any other requirements then being Windows 2k16 compatbile and the mentioned ACL's? If you Insist on a Windows proxy there are couple I collected... http://secure-web.cisco.com/17a99t4PIGRmHeQmTLY5KRhYDoGTUPwyYGXT0fAV0DVh9MsSteT4Pi-0sb_DM-mY6nLb-NLB1ftORaQ0bC7KstwyrnAci2lsLoKWzNgOiKHwBGQSVL7MMHSGJ1zHRTGIcyEuDlGdldzgihQb6_79nG9yppR2yvpbWX2uvTAEr-qZB46PVCd_d3YtLah9RzDxyJymPdDeyaAw66X6Agmqs512eb5uI1oCN3auT9qbjI11NDr8edlo3R04C-tHHqBAka4hQXEs9LavQUNcBcHFWhME6PEBNCdLVBeitC3d9ZF2rtYKIP5iFYZs1w72GL_-Xh0zawz7uiX9GcwN60Tx0m8MJQQoEMPp4v3cfSiM-pFHI7YAJRvcCfFENvZcgNXFQrzX4ZVaLIxPkV5q2fN8uGObAZKRTWkqAXwo2LE40s9waGLpTDmiXy76gE6sGFerW5m1mImQElzPWjajbWJfqSi8aD7W1TC0w42AGOqQJ60VnXhZw4CuupXzmylyd8E6D_GPtTtAdp5VPrXTEQCT20w/http%3A%2F%2Fwww1.ngtech.co.il%2Fwpe%2F2016%2F05%2F02%2Fproxy-per-internet-user-is-it-realis tic/ I know that RedWood might be good for your needs to compare...: http://secure-web.cisco.com/1ETToAy7lpIzQDnVWVx4VN460yeA4V0c0irlIMN9P5wE50B-0kLELWsz8usawuOgy7IZPOJ6iV2FTApPTEaJH1nHFLR-pnUkuG7C5E6f_fZdUofEV5UX__yh2g0MRKI7XB3x1uEdiMhlhtodTsSeJYzqWK_5Zij6_rNjYlWlY8573ATbmhIZNgkfwcaoRJl4FwO50zEAueB_tlGyikPc8FfJGgKZbcRYa7frdZcwsn9JKeQh_GISsi-_BpAETFQf6ZeZ5SGXQ5TB4z9GoPlncAf0vxingBSktcPgqF-jRLgxwHQ9nv6a-Ses-94UmCser4hIzsd6pTOHLVYY8u0OWUjlB5rRqROlH_IMkslBcFtEXaRYl1Fy2LEgL9RWaSuNFG-wRWmFw4BRtweHIpRnypuF24a2vKjwnN929-EHwHv-t-rCk8FSTQ5OfkHkP78sN0ErpcZv9GFBXOPA_7y0MTp3evd2SWbD4YBfXSZ5a3BCL4iM2Jx4KfG9SBK5KIz2TkdGzjQXIWujLFeWtOd5KIw/http%3A%2F%2Fngtech.co.il%2Fstatic%2Fredwood%2Fredwood-0.2.0.tar.xz https://secure-web.cisco.com/14Y9nKqTcPsVkJBCkyIkxFkE_XO9jqzDvqIf1yzNdAIfhtJDiPPfe1HdfUmXxsFWMojRBhFGJfxGogPbIh16U9bdtA5l-XZkZAXM-KcwTAto3X-WRpC6ogpKA9wuNMuWwgKlRAPdgz1hvOAho8mcmXlY3Zct0t1WX6qy5RJ4Yjm_Nwfk5gBzAn_HXuCRAVkwmXYtzSqdwxpxhZ7bG-nsO4bHr0CwqU2WmvzUSsTQEHERFcVTMX0B5PrzmySJtmZlzv33zvGFFwrW8SSSTSqVrxZtiiHJly8tc9e42bpY2v7tmkhkacmX50Vn5w7FcsqjKVew6Qey7TAPp2K6_7n-Dr15pHPvpunEiHUVC-ewcE5OXL3uf1bruR-XsF2xLNe1UN2TxTQdLNO5od_wmbO1KzFAq70T8o5gS7Tf5xPbUqX_GHNbHWi53302aJvDUpIH6Dlr7llzwKR1J51WdEL2XCiu61T2w-Sn7rmzQnoS8kkwzadmdrJAxXyaOcJTO82wmRT8W4fcPPWVIRzwdyAXjqg/https%3A%2F%2Fgithub.com%2Fandybalholm%2Fredwood I wrote a tiny proxy the other day which should also work fine for y
Re: [squid-users] Squid for Windows Repeatedly Crashing
Forgot to mention that this simple proxy: http://gogs.ngtech.co.il/elicro/golang-http-proxy Is a simple forward proxy I wrote. A binary packaged for any OS that GoLang supports including Windows 2k16 is there: http://gogs.ngtech.co.il/NgTech-LTD/golang-http-proxy/src/master/golang-http-proxy.tar.xz You will need some software to make it a service but these are easy to find. If you need a recommendation for one I will try to find. All The Bests, Eliezer Eliezer Croitoru Linux System Administrator Mobile: +972-5-28704261 Email: elie...@ngtech.co.il -Original Message- From: squid-users On Behalf Of elie...@ngtech.co.il Sent: Wednesday, February 27, 2019 22:44 To: 'Van Order, Drew (US - Hermitage)' Cc: squid-users@lists.squid-cache.org Subject: Re: [squid-users] Squid for Windows Repeatedly Crashing So just to be on the clear. You need a basic forward proxy that does CONNECT requests for any LAN client to safe ports? Do you have any other requirements then being Windows 2k16 compatbile and the mentioned ACL's? If you Insist on a Windows proxy there are couple I collected... http://www1.ngtech.co.il/wpe/2016/05/02/proxy-per-internet-user-is-it-realis tic/ I know that RedWood might be good for your needs to compare...: http://ngtech.co.il/static/redwood/redwood-0.2.0.tar.xz https://github.com/andybalholm/redwood I wrote a tiny proxy the other day which should also work fine for you as long as you have a working and properly configured firewall on the Server. Let me know if something fit your needs. If so you can try and test and maybe find the right culprit(ie windows or linux). Eliezer Eliezer Croitoru Linux System Administrator Mobile: +972-5-28704261 Email: elie...@ngtech.co.il -Original Message- From: Van Order, Drew (US - Hermitage) Sent: Wednesday, February 27, 2019 17:55 To: elie...@ngtech.co.il Cc: squid-users@lists.squid-cache.org Subject: RE: [squid-users] Squid for Windows Repeatedly Crashing Business objective is to enable MSFT Azure MMA's (Microsoft Monitoring Agents) blocked from the internet to send agent data to Azure Log Analytics Simple proxy No SSL bump Squid config is attached I tried disabling caching with Squid, found it crashed more frequently Squid was chosen this is intended to be a stopgap solution, and it's free. It's a battle to win over security in order to have tcp/443 opened everywhere. I'm not sure Squid is the problem, I have an identically configured Squid that bypasses the F5 working beautifully, but it's only 50 clients (MMA's) connecting, Each client takes roughly 5 connections. The clients are still going through a firewall(s). Our network folks say that neither the FW or F5 leading up to Squid report congestion. It's possible that Squid for Windows + F5 VIP are not intended to work together, but it makes sense to just have one proxy IP address. I'm getting ready to Skype with our F5 guy to compare what I'm seeing with what he's seeing. Also trying to get how many clients are going through the F5 to compare to my 'good' Squid -Original Message- From: elie...@ngtech.co.il Sent: Wednesday, February 27, 2019 9:20 AM To: Van Order, Drew (US - Hermitage) Cc: squid-users@lists.squid-cache.org Subject: [EXT] RE: [squid-users] Squid for Windows Repeatedly Crashing The setup itself is not clear to me. Is it a simple proxy? With SSL bump? Can you share or send me the squid configuration? There might be another solution for your use case that you have yet to try. Also if the purpose is not caching, why do you try to use squid? There are lots of other proxies for windows out there? (just wondering what and why have you choose Squid) Thanks, Eliezer Eliezer Croitoru Linux System Administrator Mobile: +972-5-28704261 Email: elie...@ngtech.co.il -Original Message- From: Van Order, Drew (US - Hermitage) Sent: Wednesday, February 27, 2019 05:51 To: Eliezer Croitoru ; Rafael Akchurin Cc: 'Amos Jeffries' ; squid-users@lists.squid-cache.org Subject: RE: [squid-users] Squid for Windows Repeatedly Crashing Hello folks, and thanks for keeping interest. Today I spent a bit of time learning squidclient, and have determined that the server is not in any way resource constrained. I've attached the output from mgr:info, mgr:client_list, and mgr:filedescriptors in between crashes. Was wondering if someone could explain Tout, which I presume is timeout. Of interest are the ones set to 86400, which I presume is one day. That seems like a big problem--but where is it coming from? I'm using the Cygwin Squid config defaults. There seems to be a lot of Reading next request going on before Squid recycles. I wonder if the F5 VIP is dealing with congestion through the firewall, which, in turn, is causing congestion on the pool output side, the 10.26.25.220 address. Our F5 guys have gone silent on me, I have been asking questions
Re: [squid-users] Squid for Windows Repeatedly Crashing
So just to be on the clear. You need a basic forward proxy that does CONNECT requests for any LAN client to safe ports? Do you have any other requirements then being Windows 2k16 compatbile and the mentioned ACL's? If you Insist on a Windows proxy there are couple I collected... http://www1.ngtech.co.il/wpe/2016/05/02/proxy-per-internet-user-is-it-realis tic/ I know that RedWood might be good for your needs to compare...: http://ngtech.co.il/static/redwood/redwood-0.2.0.tar.xz https://github.com/andybalholm/redwood I wrote a tiny proxy the other day which should also work fine for you as long as you have a working and properly configured firewall on the Server. Let me know if something fit your needs. If so you can try and test and maybe find the right culprit(ie windows or linux). Eliezer Eliezer Croitoru Linux System Administrator Mobile: +972-5-28704261 Email: elie...@ngtech.co.il -Original Message- From: Van Order, Drew (US - Hermitage) Sent: Wednesday, February 27, 2019 17:55 To: elie...@ngtech.co.il Cc: squid-users@lists.squid-cache.org Subject: RE: [squid-users] Squid for Windows Repeatedly Crashing Business objective is to enable MSFT Azure MMA's (Microsoft Monitoring Agents) blocked from the internet to send agent data to Azure Log Analytics Simple proxy No SSL bump Squid config is attached I tried disabling caching with Squid, found it crashed more frequently Squid was chosen this is intended to be a stopgap solution, and it's free. It's a battle to win over security in order to have tcp/443 opened everywhere. I'm not sure Squid is the problem, I have an identically configured Squid that bypasses the F5 working beautifully, but it's only 50 clients (MMA's) connecting, Each client takes roughly 5 connections. The clients are still going through a firewall(s). Our network folks say that neither the FW or F5 leading up to Squid report congestion. It's possible that Squid for Windows + F5 VIP are not intended to work together, but it makes sense to just have one proxy IP address. I'm getting ready to Skype with our F5 guy to compare what I'm seeing with what he's seeing. Also trying to get how many clients are going through the F5 to compare to my 'good' Squid -Original Message- From: elie...@ngtech.co.il Sent: Wednesday, February 27, 2019 9:20 AM To: Van Order, Drew (US - Hermitage) Cc: squid-users@lists.squid-cache.org Subject: [EXT] RE: [squid-users] Squid for Windows Repeatedly Crashing The setup itself is not clear to me. Is it a simple proxy? With SSL bump? Can you share or send me the squid configuration? There might be another solution for your use case that you have yet to try. Also if the purpose is not caching, why do you try to use squid? There are lots of other proxies for windows out there? (just wondering what and why have you choose Squid) Thanks, Eliezer Eliezer Croitoru Linux System Administrator Mobile: +972-5-28704261 Email: elie...@ngtech.co.il -Original Message- From: Van Order, Drew (US - Hermitage) Sent: Wednesday, February 27, 2019 05:51 To: Eliezer Croitoru ; Rafael Akchurin Cc: 'Amos Jeffries' ; squid-users@lists.squid-cache.org Subject: RE: [squid-users] Squid for Windows Repeatedly Crashing Hello folks, and thanks for keeping interest. Today I spent a bit of time learning squidclient, and have determined that the server is not in any way resource constrained. I've attached the output from mgr:info, mgr:client_list, and mgr:filedescriptors in between crashes. Was wondering if someone could explain Tout, which I presume is timeout. Of interest are the ones set to 86400, which I presume is one day. That seems like a big problem--but where is it coming from? I'm using the Cygwin Squid config defaults. There seems to be a lot of Reading next request going on before Squid recycles. I wonder if the F5 VIP is dealing with congestion through the firewall, which, in turn, is causing congestion on the pool output side, the 10.26.25.220 address. Our F5 guys have gone silent on me, I have been asking questions, in particular why all the F5 traffic is coming over just one IP address in the pool. In case folks wonder what the IP's are in the file descriptor output 1310 Socket 8986044*2806 40.71.12.224:443 593a6510-ebfc-4d6b-a8f0-a0411dfee098.ods.opinsights.azure.com:443 (this is Squid forwarding Windows event/perf data from an agent to Azure Log Analytics) 1311 Socket 8993015*9208 10.26.25.220:61088Reading next request (10.26.25.220) is the pool IP address of the F5 in use) 1312 Socket 8992690*8826 10.26.25.220:61436Reading next request 1313 Socket 8999169*2884 104.208.163.218:443 eus2-jobruntimedata-prod-su1.azure-automation.net:443 (Squid to Azure) 1314 Socket 8998787*2508 104.208.163.218:443 eus2-jobruntimedata-prod-su1.azure-automation.net:443 1315 Socket 118
Re: [squid-users] Squid for Windows Repeatedly Crashing
I would try deploying Squid on Linux machine running within Hyper-V just to be sure the Squid part itself works fine. Then only specifics of it running on Cygwin will remain to be uncovered. Should be very easy to setup. Couple of hours at most (you have already dedicated much more time to this). For example here is how we do it https://github.com/diladele/websafety-virtual-appliance/blob/master/scripts.ubuntu18/03_squid.sh It is even easier if you do not need to sslbump. Just apt-get update && apt-get install -y squid And voila! -Original Message- From: squid-users On Behalf Of Van Order, Drew (US - Hermitage) Sent: Wednesday, 27 February 2019 16:55 To: elie...@ngtech.co.il Cc: squid-users@lists.squid-cache.org Subject: Re: [squid-users] Squid for Windows Repeatedly Crashing Business objective is to enable MSFT Azure MMA's (Microsoft Monitoring Agents) blocked from the internet to send agent data to Azure Log Analytics Simple proxy No SSL bump Squid config is attached I tried disabling caching with Squid, found it crashed more frequently Squid was chosen this is intended to be a stopgap solution, and it's free. It's a battle to win over security in order to have tcp/443 opened everywhere. I'm not sure Squid is the problem, I have an identically configured Squid that bypasses the F5 working beautifully, but it's only 50 clients (MMA's) connecting, Each client takes roughly 5 connections. The clients are still going through a firewall(s). Our network folks say that neither the FW or F5 leading up to Squid report congestion. It's possible that Squid for Windows + F5 VIP are not intended to work together, but it makes sense to just have one proxy IP address. I'm getting ready to Skype with our F5 guy to compare what I'm seeing with what he's seeing. Also trying to get how many clients are going through the F5 to compare to my 'good' Squid -Original Message- From: elie...@ngtech.co.il Sent: Wednesday, February 27, 2019 9:20 AM To: Van Order, Drew (US - Hermitage) Cc: squid-users@lists.squid-cache.org Subject: [EXT] RE: [squid-users] Squid for Windows Repeatedly Crashing The setup itself is not clear to me. Is it a simple proxy? With SSL bump? Can you share or send me the squid configuration? There might be another solution for your use case that you have yet to try. Also if the purpose is not caching, why do you try to use squid? There are lots of other proxies for windows out there? (just wondering what and why have you choose Squid) Thanks, Eliezer Eliezer Croitoru Linux System Administrator Mobile: +972-5-28704261 Email: elie...@ngtech.co.il -Original Message- From: Van Order, Drew (US - Hermitage) Sent: Wednesday, February 27, 2019 05:51 To: Eliezer Croitoru ; Rafael Akchurin Cc: 'Amos Jeffries' ; squid-users@lists.squid-cache.org Subject: RE: [squid-users] Squid for Windows Repeatedly Crashing Hello folks, and thanks for keeping interest. Today I spent a bit of time learning squidclient, and have determined that the server is not in any way resource constrained. I've attached the output from mgr:info, mgr:client_list, and mgr:filedescriptors in between crashes. Was wondering if someone could explain Tout, which I presume is timeout. Of interest are the ones set to 86400, which I presume is one day. That seems like a big problem--but where is it coming from? I'm using the Cygwin Squid config defaults. There seems to be a lot of Reading next request going on before Squid recycles. I wonder if the F5 VIP is dealing with congestion through the firewall, which, in turn, is causing congestion on the pool output side, the 10.26.25.220 address. Our F5 guys have gone silent on me, I have been asking questions, in particular why all the F5 traffic is coming over just one IP address in the pool. In case folks wonder what the IP's are in the file descriptor output 1310 Socket 8986044*2806 40.71.12.224:443 593a6510-ebfc-4d6b-a8f0-a0411dfee098.ods.opinsights.azure.com:443 (this is Squid forwarding Windows event/perf data from an agent to Azure Log Analytics) 1311 Socket 8993015*9208 10.26.25.220:61088Reading next request (10.26.25.220) is the pool IP address of the F5 in use) 1312 Socket 8992690*8826 10.26.25.220:61436Reading next request 1313 Socket 8999169*2884 104.208.163.218:443 eus2-jobruntimedata-prod-su1.azure-automation.net:443 (Squid to Azure) 1314 Socket 8998787*2508 104.208.163.218:443 eus2-jobruntimedata-prod-su1.azure-automation.net:443 1315 Socket 118 119*3924 10.26.25.220:52153Idle client: Waiting for next request 1316 Socket 9001382*8697 10.26.25.220:54786Reading next request This is from a box that restarts squid every few minutes. Typical cache.log snippet 2019/02/26 21:24:22 kid1| storeDirWriteCleanLogs: Starting... 2019/02/26 21:24
Re: [squid-users] Squid for Windows Repeatedly Crashing
Business objective is to enable MSFT Azure MMA's (Microsoft Monitoring Agents) blocked from the internet to send agent data to Azure Log Analytics Simple proxy No SSL bump Squid config is attached I tried disabling caching with Squid, found it crashed more frequently Squid was chosen this is intended to be a stopgap solution, and it's free. It's a battle to win over security in order to have tcp/443 opened everywhere. I'm not sure Squid is the problem, I have an identically configured Squid that bypasses the F5 working beautifully, but it's only 50 clients (MMA's) connecting, Each client takes roughly 5 connections. The clients are still going through a firewall(s). Our network folks say that neither the FW or F5 leading up to Squid report congestion. It's possible that Squid for Windows + F5 VIP are not intended to work together, but it makes sense to just have one proxy IP address. I'm getting ready to Skype with our F5 guy to compare what I'm seeing with what he's seeing. Also trying to get how many clients are going through the F5 to compare to my 'good' Squid -Original Message- From: elie...@ngtech.co.il Sent: Wednesday, February 27, 2019 9:20 AM To: Van Order, Drew (US - Hermitage) Cc: squid-users@lists.squid-cache.org Subject: [EXT] RE: [squid-users] Squid for Windows Repeatedly Crashing The setup itself is not clear to me. Is it a simple proxy? With SSL bump? Can you share or send me the squid configuration? There might be another solution for your use case that you have yet to try. Also if the purpose is not caching, why do you try to use squid? There are lots of other proxies for windows out there? (just wondering what and why have you choose Squid) Thanks, Eliezer Eliezer Croitoru Linux System Administrator Mobile: +972-5-28704261 Email: elie...@ngtech.co.il -Original Message- From: Van Order, Drew (US - Hermitage) Sent: Wednesday, February 27, 2019 05:51 To: Eliezer Croitoru ; Rafael Akchurin Cc: 'Amos Jeffries' ; squid-users@lists.squid-cache.org Subject: RE: [squid-users] Squid for Windows Repeatedly Crashing Hello folks, and thanks for keeping interest. Today I spent a bit of time learning squidclient, and have determined that the server is not in any way resource constrained. I've attached the output from mgr:info, mgr:client_list, and mgr:filedescriptors in between crashes. Was wondering if someone could explain Tout, which I presume is timeout. Of interest are the ones set to 86400, which I presume is one day. That seems like a big problem--but where is it coming from? I'm using the Cygwin Squid config defaults. There seems to be a lot of Reading next request going on before Squid recycles. I wonder if the F5 VIP is dealing with congestion through the firewall, which, in turn, is causing congestion on the pool output side, the 10.26.25.220 address. Our F5 guys have gone silent on me, I have been asking questions, in particular why all the F5 traffic is coming over just one IP address in the pool. In case folks wonder what the IP's are in the file descriptor output 1310 Socket 8986044*2806 40.71.12.224:443 593a6510-ebfc-4d6b-a8f0-a0411dfee098.ods.opinsights.azure.com:443 (this is Squid forwarding Windows event/perf data from an agent to Azure Log Analytics) 1311 Socket 8993015*9208 10.26.25.220:61088Reading next request (10.26.25.220) is the pool IP address of the F5 in use) 1312 Socket 8992690*8826 10.26.25.220:61436Reading next request 1313 Socket 8999169*2884 104.208.163.218:443 eus2-jobruntimedata-prod-su1.azure-automation.net:443 (Squid to Azure) 1314 Socket 8998787*2508 104.208.163.218:443 eus2-jobruntimedata-prod-su1.azure-automation.net:443 1315 Socket 118 119*3924 10.26.25.220:52153Idle client: Waiting for next request 1316 Socket 9001382*8697 10.26.25.220:54786Reading next request This is from a box that restarts squid every few minutes. Typical cache.log snippet 2019/02/26 21:24:22 kid1| storeDirWriteCleanLogs: Starting... 2019/02/26 21:24:22 kid1| Finished. Wrote 0 entries. 2019/02/26 21:24:22 kid1| Took 0.00 seconds ( 0.00 entries/sec). 2019/02/26 21:24:26 kid1| Set Current Directory to /var/cache/squid 2019/02/26 21:24:26 kid1| Starting Squid Cache version 3.5.28 for x86_64-unknown-cygwin... 2019/02/26 21:24:26 kid1| Service Name: squid 2019/02/26 21:24:26 kid1| Process ID 1796 2019/02/26 21:24:26 kid1| Process Roles: worker 2019/02/26 21:24:26 kid1| With 3200 file descriptors available 2019/02/26 21:24:26 kid1| Initializing IP Cache... 2019/02/26 21:24:26 kid1| parseEtcHosts: /etc/hosts: (2) No such file or directory 2019/02/26 21:24:26 kid1| DNS Socket created at [::], FD 5 2019/02/26 21:24:26 kid1| DNS Socket created at 0.0.0.0, FD 6 2019/02/26 21:24:26 kid1| Adding nameserver 208.67.220.220 from squid.conf 2019/02/26 21:
Re: [squid-users] Squid for Windows Repeatedly Crashing
The setup itself is not clear to me. Is it a simple proxy? With SSL bump? Can you share or send me the squid configuration? There might be another solution for your use case that you have yet to try. Also if the purpose is not caching, why do you try to use squid? There are lots of other proxies for windows out there? (just wondering what and why have you choose Squid) Thanks, Eliezer Eliezer Croitoru Linux System Administrator Mobile: +972-5-28704261 Email: elie...@ngtech.co.il -Original Message- From: Van Order, Drew (US - Hermitage) Sent: Wednesday, February 27, 2019 05:51 To: Eliezer Croitoru ; Rafael Akchurin Cc: 'Amos Jeffries' ; squid-users@lists.squid-cache.org Subject: RE: [squid-users] Squid for Windows Repeatedly Crashing Hello folks, and thanks for keeping interest. Today I spent a bit of time learning squidclient, and have determined that the server is not in any way resource constrained. I've attached the output from mgr:info, mgr:client_list, and mgr:filedescriptors in between crashes. Was wondering if someone could explain Tout, which I presume is timeout. Of interest are the ones set to 86400, which I presume is one day. That seems like a big problem--but where is it coming from? I'm using the Cygwin Squid config defaults. There seems to be a lot of Reading next request going on before Squid recycles. I wonder if the F5 VIP is dealing with congestion through the firewall, which, in turn, is causing congestion on the pool output side, the 10.26.25.220 address. Our F5 guys have gone silent on me, I have been asking questions, in particular why all the F5 traffic is coming over just one IP address in the pool. In case folks wonder what the IP's are in the file descriptor output 1310 Socket 8986044*2806 40.71.12.224:443 593a6510-ebfc-4d6b-a8f0-a0411dfee098.ods.opinsights.azure.com:443 (this is Squid forwarding Windows event/perf data from an agent to Azure Log Analytics) 1311 Socket 8993015*9208 10.26.25.220:61088Reading next request (10.26.25.220) is the pool IP address of the F5 in use) 1312 Socket 8992690*8826 10.26.25.220:61436Reading next request 1313 Socket 8999169*2884 104.208.163.218:443 eus2-jobruntimedata-prod-su1.azure-automation.net:443 (Squid to Azure) 1314 Socket 8998787*2508 104.208.163.218:443 eus2-jobruntimedata-prod-su1.azure-automation.net:443 1315 Socket 118 119*3924 10.26.25.220:52153Idle client: Waiting for next request 1316 Socket 9001382*8697 10.26.25.220:54786Reading next request This is from a box that restarts squid every few minutes. Typical cache.log snippet 2019/02/26 21:24:22 kid1| storeDirWriteCleanLogs: Starting... 2019/02/26 21:24:22 kid1| Finished. Wrote 0 entries. 2019/02/26 21:24:22 kid1| Took 0.00 seconds ( 0.00 entries/sec). 2019/02/26 21:24:26 kid1| Set Current Directory to /var/cache/squid 2019/02/26 21:24:26 kid1| Starting Squid Cache version 3.5.28 for x86_64-unknown-cygwin... 2019/02/26 21:24:26 kid1| Service Name: squid 2019/02/26 21:24:26 kid1| Process ID 1796 2019/02/26 21:24:26 kid1| Process Roles: worker 2019/02/26 21:24:26 kid1| With 3200 file descriptors available 2019/02/26 21:24:26 kid1| Initializing IP Cache... 2019/02/26 21:24:26 kid1| parseEtcHosts: /etc/hosts: (2) No such file or directory 2019/02/26 21:24:26 kid1| DNS Socket created at [::], FD 5 2019/02/26 21:24:26 kid1| DNS Socket created at 0.0.0.0, FD 6 2019/02/26 21:24:26 kid1| Adding nameserver 208.67.220.220 from squid.conf 2019/02/26 21:24:26 kid1| Adding nameserver 208.67.222.222 from squid.conf 2019/02/26 21:24:26 kid1| Logfile: opening log daemon:/var/log/squid/access.log 2019/02/26 21:24:26 kid1| Logfile Daemon: opening log /var/log/squid/access.log 2019/02/26 21:24:26 kid1| WARNING: no_suid: setuid(0): (22) Invalid argument 2019/02/26 21:24:26 kid1| Store logging disabled 2019/02/26 21:24:26 kid1| Swap maxSize 3072000 + 262144 KB, estimated 256472 objects 2019/02/26 21:24:26 kid1| Target number of buckets: 12823 2019/02/26 21:24:26 kid1| Using 16384 Store buckets 2019/02/26 21:24:26 kid1| Max Mem size: 262144 KB 2019/02/26 21:24:26 kid1| Max Swap size: 3072000 KB 2019/02/26 21:24:26 kid1| Rebuilding storage in /cygdrive/e/squid/cache (clean log) 2019/02/26 21:24:26 kid1| Using Least Load store dir selection 2019/02/26 21:24:26 kid1| Set Current Directory to /var/cache/squid 2019/02/26 21:24:26 kid1| Finished loading MIME types and icons. 2019/02/26 21:24:26 kid1| HTCP Disabled. 2019/02/26 21:24:26 kid1| Squid plugin modules loaded: 0 2019/02/26 21:24:26 kid1| Adaptation support is off. 2019/02/26 21:24:26 kid1| Accepting HTTP Socket connections at local=10.26.24.65:3128 remote=[::] FD 12 flags=9 2019/02/26 21:24:26 kid1| Done reading /cygdrive/e/squid/cache swaplog (0 entries) 2019/02/26 21:24:26 kid1| Store rebuilding is 0.00% complete 2019/02/26 21:24:26 kid1| Finished rebuilding storage from disk. 2019/02/26 21:24:26 kid1|
Re: [squid-users] Squid for Windows Repeatedly Crashing
It depends on the hardware in the server grade Windows. It can take more then 3k conn's for 100%. It's possible that squid was not designed for windows 2k16 Eliezer On 2019-02-24 15:47, Rafael Akchurin wrote: As far as I know the internal FD limit for Windows build is around 3K - might be being existed and thus unexpected behavior raising its ugly head.. -Original Message- From: squid-users On Behalf Of Van Order, Drew (US - Hermitage) Sent: Sunday, 24 February 2019 14:40 To: elie...@ngtech.co.il; 'Amos Jeffries' ; squid-users@lists.squid-cache.org Subject: Re: [squid-users] Squid for Windows Repeatedly Crashing This is helpful, and I especially appreciate the time given it is the weekend. ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] Squid for Windows Repeatedly Crashing
It depends on the hardware in the server grade Windows. It can take more then 3k conn's for 100%. It's possible that squid was not designed for windows 2k16 Eliezer Croitoru Linux System Administrator Mobile: +972-5-28704261 Email: elie...@ngtech.co.il -Original Message- From: Rafael Akchurin Sent: Sunday, February 24, 2019 15:47 To: Van Order, Drew (US - Hermitage) ; elie...@ngtech.co.il; 'Amos Jeffries' ; squid-users@lists.squid-cache.org Subject: RE: [squid-users] Squid for Windows Repeatedly Crashing As far as I know the internal FD limit for Windows build is around 3K - might be being existed and thus unexpected behavior raising its ugly head.. -Original Message- From: squid-users On Behalf Of Van Order, Drew (US - Hermitage) Sent: Sunday, 24 February 2019 14:40 To: elie...@ngtech.co.il; 'Amos Jeffries' ; squid-users@lists.squid-cache.org Subject: Re: [squid-users] Squid for Windows Repeatedly Crashing This is helpful, and I especially appreciate the time given it is the weekend. The Squids are confusing me, as everything is well behaved at the moment. One server was erroring off and on for a few hours earlier today, but stopped after a reboot. It does appear that redirecting roughly 125 servers to no longer use the proxy has helped. Unfortunately, our F5 guy can't tell me how many IP addresses remain coming into this F5 VIP, which would give me the number of servers, and an idea how loaded this thing is. I have good reason to believe it is under 1,000. He has shown us graphs indicating the VIP isn't stressed, but I will keep working on him, b/c I can't imagine not being able to report how many distinct IP addresses hit the VIP. I don't have a Visio, but Server running the Microsoft Monitoring Agent sends data over tcp/443-->Internal facing firewall(s)-->F5 VIP-->one of 4 Squids-->internet Each of the 4 VMWare Squids has 4 proc and 8 GB memory, 10 GB NIC. We're a large enterprise with multiple data centers and many subnets, so there are quite a few firewalls, and most of the time a server must go through more than one firewall. Can't help but wonder if firewall exhaustion could cause the symptoms. Revision: I typed the above last night. This morning, the server that had been erroring is at it again, but stopped. Others are fine. Interesting problem. -Original Message- From: elie...@ngtech.co.il Sent: Saturday, February 23, 2019 12:16 PM To: Van Order, Drew (US - Hermitage) ; 'Amos Jeffries' ; squid-users@lists.squid-cache.org Subject: [EXT] RE: [squid-users] Squid for Windows Repeatedly Crashing The next tool might help you to understand the status of the open connections. If the socket is being closed( I think Windows Server 2016 is a very good OS...). https://secure-web.cisco.com/1gLLf4HP_bwYOteW6x8gJ8EGyBrYzTMzMIi7P6q7aGi136WObNRd7uZQkrv-CKTO7ipHpLgOvHaGbzxLT7RpG6AGtkeTHUn2O8-CIAgcBOCUzn6KyZoPhqsAcpIXokXWcjlWHdUVUwlZVT0WKEhuOuAGvw2washhJEOg1Gcbsf99cy7ofqJfuTc-fS23KxfiE8W-2GLLNuF_J8q5uGJdvUMhm6HN-4CO3c_i8wxOlHrxgX3GjSLbLo8odnA6YctD5A01sjW3dpC4oiioIkGY7gDY-hjSSNYr_xoZzsixScColG-JRDlR3uktjsFF5JCkU1EROfoOfUHsDdeJ0IV2Cpk6yzbSPNNno7jV5BmZSsmR_jRgW7WJa4eVhKUvicMfy8RBespjtbfk17lUf9JamqmxPBtP2eHsiIb4_wk9iJfRr_S-aA1Ve7rPDmCXm9bZ9HRmXphi8o5AeYMWbK9DTrnmPDmFamis922AT6F4KUuBvS3PKqeCkT3EUuGmlwHXxCiJGwYBKXQmOehcFbqgfFQ/https%3A%2F%2Fwww.nirsoft.net%2Futils%2Fcports.html There is a possibility that some OS TCP limit is being reached and there for the socket closure. If you are using F5 you can easily find out the load at the crash point. I assume that if a normal Squid instance can take a load of 900k requests per second in somewhat constant rate for more than a minute then the issue might be else where then squid. I am not sure but pretty sure that if you do not have anyone that is knowledgeable enough about windows sockets, sessions and FW limitations you will either: - learn it your self - find an expert - use an OS that is more then 20% supported by any of the Squid-Cache team members and other developers around the globe. Just to say a good word about Windows Server 2016, I compared it to a Windows 10 under load and it seems to take a lot more load. Also it not just takes the load but balance it well (on an open source windows designed software). Also if you have a specific use case maybe a specific proxy can be customized for it. Let me know if you wish to shed more details on the configuration so I can take my time and understand if there is a solution else then Squid. Eliezeer Eliezer Croitoru Linux System Administrator Mobile: +972-5-28704261 Email: elie...@ngtech.co.il -Original Message- From: squid-users On Behalf Of Van Order, Drew (US - Hermitage) Sent: Friday, February 22, 2019 15:32 To: Amos Jeffries ; squid-users@lists.squid-cache.org Subject: Re: [squid-users] Squid for Windows Repeatedly Crashing The test bo
Re: [squid-users] Squid for Windows Repeatedly Crashing
Wondered that too, and experimented with the squid.conf setting to see when warning messages appear, which was 1,000. Default cygwin setting of 3,200 is plenty. Thanks for the idea, though! Sent via the Samsung Galaxy S8+, an AT&T 5G Evolution smartphone Original message From: Rafael Akchurin Date: 2/24/19 8:47 AM (GMT-05:00) To: "Van Order, Drew (US - Hermitage)" , elie...@ngtech.co.il, 'Amos Jeffries' , squid-users@lists.squid-cache.org Subject: [EXT] RE: [squid-users] Squid for Windows Repeatedly Crashing As far as I know the internal FD limit for Windows build is around 3K - might be being existed and thus unexpected behavior raising its ugly head.. -Original Message- From: squid-users On Behalf Of Van Order, Drew (US - Hermitage) Sent: Sunday, 24 February 2019 14:40 To: elie...@ngtech.co.il; 'Amos Jeffries' ; squid-users@lists.squid-cache.org Subject: Re: [squid-users] Squid for Windows Repeatedly Crashing This is helpful, and I especially appreciate the time given it is the weekend. The Squids are confusing me, as everything is well behaved at the moment. One server was erroring off and on for a few hours earlier today, but stopped after a reboot. It does appear that redirecting roughly 125 servers to no longer use the proxy has helped. Unfortunately, our F5 guy can't tell me how many IP addresses remain coming into this F5 VIP, which would give me the number of servers, and an idea how loaded this thing is. I have good reason to believe it is under 1,000. He has shown us graphs indicating the VIP isn't stressed, but I will keep working on him, b/c I can't imagine not being able to report how many distinct IP addresses hit the VIP. I don't have a Visio, but Server running the Microsoft Monitoring Agent sends data over tcp/443-->Internal facing firewall(s)-->F5 VIP-->one of 4 Squids-->internet Each of the 4 VMWare Squids has 4 proc and 8 GB memory, 10 GB NIC. We're a large enterprise with multiple data centers and many subnets, so there are quite a few firewalls, and most of the time a server must go through more than one firewall. Can't help but wonder if firewall exhaustion could cause the symptoms. Revision: I typed the above last night. This morning, the server that had been erroring is at it again, but stopped. Others are fine. Interesting problem. -Original Message- From: elie...@ngtech.co.il Sent: Saturday, February 23, 2019 12:16 PM To: Van Order, Drew (US - Hermitage) ; 'Amos Jeffries' ; squid-users@lists.squid-cache.org Subject: [EXT] RE: [squid-users] Squid for Windows Repeatedly Crashing The next tool might help you to understand the status of the open connections. If the socket is being closed( I think Windows Server 2016 is a very good OS...). https://secure-web.cisco.com/1gLLf4HP_bwYOteW6x8gJ8EGyBrYzTMzMIi7P6q7aGi136WObNRd7uZQkrv-CKTO7ipHpLgOvHaGbzxLT7RpG6AGtkeTHUn2O8-CIAgcBOCUzn6KyZoPhqsAcpIXokXWcjlWHdUVUwlZVT0WKEhuOuAGvw2washhJEOg1Gcbsf99cy7ofqJfuTc-fS23KxfiE8W-2GLLNuF_J8q5uGJdvUMhm6HN-4CO3c_i8wxOlHrxgX3GjSLbLo8odnA6YctD5A01sjW3dpC4oiioIkGY7gDY-hjSSNYr_xoZzsixScColG-JRDlR3uktjsFF5JCkU1EROfoOfUHsDdeJ0IV2Cpk6yzbSPNNno7jV5BmZSsmR_jRgW7WJa4eVhKUvicMfy8RBespjtbfk17lUf9JamqmxPBtP2eHsiIb4_wk9iJfRr_S-aA1Ve7rPDmCXm9bZ9HRmXphi8o5AeYMWbK9DTrnmPDmFamis922AT6F4KUuBvS3PKqeCkT3EUuGmlwHXxCiJGwYBKXQmOehcFbqgfFQ/https%3A%2F%2Fwww.nirsoft.net%2Futils%2Fcports.html There is a possibility that some OS TCP limit is being reached and there for the socket closure. If you are using F5 you can easily find out the load at the crash point. I assume that if a normal Squid instance can take a load of 900k requests per second in somewhat constant rate for more than a minute then the issue might be else where then squid. I am not sure but pretty sure that if you do not have anyone that is knowledgeable enough about windows sockets, sessions and FW limitations you will either: - learn it your self - find an expert - use an OS that is more then 20% supported by any of the Squid-Cache team members and other developers around the globe. Just to say a good word about Windows Server 2016, I compared it to a Windows 10 under load and it seems to take a lot more load. Also it not just takes the load but balance it well (on an open source windows designed software). Also if you have a specific use case maybe a specific proxy can be customized for it. Let me know if you wish to shed more details on the configuration so I can take my time and understand if there is a solution else then Squid. Eliezeer Eliezer Croitoru Linux System Administrator Mobile: +972-5-28704261 Email: elie...@ngtech.co.il -Original Message- From: squid-users On Behalf Of Van Order, Drew (US - Hermitage) Sent: Friday, February 22, 2019 15:32 To: Amos Jeffries ; squid-users@lists.squid-cache.org Subject: Re: [squid-users] Squid for Windows Repeatedly Crashing
Re: [squid-users] Squid for Windows Repeatedly Crashing
As far as I know the internal FD limit for Windows build is around 3K - might be being existed and thus unexpected behavior raising its ugly head.. -Original Message- From: squid-users On Behalf Of Van Order, Drew (US - Hermitage) Sent: Sunday, 24 February 2019 14:40 To: elie...@ngtech.co.il; 'Amos Jeffries' ; squid-users@lists.squid-cache.org Subject: Re: [squid-users] Squid for Windows Repeatedly Crashing This is helpful, and I especially appreciate the time given it is the weekend. The Squids are confusing me, as everything is well behaved at the moment. One server was erroring off and on for a few hours earlier today, but stopped after a reboot. It does appear that redirecting roughly 125 servers to no longer use the proxy has helped. Unfortunately, our F5 guy can't tell me how many IP addresses remain coming into this F5 VIP, which would give me the number of servers, and an idea how loaded this thing is. I have good reason to believe it is under 1,000. He has shown us graphs indicating the VIP isn't stressed, but I will keep working on him, b/c I can't imagine not being able to report how many distinct IP addresses hit the VIP. I don't have a Visio, but Server running the Microsoft Monitoring Agent sends data over tcp/443-->Internal facing firewall(s)-->F5 VIP-->one of 4 Squids-->internet Each of the 4 VMWare Squids has 4 proc and 8 GB memory, 10 GB NIC. We're a large enterprise with multiple data centers and many subnets, so there are quite a few firewalls, and most of the time a server must go through more than one firewall. Can't help but wonder if firewall exhaustion could cause the symptoms. Revision: I typed the above last night. This morning, the server that had been erroring is at it again, but stopped. Others are fine. Interesting problem. -Original Message- From: elie...@ngtech.co.il Sent: Saturday, February 23, 2019 12:16 PM To: Van Order, Drew (US - Hermitage) ; 'Amos Jeffries' ; squid-users@lists.squid-cache.org Subject: [EXT] RE: [squid-users] Squid for Windows Repeatedly Crashing The next tool might help you to understand the status of the open connections. If the socket is being closed( I think Windows Server 2016 is a very good OS...). https://secure-web.cisco.com/1gLLf4HP_bwYOteW6x8gJ8EGyBrYzTMzMIi7P6q7aGi136WObNRd7uZQkrv-CKTO7ipHpLgOvHaGbzxLT7RpG6AGtkeTHUn2O8-CIAgcBOCUzn6KyZoPhqsAcpIXokXWcjlWHdUVUwlZVT0WKEhuOuAGvw2washhJEOg1Gcbsf99cy7ofqJfuTc-fS23KxfiE8W-2GLLNuF_J8q5uGJdvUMhm6HN-4CO3c_i8wxOlHrxgX3GjSLbLo8odnA6YctD5A01sjW3dpC4oiioIkGY7gDY-hjSSNYr_xoZzsixScColG-JRDlR3uktjsFF5JCkU1EROfoOfUHsDdeJ0IV2Cpk6yzbSPNNno7jV5BmZSsmR_jRgW7WJa4eVhKUvicMfy8RBespjtbfk17lUf9JamqmxPBtP2eHsiIb4_wk9iJfRr_S-aA1Ve7rPDmCXm9bZ9HRmXphi8o5AeYMWbK9DTrnmPDmFamis922AT6F4KUuBvS3PKqeCkT3EUuGmlwHXxCiJGwYBKXQmOehcFbqgfFQ/https%3A%2F%2Fwww.nirsoft.net%2Futils%2Fcports.html There is a possibility that some OS TCP limit is being reached and there for the socket closure. If you are using F5 you can easily find out the load at the crash point. I assume that if a normal Squid instance can take a load of 900k requests per second in somewhat constant rate for more than a minute then the issue might be else where then squid. I am not sure but pretty sure that if you do not have anyone that is knowledgeable enough about windows sockets, sessions and FW limitations you will either: - learn it your self - find an expert - use an OS that is more then 20% supported by any of the Squid-Cache team members and other developers around the globe. Just to say a good word about Windows Server 2016, I compared it to a Windows 10 under load and it seems to take a lot more load. Also it not just takes the load but balance it well (on an open source windows designed software). Also if you have a specific use case maybe a specific proxy can be customized for it. Let me know if you wish to shed more details on the configuration so I can take my time and understand if there is a solution else then Squid. Eliezeer Eliezer Croitoru Linux System Administrator Mobile: +972-5-28704261 Email: elie...@ngtech.co.il -Original Message- From: squid-users On Behalf Of Van Order, Drew (US - Hermitage) Sent: Friday, February 22, 2019 15:32 To: Amos Jeffries ; squid-users@lists.squid-cache.org Subject: Re: [squid-users] Squid for Windows Repeatedly Crashing The test box I set up outside the F5 finally started exhibiting these errors, once I pointed roughly 60 machines to it. It took a few hours. Sounds like this narrows it down to either the OS itself (seems unlikely, other apps would crash), or the litany of agents our security folks have mandated. It may indeed be necessary to move to Linux. Thank you very much for your time! -Original Message- From: Amos Jeffries Sent: Thursday, February 21, 2019 11:31 PM To: Van Order, Drew (US - Hermitage) ; squid-users@lists.squid-cache.org Subject: [EXT] Re: [s
Re: [squid-users] Squid for Windows Repeatedly Crashing
This is helpful, and I especially appreciate the time given it is the weekend. The Squids are confusing me, as everything is well behaved at the moment. One server was erroring off and on for a few hours earlier today, but stopped after a reboot. It does appear that redirecting roughly 125 servers to no longer use the proxy has helped. Unfortunately, our F5 guy can't tell me how many IP addresses remain coming into this F5 VIP, which would give me the number of servers, and an idea how loaded this thing is. I have good reason to believe it is under 1,000. He has shown us graphs indicating the VIP isn't stressed, but I will keep working on him, b/c I can't imagine not being able to report how many distinct IP addresses hit the VIP. I don't have a Visio, but Server running the Microsoft Monitoring Agent sends data over tcp/443-->Internal facing firewall(s)-->F5 VIP-->one of 4 Squids-->internet Each of the 4 VMWare Squids has 4 proc and 8 GB memory, 10 GB NIC. We're a large enterprise with multiple data centers and many subnets, so there are quite a few firewalls, and most of the time a server must go through more than one firewall. Can't help but wonder if firewall exhaustion could cause the symptoms. Revision: I typed the above last night. This morning, the server that had been erroring is at it again, but stopped. Others are fine. Interesting problem. -Original Message- From: elie...@ngtech.co.il Sent: Saturday, February 23, 2019 12:16 PM To: Van Order, Drew (US - Hermitage) ; 'Amos Jeffries' ; squid-users@lists.squid-cache.org Subject: [EXT] RE: [squid-users] Squid for Windows Repeatedly Crashing The next tool might help you to understand the status of the open connections. If the socket is being closed( I think Windows Server 2016 is a very good OS...). https://secure-web.cisco.com/1gLLf4HP_bwYOteW6x8gJ8EGyBrYzTMzMIi7P6q7aGi136WObNRd7uZQkrv-CKTO7ipHpLgOvHaGbzxLT7RpG6AGtkeTHUn2O8-CIAgcBOCUzn6KyZoPhqsAcpIXokXWcjlWHdUVUwlZVT0WKEhuOuAGvw2washhJEOg1Gcbsf99cy7ofqJfuTc-fS23KxfiE8W-2GLLNuF_J8q5uGJdvUMhm6HN-4CO3c_i8wxOlHrxgX3GjSLbLo8odnA6YctD5A01sjW3dpC4oiioIkGY7gDY-hjSSNYr_xoZzsixScColG-JRDlR3uktjsFF5JCkU1EROfoOfUHsDdeJ0IV2Cpk6yzbSPNNno7jV5BmZSsmR_jRgW7WJa4eVhKUvicMfy8RBespjtbfk17lUf9JamqmxPBtP2eHsiIb4_wk9iJfRr_S-aA1Ve7rPDmCXm9bZ9HRmXphi8o5AeYMWbK9DTrnmPDmFamis922AT6F4KUuBvS3PKqeCkT3EUuGmlwHXxCiJGwYBKXQmOehcFbqgfFQ/https%3A%2F%2Fwww.nirsoft.net%2Futils%2Fcports.html There is a possibility that some OS TCP limit is being reached and there for the socket closure. If you are using F5 you can easily find out the load at the crash point. I assume that if a normal Squid instance can take a load of 900k requests per second in somewhat constant rate for more than a minute then the issue might be else where then squid. I am not sure but pretty sure that if you do not have anyone that is knowledgeable enough about windows sockets, sessions and FW limitations you will either: - learn it your self - find an expert - use an OS that is more then 20% supported by any of the Squid-Cache team members and other developers around the globe. Just to say a good word about Windows Server 2016, I compared it to a Windows 10 under load and it seems to take a lot more load. Also it not just takes the load but balance it well (on an open source windows designed software). Also if you have a specific use case maybe a specific proxy can be customized for it. Let me know if you wish to shed more details on the configuration so I can take my time and understand if there is a solution else then Squid. Eliezeer Eliezer Croitoru Linux System Administrator Mobile: +972-5-28704261 Email: elie...@ngtech.co.il -Original Message- From: squid-users On Behalf Of Van Order, Drew (US - Hermitage) Sent: Friday, February 22, 2019 15:32 To: Amos Jeffries ; squid-users@lists.squid-cache.org Subject: Re: [squid-users] Squid for Windows Repeatedly Crashing The test box I set up outside the F5 finally started exhibiting these errors, once I pointed roughly 60 machines to it. It took a few hours. Sounds like this narrows it down to either the OS itself (seems unlikely, other apps would crash), or the litany of agents our security folks have mandated. It may indeed be necessary to move to Linux. Thank you very much for your time! -Original Message- From: Amos Jeffries Sent: Thursday, February 21, 2019 11:31 PM To: Van Order, Drew (US - Hermitage) ; squid-users@lists.squid-cache.org Subject: [EXT] Re: [squid-users] Squid for Windows Repeatedly Crashing On 22/02/19 4:21 am, Van Order, Drew (US - Hermitage) wrote: > Thank you for replying, and that's an excellent point. > > Short answer--definitely not in a container, these are garden variety VMWare instances. I've already flagged the OS power settings to maximum performance, so nothing should be going to sleep. I'll doublecheck, though. > > So
Re: [squid-users] Squid for Windows Repeatedly Crashing
The next tool might help you to understand the status of the open connections. If the socket is being closed( I think Windows Server 2016 is a very good OS...). https://www.nirsoft.net/utils/cports.html There is a possibility that some OS TCP limit is being reached and there for the socket closure. If you are using F5 you can easily find out the load at the crash point. I assume that if a normal Squid instance can take a load of 900k requests per second in somewhat constant rate for more then a minute then the issue might be else where then squid. I am not sure but pretty sure that if you do not have anyone that is knowledgeable enough about windows sockets, sessions and FW limitations you will either: - learn it your self - find an expert - use an OS that is more then 20% supported by any of the Squid-Cache team members and other developers around the globe. Just to say a good word about Windows Server 2016, I compared it to a Windows 10 under load and it seems to take a lot more load. Also it not just takes the load but balance it well (on an open source windows designed software). Also if you have a specific use case maybe a specific proxy can be customized for it. Let me know if you wish to shed more details on the configuration so I can take my time and understand if there is a solution else then Squid. Eliezeer Eliezer Croitoru Linux System Administrator Mobile: +972-5-28704261 Email: elie...@ngtech.co.il -Original Message- From: squid-users On Behalf Of Van Order, Drew (US - Hermitage) Sent: Friday, February 22, 2019 15:32 To: Amos Jeffries ; squid-users@lists.squid-cache.org Subject: Re: [squid-users] Squid for Windows Repeatedly Crashing The test box I set up outside the F5 finally started exhibiting these errors, once I pointed roughly 60 machines to it. It took a few hours. Sounds like this narrows it down to either the OS itself (seems unlikely, other apps would crash), or the litany of agents our security folks have mandated. It may indeed be necessary to move to Linux. Thank you very much for your time! -Original Message- From: Amos Jeffries Sent: Thursday, February 21, 2019 11:31 PM To: Van Order, Drew (US - Hermitage) ; squid-users@lists.squid-cache.org Subject: [EXT] Re: [squid-users] Squid for Windows Repeatedly Crashing On 22/02/19 4:21 am, Van Order, Drew (US - Hermitage) wrote: > Thank you for replying, and that's an excellent point. > > Short answer--definitely not in a container, these are garden variety VMWare instances. I've already flagged the OS power settings to maximum performance, so nothing should be going to sleep. I'll doublecheck, though. > > So, if I understand correctly, this error could also be indicative of an issue in between the agent and Squid. Agents first go through a firewall, then the F5 before reaching Squid. No that is not what I meant. The port Squid has already opened and used syscall listen(2) on is what is being closed (or its address corrupted) outside of Squid. That should only ever be closed by Squid itself. Thus the error. It is being closed repeatedly. Thus the abort/shutdown. This is not a crash, it is intentional shutdown by Squid due to these fatal (non-recoverable) errors. > > [Stopped, reason:Listener socket closed job1]: (14) Bad address > > Any thoughts on this error, which tends to be more common than the other? > > 2019/02/20 09:42:33 kid1| comm_poll: poll failure: (14) Bad address > 2019/02/20 09:42:33 kid1| Select loop Error. Retry 2 > Notice how the error from the OS "(14) Bad Address" is the same. This is just another display of the same problem. Maybe the poll() layer reporting the exact same error as Squid tries to recover. Maybe for other non-listener ports also being corrupted somehow. If non-listener ports are having that same error it would be a sign the machine memory is being corrupted rather than other software touching the listener ports specifically. ( The details you have provided so far have no hints about where the problem may be coming from, and I am not having any ideas about possibilities either. I just hope the above explanation of meaning can help you think of things to look at for more hints on this very weird issue. ) Amos This message (including any attachments) contains confidential information intended for a specific individual and purpose, and is protected by law. If you are not the intended recipient, you should delete this message and any disclosure, copying, or distribution of this message, or the taking of any action based on it, by you is strictly prohibited. v.E.1 ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] Squid for Windows Repeatedly Crashing
The test box I set up outside the F5 finally started exhibiting these errors, once I pointed roughly 60 machines to it. It took a few hours. Sounds like this narrows it down to either the OS itself (seems unlikely, other apps would crash), or the litany of agents our security folks have mandated. It may indeed be necessary to move to Linux. Thank you very much for your time! -Original Message- From: Amos Jeffries Sent: Thursday, February 21, 2019 11:31 PM To: Van Order, Drew (US - Hermitage) ; squid-users@lists.squid-cache.org Subject: [EXT] Re: [squid-users] Squid for Windows Repeatedly Crashing On 22/02/19 4:21 am, Van Order, Drew (US - Hermitage) wrote: > Thank you for replying, and that's an excellent point. > > Short answer--definitely not in a container, these are garden variety VMWare > instances. I've already flagged the OS power settings to maximum performance, > so nothing should be going to sleep. I'll doublecheck, though. > > So, if I understand correctly, this error could also be indicative of an > issue in between the agent and Squid. Agents first go through a firewall, > then the F5 before reaching Squid. No that is not what I meant. The port Squid has already opened and used syscall listen(2) on is what is being closed (or its address corrupted) outside of Squid. That should only ever be closed by Squid itself. Thus the error. It is being closed repeatedly. Thus the abort/shutdown. This is not a crash, it is intentional shutdown by Squid due to these fatal (non-recoverable) errors. > > [Stopped, reason:Listener socket closed job1]: (14) Bad address > > Any thoughts on this error, which tends to be more common than the other? > > 2019/02/20 09:42:33 kid1| comm_poll: poll failure: (14) Bad address > 2019/02/20 09:42:33 kid1| Select loop Error. Retry 2 > Notice how the error from the OS "(14) Bad Address" is the same. This is just another display of the same problem. Maybe the poll() layer reporting the exact same error as Squid tries to recover. Maybe for other non-listener ports also being corrupted somehow. If non-listener ports are having that same error it would be a sign the machine memory is being corrupted rather than other software touching the listener ports specifically. ( The details you have provided so far have no hints about where the problem may be coming from, and I am not having any ideas about possibilities either. I just hope the above explanation of meaning can help you think of things to look at for more hints on this very weird issue. ) Amos This message (including any attachments) contains confidential information intended for a specific individual and purpose, and is protected by law. If you are not the intended recipient, you should delete this message and any disclosure, copying, or distribution of this message, or the taking of any action based on it, by you is strictly prohibited. v.E.1 ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] Squid for Windows Repeatedly Crashing
On 22/02/19 4:21 am, Van Order, Drew (US - Hermitage) wrote: > Thank you for replying, and that's an excellent point. > > Short answer--definitely not in a container, these are garden variety VMWare > instances. I've already flagged the OS power settings to maximum performance, > so nothing should be going to sleep. I'll doublecheck, though. > > So, if I understand correctly, this error could also be indicative of an > issue in between the agent and Squid. Agents first go through a firewall, > then the F5 before reaching Squid. No that is not what I meant. The port Squid has already opened and used syscall listen(2) on is what is being closed (or its address corrupted) outside of Squid. That should only ever be closed by Squid itself. Thus the error. It is being closed repeatedly. Thus the abort/shutdown. This is not a crash, it is intentional shutdown by Squid due to these fatal (non-recoverable) errors. > > [Stopped, reason:Listener socket closed job1]: (14) Bad address > > Any thoughts on this error, which tends to be more common than the other? > > 2019/02/20 09:42:33 kid1| comm_poll: poll failure: (14) Bad address > 2019/02/20 09:42:33 kid1| Select loop Error. Retry 2 > Notice how the error from the OS "(14) Bad Address" is the same. This is just another display of the same problem. Maybe the poll() layer reporting the exact same error as Squid tries to recover. Maybe for other non-listener ports also being corrupted somehow. If non-listener ports are having that same error it would be a sign the machine memory is being corrupted rather than other software touching the listener ports specifically. ( The details you have provided so far have no hints about where the problem may be coming from, and I am not having any ideas about possibilities either. I just hope the above explanation of meaning can help you think of things to look at for more hints on this very weird issue. ) Amos ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] Squid for Windows Repeatedly Crashing
Thank you for replying, and that's an excellent point. Short answer--definitely not in a container, these are garden variety VMWare instances. I've already flagged the OS power settings to maximum performance, so nothing should be going to sleep. I'll doublecheck, though. So, if I understand correctly, this error could also be indicative of an issue in between the agent and Squid. Agents first go through a firewall, then the F5 before reaching Squid. [Stopped, reason:Listener socket closed job1]: (14) Bad address Any thoughts on this error, which tends to be more common than the other? 2019/02/20 09:42:33 kid1| comm_poll: poll failure: (14) Bad address 2019/02/20 09:42:33 kid1| Select loop Error. Retry 2 -Original Message- From: Amos Jeffries Sent: Thursday, February 21, 2019 6:38 AM To: squid-users@lists.squid-cache.org Subject: Re: [squid-users] Squid for Windows Repeatedly Crashing On 21/02/19 6:23 pm, Van Order, Drew (US - Hermitage) wrote: > Hello folks, we're running Squid 3.5.28 on four Windows 2016 VM's, > each has 4 CPU, 8 GB memory, 10 GB NIC. We implemented Squid to > support forwarding Azure Log Analytics data, it's all CONNECT. The > Squids are load balanced through a F5. There are less than 1,000 > servers sending data to Log Analytics. > > > > All four Squids are regularly crashing, and I don't know how to > interpret the errors in cache.log. It crashes if we disable caching too. > ... > > 2019/02/20 09:42:32 kid1| FD 12, 10.5.11.12 [Stopped, reason:Listener > socket closed job1]: (14) Bad address > Something other than Squid closed the network socket Squid was using to receive new client connections (the http_port socket). The only things which should know that socket even exists are Squid and the operating system. This is not an error I've seen before. Is this Squid maybe running in a container or VM which is being hibernated, or suspended, or anything along those lines which may cause the OS filedescriptors to change unexpectedly? Amos This message (including any attachments) contains confidential information intended for a specific individual and purpose, and is protected by law. If you are not the intended recipient, you should delete this message and any disclosure, copying, or distribution of this message, or the taking of any action based on it, by you is strictly prohibited. v.E.1 ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] Squid for Windows Repeatedly Crashing
This was my first concern with posting, having managed NetView/AIX for years-that folks may focus on the choice of OS versus the problem at hand. This is not meant to be an enterprise solution, it's something for a team of non-network engineers to use to support passing tcp/443 traffic from servers blocked from the internet. Why not choose the platform that is most familiar to the people that will need to support it? I've got a fifth Windows Squid test box running outside the F5 that has yet to error, but it only has a handful of agents sending Log Analytics data. The F5's have been checked out, so the hope is that a Squid config or OS registry change will fix this. However, if the consensus is that the Windows port is unstable under any kind of load, then we'd have to consider options. From: elie...@ngtech.co.il Sent: Thursday, February 21, 2019 12:50 AM To: Van Order, Drew (US - Hermitage) ; squid-users@lists.squid-cache.org Subject: [EXT] RE: [squid-users] Squid for Windows Repeatedly Crashing May I ask about the usage of Windows 2016 VM's compared to CentOS or Ubuntu? Eliezer Eliezer Croitoru<http://secure-web.cisco.com/1QlNYIFwJYHHQ7Gju-o31exeEfzmLfLHD-tlPBXtA4AjezZjFRrjCjQhCtZ3finQTxn34ZnlGjGrThEMYGWtTzylNEw-ofQAp8U32g0ctuACuPLDsaX0vdvlccEM9yAFrtly-r6W9v8aAND2sTwtjG_DdCWCqHr20GzEBelQB5zTXPLSrBWKwb2lQG4S9q1TfNVRxihuQEW_4yLWhCq4aD2qelhYU-Z_IcKwsQ5SDh_RAGgz1tx_F3PIGffKM9wlJgUgh75XoWakVDRnMuUx_OdOl2IgHlZsveHKOFhCiBjHKJ5MsZCUVdB2EsQ-WarU2je57Od_AQy8Le44KosAmxe7QcJbvSFxmUm6Gea-lNQZPJ__ZRbR8U-OurUTPnV8l2paOjYM2srjFrDknyxo_5KwLIn6pqIR2O2RpC6mUFl4Jj2LOaSoPW1RPPepT4-bm-YdpU5ZU9rymMsiZWEIxrhT1IJYxMYP7HuQFJ-4MmxrZAY2yUaMbB9tiyHf2CimH/http%3A%2F%2Fngtech.co.il%2Fmain-en%2F> Linux System Administrator Mobile: +972-5-28704261 Email: elie...@ngtech.co.il<mailto:elie...@ngtech.co.il> [cid:image001.png@01D2675E.DCF360D0] From: squid-users mailto:squid-users-boun...@lists.squid-cache.org>> On Behalf Of Van Order, Drew (US - Hermitage) Sent: Thursday, February 21, 2019 07:23 To: squid-users@lists.squid-cache.org<mailto:squid-users@lists.squid-cache.org> Subject: [squid-users] Squid for Windows Repeatedly Crashing Hello folks, we're running Squid 3.5.28 on four Windows 2016 VM's, each has 4 CPU, 8 GB memory, 10 GB NIC. We implemented Squid to support forwarding Azure Log Analytics data, it's all CONNECT. The Squids are load balanced through a F5. There are less than 1,000 servers sending data to Log Analytics. All four Squids are regularly crashing, and I don't know how to interpret the errors in cache.log. It crashes if we disable caching too. Any insight is appreciated-I've inherited this responsibility and more a cloud engineer than a network specialist. Thanks in advance! Typical error sequence in cache.log 2019/02/20 09:42:32 kid1| FD 12, 10.5.11.12 [Stopped, reason:Listener socket closed job1]: (14) Bad address 2019/02/20 09:42:32 kid1| FD 12, 10.5.11.12 [Stopped, reason:Listener socket closed job1]: (14) Bad address 2019/02/20 09:42:32 kid1| FD 12, 10.5.11.12 [Stopped, reason:Listener socket closed job1]: (14) Bad address 2019/02/20 09:42:32 kid1| FD 12, 10.5.11.12 [Stopped, reason:Listener socket closed job1]: (14) Bad address 2019/02/20 09:42:32 kid1| FD 12, 10.5.11.12 [Stopped, reason:Listener socket closed job1]: (14) Bad address 2019/02/20 09:42:32 kid1| FD 12, 10.5.11.12 [Stopped, reason:Listener socket closed job1]: (14) Bad address 2019/02/20 09:42:32 kid1| FD 12, 10.5.11.12 [Stopped, reason:Listener socket closed job1]: (14) Bad address 2019/02/20 09:42:33 kid1| FD 12, 10.5.11.12 [Stopped, reason:Listener socket closed job1]: (14) Bad address 2019/02/20 09:42:33 kid1| FD 12, 10.5.11.12 [Stopped, reason:Listener socket closed job1]: (14) Bad address 2019/02/20 09:42:33 kid1| FD 12, 10.5.11.12 [Stopped, reason:Listener socket closed job1]: (14) Bad address 2019/02/20 09:42:33 kid1| FD 12, 10.5.11.12 [Stopped, reason:Listener socket closed job1]: (14) Bad address 2019/02/20 09:42:33 kid1| FD 12, 10.5.11.12 [Stopped, reason:Listener socket closed job1]: (14) Bad address 2019/02/20 09:42:33 kid1| comm_poll: poll failure: (14) Bad address 2019/02/20 09:42:33 kid1| Select loop Error. Retry 1 2019/02/20 09:42:33 kid1| comm_poll: poll failure: (14) Bad address 2019/02/20 09:42:33 kid1| Select loop Error. Retry 2 2019/02/20 09:42:33 kid1| comm_poll: poll failure: (14) Bad address 2019/02/20 09:42:33 kid1| Select loop Error. Retry 3 2019/02/20 09:42:33 kid1| comm_poll: poll failure: (14) Bad address 2019/02/20 09:42:33 kid1| Select loop Error. Retry 4 2019/02/20 09:42:33 kid1| comm_poll: poll failure: (14) Bad address 2019/02/20 09:42:33 kid1| Select loop Error. Retry 5 2019/02/20 09:42:33 kid1| comm_poll: poll failure: (14) Bad address 2019/02/20 09:42:33 kid1| Select loop Error. Retry 6 2019/02/20 09:42:33 kid1| comm_poll: poll failure: (14) Bad address 2019/02/20 09:42:33
Re: [squid-users] Squid for Windows Repeatedly Crashing
On 21/02/19 6:23 pm, Van Order, Drew (US - Hermitage) wrote: > Hello folks, we’re running Squid 3.5.28 on four Windows 2016 VM’s, each > has 4 CPU, 8 GB memory, 10 GB NIC. We implemented Squid to support > forwarding Azure Log Analytics data, it’s all CONNECT. The Squids are > load balanced through a F5. There are less than 1,000 servers sending > data to Log Analytics. > > > > All four Squids are regularly crashing, and I don’t know how to > interpret the errors in cache.log. It crashes if we disable caching too. > ... > > 2019/02/20 09:42:32 kid1| FD 12, 10.5.11.12 [Stopped, reason:Listener > socket closed job1]: (14) Bad address > Something other than Squid closed the network socket Squid was using to receive new client connections (the http_port socket). The only things which should know that socket even exists are Squid and the operating system. This is not an error I've seen before. Is this Squid maybe running in a container or VM which is being hibernated, or suspended, or anything along those lines which may cause the OS filedescriptors to change unexpectedly? Amos ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
[squid-users] Squid for Windows Repeatedly Crashing
Hello folks, we're running Squid 3.5.28 on four Windows 2016 VM's, each has 4 CPU, 8 GB memory, 10 GB NIC. We implemented Squid to support forwarding Azure Log Analytics data, it's all CONNECT. The Squids are load balanced through a F5. There are less than 1,000 servers sending data to Log Analytics. All four Squids are regularly crashing, and I don't know how to interpret the errors in cache.log. It crashes if we disable caching too. Any insight is appreciated-I've inherited this responsibility and more a cloud engineer than a network specialist. Thanks in advance! Typical error sequence in cache.log 2019/02/20 09:42:32 kid1| FD 12, 10.5.11.12 [Stopped, reason:Listener socket closed job1]: (14) Bad address 2019/02/20 09:42:32 kid1| FD 12, 10.5.11.12 [Stopped, reason:Listener socket closed job1]: (14) Bad address 2019/02/20 09:42:32 kid1| FD 12, 10.5.11.12 [Stopped, reason:Listener socket closed job1]: (14) Bad address 2019/02/20 09:42:32 kid1| FD 12, 10.5.11.12 [Stopped, reason:Listener socket closed job1]: (14) Bad address 2019/02/20 09:42:32 kid1| FD 12, 10.5.11.12 [Stopped, reason:Listener socket closed job1]: (14) Bad address 2019/02/20 09:42:32 kid1| FD 12, 10.5.11.12 [Stopped, reason:Listener socket closed job1]: (14) Bad address 2019/02/20 09:42:32 kid1| FD 12, 10.5.11.12 [Stopped, reason:Listener socket closed job1]: (14) Bad address 2019/02/20 09:42:33 kid1| FD 12, 10.5.11.12 [Stopped, reason:Listener socket closed job1]: (14) Bad address 2019/02/20 09:42:33 kid1| FD 12, 10.5.11.12 [Stopped, reason:Listener socket closed job1]: (14) Bad address 2019/02/20 09:42:33 kid1| FD 12, 10.5.11.12 [Stopped, reason:Listener socket closed job1]: (14) Bad address 2019/02/20 09:42:33 kid1| FD 12, 10.5.11.12 [Stopped, reason:Listener socket closed job1]: (14) Bad address 2019/02/20 09:42:33 kid1| FD 12, 10.5.11.12 [Stopped, reason:Listener socket closed job1]: (14) Bad address 2019/02/20 09:42:33 kid1| comm_poll: poll failure: (14) Bad address 2019/02/20 09:42:33 kid1| Select loop Error. Retry 1 2019/02/20 09:42:33 kid1| comm_poll: poll failure: (14) Bad address 2019/02/20 09:42:33 kid1| Select loop Error. Retry 2 2019/02/20 09:42:33 kid1| comm_poll: poll failure: (14) Bad address 2019/02/20 09:42:33 kid1| Select loop Error. Retry 3 2019/02/20 09:42:33 kid1| comm_poll: poll failure: (14) Bad address 2019/02/20 09:42:33 kid1| Select loop Error. Retry 4 2019/02/20 09:42:33 kid1| comm_poll: poll failure: (14) Bad address 2019/02/20 09:42:33 kid1| Select loop Error. Retry 5 2019/02/20 09:42:33 kid1| comm_poll: poll failure: (14) Bad address 2019/02/20 09:42:33 kid1| Select loop Error. Retry 6 2019/02/20 09:42:33 kid1| comm_poll: poll failure: (14) Bad address 2019/02/20 09:42:33 kid1| Select loop Error. Retry 7 2019/02/20 09:42:33 kid1| comm_poll: poll failure: (14) Bad address 2019/02/20 09:42:33 kid1| Select loop Error. Retry 8 2019/02/20 09:42:33 kid1| comm_poll: poll failure: (14) Bad address 2019/02/20 09:42:33 kid1| Select loop Error. Retry 9 2019/02/20 09:42:33 kid1| comm_poll: poll failure: (14) Bad address 2019/02/20 09:42:33 kid1| Select loop Error. Retry 10 2019/02/20 09:42:33 kid1| Closing HTTP port 10.5.11.12:3128 FATAL: Event loop exited with failure. Squid Cache (Version 3.5.28): Terminated abnormally. CPU Usage: 12.640 seconds = 4.234 user + 8.406 sys Maximum Resident Size: 5159680 KB Page faults with physical i/o: 20341 Squid restarts, and will often start erroring right away: 2019/02/20 09:42:33 kid1| storeDirWriteCleanLogs: Starting... 2019/02/20 09:42:33 kid1| Finished. Wrote 0 entries. 2019/02/20 09:42:33 kid1| Took 0.00 seconds ( 0.00 entries/sec). 2019/02/20 09:42:36 kid1| Set Current Directory to /cygdrive/e/squid/var/coredump 2019/02/20 09:42:36 kid1| Starting Squid Cache version 3.5.28 for x86_64-unknown-cygwin... 2019/02/20 09:42:36 kid1| Service Name: squid 2019/02/20 09:42:36 kid1| Process ID 2292 2019/02/20 09:42:36 kid1| Process Roles: worker 2019/02/20 09:42:36 kid1| With 3200 file descriptors available 2019/02/20 09:42:36 kid1| Initializing IP Cache... 2019/02/20 09:42:36 kid1| parseEtcHosts: /etc/hosts: (2) No such file or directory 2019/02/20 09:42:36 kid1| DNS Socket created at [::], FD 5 2019/02/20 09:42:36 kid1| DNS Socket created at 0.0.0.0, FD 6 2019/02/20 09:42:36 kid1| Adding nameserver 208.67.220.220 from squid.conf 2019/02/20 09:42:36 kid1| Adding nameserver 208.67.222.222 from squid.conf 2019/02/20 09:42:36 kid1| Logfile: opening log daemon:/var/log/squid/access.log 2019/02/20 09:42:36 kid1| Logfile Daemon: opening log /var/log/squid/access.log 2019/02/20 09:42:36 kid1| WARNING: no_suid: setuid(0): (22) Invalid argument 2019/02/20 09:42:36 kid1| Store logging disabled 2019/02/20 09:42:36 kid1| Swap maxSize 262144 + 262144 KB, estimated 40329 objects 2019/02/20 09:42:36 kid1| Target number of buckets: 2016 2019/02/20 09:42:36 kid1| Using 8192 Store buckets 2019/02/20 09:42:36 kid1| Max Mem size: 262144 KB 2019/02/20 09:42:36
