Re: [squid-users] a lot of TCP_SWAPFAIL_MISS/200
Hallo, Dan, Du meintest am 10.08.15 zum Thema Re: [squid-users] a lot of TCP_SWAPFAIL_MISS/200: Native English speaker here -- though I'm not HackXBack, just see that he hasn't replied to you yet. [...] I don't know why people don't give the bug higher priority as it is significantly reducing the hit ratio HTH :) Thank you so much! That longer text I can read, and I can understand it! Viele Gruesse! Helmut ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] a lot of TCP_SWAPFAIL_MISS/200
Quoting Helmut Hullen hul...@t-online.de: Hallo, HackXBack, Du meintest am 07.08.15: yea joe i dont know why ppl dnt give this bug importance while it deduce alot of hit ratio Can you please translate this kind of pidgin english into usual written english, to please all foreign readers who have only learned this written english? Thank you! Please excuse my gerlish. Viele Gruesse! Helmut Hi Helmut, Native English speaker here -- though I'm not HackXBack, just see that he hasn't replied to you yet. Yeah Joe, I don't know why people don't give the bug importance while it deduce[sic] a lot of hit ratio Here's the same phrase worded the way I think that HaxkXBack /meant/ -- Yeah Joe, I don't know why people don't give the bug higher priority as it is significantly reducing the hit ratio HTH :) -Dan smime.p7s Description: S/MIME Signature ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] a lot of TCP_SWAPFAIL_MISS/200
On 8/08/2015 10:38 a.m., HackXBack wrote: yea joe i dont know why ppl dnt give this bug importance while it deduce a lot of hit ratio yea. um ... isn't that your job ? Answering the question of: Why cant Squid read this file off disk? sure sounds like something the admin in charge is responsible for finding out. The ppl working on Squid are consultants, fixing first the bugs our customers pay to get fixed ASAP. And/or hobbyists, in our spare time fixing the ones we want to fix, just for fun (...F ...U ...N). Anyhow. No less than three of us main developers and a few of the downstream distro guys have already spent a fair amount of time (paid and unpaid) fixing the bugs inside Squid we identified as even _possibly_ leading to SWAPFAIL. In the end it might simply be your HDD dying. Nothing *we* can do about that. You or the init scripts could be sending Squid KILL signals instead of shutdown (SIGHUP) signals. Also something for you to fix, not us. IIRC HackXBack is dealing with huge caches and forced short shutdown times. That is one recipe for SWAPFAIL, for known reasons. And is being worked on by hobbyists in spare time because despite many complaints *nobody with money has ever hired anyone to fix that group of failures*. BUT. It is still unknown if that is actually HackXback's problem, several bug fixes in the latest Squid so far have apparently made no dent it. What can be done by others IS being done by others. In free software everyone plays a part. Especially you (for all values of 'you' reading this). So we need you to provide details, lots of details, about what exactly is happening and answer the question of 'why your Squid can't load the files off disk?' Maybe you solve it yourself in the process of investigating. (yay!) No joke. This has happened repeatedly in the past already. Bad HDD firmware, broken FS system drivers, dying HDD, force-kill init scripts, bad NFS network transfers, even a bad Intranet router firmware once. All found and fixed by admin looking into SWAPFAIL. Squid is usually just the messenger with SWAPFAIL. Whatever happens the answer and research sheds light on the problem, and only then it can possibly be fixed somehow. Without that answer we are just stabbing about blindly in the dark trying to swat a flying bug. Cheers Amos ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] a lot of TCP_SWAPFAIL_MISS/200
Hallo, HackXBack, Du meintest am 07.08.15: yea joe i dont know why ppl dnt give this bug importance while it deduce alot of hit ratio Can you please translate this kind of pidgin english into usual written english, to please all foreign readers who have only learned this written english? Thank you! Please excuse my gerlish. Viele Gruesse! Helmut ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] a lot of TCP_SWAPFAIL_MISS/200
yea joe i dont know why ppl dnt give this bug importance while it deduce a lot of hit ratio -- View this message in context: http://squid-web-proxy-cache.1019090.n4.nabble.com/a-lot-of-TCP-SWAPFAIL-MISS-200-tp4672011p4672636.html Sent from the Squid - Users mailing list archive at Nabble.com. ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] a lot of TCP_SWAPFAIL_MISS/200
but this happen only with version 3.5 , and it increase after restarting squid or rebooting system this is bug in 3.5 and it decrease the HIT ratio , you dont think so ? -- View this message in context: http://squid-web-proxy-cache.1019090.n4.nabble.com/a-lot-of-TCP-SWAPFAIL-MISS-200-tp4672011p4672311.html Sent from the Squid - Users mailing list archive at Nabble.com. ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] a lot of TCP_SWAPFAIL_MISS/200
yes am using AUFS cache_dir directive -- View this message in context: http://squid-web-proxy-cache.1019090.n4.nabble.com/a-lot-of-TCP-SWAPFAIL-MISS-200-tp4672011p4672316.html Sent from the Squid - Users mailing list archive at Nabble.com. ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] a lot of TCP_SWAPFAIL_MISS/200
On 19/07/2015 13:23, HackXBack wrote: yes am using AUFS cache_dir directive With how many workers? Eliezer ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] a lot of TCP_SWAPFAIL_MISS/200
top shows only 1 worker for squid .. top - 16:24:51 up 5 days, 3:22, 2 users, load average: 2.06, 1.18, 0.82 Tasks: 158 total, 2 running, 156 sleeping, 0 stopped, 0 zombie %Cpu(s): 1.7 us, 0.7 sy, 0.0 ni, 96.7 id, 0.3 wa, 0.0 hi, 0.6 si, 0.0 st KiB Mem: 32928480 total, 29039108 used, 3889372 free, 4273996 buffers KiB Swap: 9526268 total,0 used, 9526268 free, 10857212 cached PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND 25178 proxy 20 0 8607m 8.2g 6912 R 17.9 26.1 2:29.69 squid 9187 unbound 20 0 72860 24m 1208 S 0.3 0.1 1:00.63 unbound 25247 root 20 0 000 S 0.3 0.0 0:00.04 kworker/0:2 25587 root 20 0 23320 1704 1180 R 0.3 0.0 0:00.05 top 1 root 20 0 10648 760 624 S 0.0 0.0 0:03.79 init 2 root 20 0 000 S 0.0 0.0 0:00.04 kthreadd 3 root 20 0 000 S 0.0 0.0 2:25.37 ksoftirqd/0 5 root 20 0 000 S 0.0 0.0 0:00.00 kworker/u:0 6 root rt 0 000 S 0.0 0.0 0:01.97 migration/0 7 root rt 0 000 S 0.0 0.0 0:03.82 watchdog/0 8 root rt 0 000 S 0.0 0.0 0:00.01 migration/1 10 root 20 0 000 S 0.0 0.0 0:05.54 ksoftirqd/1 12 root rt 0 000 S 0.0 0.0 0:00.69 watchdog/1 13 root rt 0 000 S 0.0 0.0 0:00.00 migration/2 15 root 20 0 000 S 0.0 0.0 0:02.05 ksoftirqd/2 16 root rt 0 000 S 0.0 0.0 0:00.63 watchdog/2 17 root rt 0 000 S 0.0 0.0 0:00.00 migration/3 19 root 20 0 000 S 0.0 0.0 0:01.93 ksoftirqd/3 20 root rt 0 000 S 0.0 0.0 0:00.61 watchdog/3 21 root rt 0 000 S 0.0 0.0 0:00.10 migration/4 23 root 20 0 000 S 0.0 0.0 0:00.29 ksoftirqd/4 24 root rt 0 000 S 0.0 0.0 0:00.93 watchdog/4 25 root rt 0 000 S 0.0 0.0 0:00.10 migration/5 27 root 20 0 000 S 0.0 0.0 0:00.22 ksoftirqd/5 28 root rt 0 000 S 0.0 0.0 0:00.96 watchdog/5 29 root rt 0 000 S 0.0 0.0 0:00.10 migration/6 31 root 20 0 000 S 0.0 0.0 0:00.22 ksoftirqd/6 32 root rt 0 000 S 0.0 0.0 0:00.80 watchdog/6 33 root rt 0 000 S 0.0 0.0 0:00.11 migration/7 35 root 20 0 000 S 0.0 0.0 0:00.22 ksoftirqd/7 36 root rt 0 000 S 0.0 0.0 0:00.79 watchdog/7 37 root 0 -20 000 S 0.0 0.0 0:00.00 cpuset 38 root 0 -20 000 S 0.0 0.0 0:00.00 khelper 39 root 20 0 000 S 0.0 0.0 0:00.00 kdevtmpfs 40 root 0 -20 000 S 0.0 0.0 0:00.00 netns 41 root 20 0 000 S 0.0 0.0 6:50.08 sync_supers 42 root 20 0 000 S 0.0 0.0 0:00.01 bdi-default 43 root 0 -20 000 S 0.0 0.0 0:00.00 kintegrityd -- View this message in context: http://squid-web-proxy-cache.1019090.n4.nabble.com/a-lot-of-TCP-SWAPFAIL-MISS-200-tp4672011p4672320.html Sent from the Squid - Users mailing list archive at Nabble.com. ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] a lot of TCP_SWAPFAIL_MISS/200
On 19/07/2015 10:09 p.m., HackXBack wrote:
but this happen only with version 3.5 , and it increase after restarting
squid or rebooting system
this is bug in 3.5 and it decrease the HIT ratio ,
you dont think so ?
I've seen a lot of reports of it happening on all Squid versions where
shutdown did not properly finished saving to those cache files. Usually
gets bad with very short configured shutdown_lifetime values. Although
it can happen anytime shutdown interrupts a busy set of clients.
FYI: attached is a backport of a Squid-4 change which should reduce that
shutdown situation being a problem. If you would like to test we can see
if it helps with your case.
SWAPFAIL also gets seen a lot on caches (or individual files) that were
created by very old Squid versions (older than 3.1 or 3.3 IIRC) before
we added checksum validation. Those old objects metadata fail the
checksum due to format differences at the bit level and SWAPFAIL is done
to replace them with guaranteed clean content.
The same checksum fails can also still happen on objects created with a
32-bit Squid loaded by a 64-bit Squid, and vice-versa. For the same reasons.
Then there is genuine HDD corruption, mutiple Squid processes or workers
touching each others cache files, etc. other programs touching them etc.
So many causes it is guesswork. But an educated guess says it is
shutdown related errors.
Amos
=== modified file 'src/base/RunnersRegistry.cc'
--- src/base/RunnersRegistry.cc 2015-01-13 09:13:49 +
+++ src/base/RunnersRegistry.cc 2015-07-19 15:08:16 +
@@ -15,40 +15,48 @@
/// all known runners
static Runners *TheRunners = NULL;
/// safely returns registered runners, initializing structures as needed
static Runners
GetRunners()
{
if (!TheRunners)
TheRunners = new Runners;
return *TheRunners;
}
int
RegisterRunner(RegisteredRunner *rr)
{
Runners runners = GetRunners();
runners.insert(rr);
return runners.size();
}
+int
+DeregisterRunner(RegisteredRunner *rr)
+{
+Runners runners = GetRunners();
+runners.erase(rr);
+return runners.size();
+}
+
void
RunRegistered(const RegisteredRunner::Method m)
{
Runners runners = GetRunners();
typedef Runners::iterator RRI;
for (RRI i = runners.begin(); i != runners.end(); ++i)
((*i)-*m)();
if (m == RegisteredRunner::finishShutdown) {
delete TheRunners;
TheRunners = NULL;
}
}
bool
UseThisStatic(const void *)
{
return true;
}
=== modified file 'src/base/RunnersRegistry.h'
--- src/base/RunnersRegistry.h 2015-01-13 09:13:49 +
+++ src/base/RunnersRegistry.h 2015-07-19 15:08:16 +
@@ -51,54 +51,63 @@
virtual void claimMemoryNeeds() {}
/// Called after claimMemoryNeeds().
/// Meant for activating modules and features using a finalized
/// configuration with known memory requirements.
virtual void useConfig() {}
/* Reconfiguration events */
/// Called after parsing squid.conf during reconfiguration.
/// Meant for adjusting the module state based on configuration changes.
virtual void syncConfig() {}
/* Shutdown events */
/// Called after receiving a shutdown request and before stopping the main
/// loop. At least one main loop iteration is guaranteed after this call.
/// Meant for cleanup and state saving that may require other modules.
virtual void startShutdown() {}
+/// Called after shutdown_lifetime grace period ends and before stopping
+/// the main loop. At least one main loop iteration is guaranteed after
+/// this call.
+/// Meant for cleanup and state saving that may require other modules.
+virtual void endingShutdown() {}
+
/// Called after stopping the main loop and before releasing memory.
/// Meant for quick/basic cleanup that does not require any other modules.
virtual ~RegisteredRunner() {}
/// exists to simplify caller interface; override the destructor instead
void finishShutdown() { delete this; }
/// a pointer to one of the above notification methods
typedef void (RegisteredRunner::*Method)();
};
/// registers a given runner with the given registry and returns registry count
int RegisterRunner(RegisteredRunner *rr);
+/// de-registers a given runner with the given registry and returns registry
count
+int DeregisterRunner(RegisteredRunner *rr);
+
/// Calls a given method of all runners.
/// All runners are destroyed after the finishShutdown() call.
void RunRegistered(const RegisteredRunner::Method m);
/// convenience macro to describe/debug the caller and the method being called
#define RunRegisteredHere(m) \
debugs(1, 2, running # m); \
RunRegistered(m)
/// convenience function to use an otherwise unreferenced static variable
bool UseThisStatic(const void *);
/// convenience macro: register one RegisteredRunner kid as early as possible
#define
Re: [squid-users] a lot of TCP_SWAPFAIL_MISS/200
On 2/07/2015 11:31 a.m., HackXBack wrote: after upgrading to 3.5.5 i see in cache.log 2015/07/02 01:51:51 kid1| DiskThreadsDiskFile::openDone: (2) No such file or directory 2015/07/02 01:51:51 kid1| /cache01/2/16/AA/0016AA3B - ORIGINAL_DST/203.77.186.75 video/mp4 access.log TCP_SWAPFAIL_MISS/200 Your cache index (from swap.state) does not match what objects actually exist on disk. This is Squid auto-recovery fetching a new copy from the network. Amos ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
[squid-users] a lot of TCP_SWAPFAIL_MISS/200
after upgrading to 3.5.5 i see in cache.log 2015/07/02 01:51:51 kid1| DiskThreadsDiskFile::openDone: (2) No such file or directory 2015/07/02 01:51:51 kid1| /cache01/2/16/AA/0016AA3B - ORIGINAL_DST/203.77.186.75 video/mp4 access.log TCP_SWAPFAIL_MISS/200 -- View this message in context: http://squid-web-proxy-cache.1019090.n4.nabble.com/a-lot-of-TCP-SWAPFAIL-MISS-200-tp4672011.html Sent from the Squid - Users mailing list archive at Nabble.com. ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
