While testing the large rock store with trunk revision 13295 I stumbled upon at the following nasty bug:
<cache.log> 2014/03/02 05:33:24 kid1| memCopy: could not find start of [374,32768) in memory. 2014/03/02 05:33:24 kid1| mem_hdr::debugDump: lowest offset: 0 highest offset + 1: 10156823. 2014/03/02 05:33:24 kid1| mem_hdr::debugDump: Current available data is: [0,374) - . 2014/03/02 05:33:24 kid1| Closing HTTP port 10.240.254.52:8081 2014/03/02 05:33:24 kid1| storeDirWriteCleanLogs: Starting... 2014/03/02 05:33:24 kid1| Finished. Wrote 0 entries. 2014/03/02 05:33:24 kid1| Took 0.00 seconds ( 0.00 entries/sec). FATAL: Squid has attempted to read data from memory that is not present. This is an indication of of (pre-3.0) code that hasn't been updated to deal with sparse objects in memory. Squid should coredump.allowing to review the cause. Immediately preceding this message is a dump of the available data in the format [start,end). The [ means from the value, the ) means up to the value. I.e. [1,5) means that there are 4 bytes of data, at offsets 1,2,3,4. Squid Cache (Version 3.HEAD.13295): Terminated abnormally. </cache.log> It doesn't happen when browsing solo, but the moment I redirect 150 mbps of real user traffic Squid crashes immediately and keeps crashing with this same message again and again. I managed to isolate the reason for crashing - cache_mem != 0. Here's my simple test config: <squid.conf> http_port 10.240.254.52:8081 tproxy http_access allow all cache allow all cache_mem 0 MB debug_options ALL,1 coredump_dir /var/log/squid3 shutdown_lifetime 0 cache_dir rock /test 100 max-size=32753 </squid.conf> With this config, Squid works OK. Changing cache_mem to anything different then zero makes Squid crashing . Same config with aufs works OK even with cache_mem enabled. Squid 3.4.3 also works OK with cache_mem != 0, so it has to be a large rock related issue. Unfortunately, no core dumps are available for unknown reason. kill -3 <pid of kid1> generates core file in /var/log/squid3, but the crash - doesn't. Tried running squid -C - didn't help - still no core dump generated during to the crash. Ubuntu 12.04.04 Server 64-bit. Configure options: '--prefix=/usr' '--libexecdir=${prefix}/lib/squid3' '--sysconfdir=/etc/squid3' '--localstatedir=/var' '--sharedstatedir=${localstatedir}/run/squid' '--with-pidfile=${localstatedir}/run/squid3.pid' '--with-logdir=${localstatedir}/log/squid3' '--datadir=${prefix}/share/squid3' '--disable-maintainer-mode' '--disable-dependency-tracking' '--enable-inline' '--enable-storeio=aufs,rock' '--enable-removal-policies=lru,heap' '--disable-delay-pools' '--enable-esi' '--disable-ecap' '--disable-wccp' '--disable-wccpv2' '--enable-snmp' '--enable-eui' '--enable-htcp' '--disable-ssl' '--disable-cache-digests' '--enable-http-violations' '--enable-linux-netfilter' '--disable-follow-x-forwarded-for' '--disable-auth' '--disable-auth-basic' '--disable-auth-ntlm' '--disable-auth-negotiate' '--disable-auth-digest' '--enable-log-daemon-helpers=file' '--enable-external-acl-helpers=none' '--disable-url-rewrite-helpers' '--enable-storeid-rewrite-helpers' '--disable-ipv6' '--enable-zph-qos' '--disable-translation' '--disable-auto-locale' '--with-included-ltdl' '--with-default-user=proxy' '--with-large-files' '--without-krb5-config' '--with-filedescriptors=524288' '--enable-underscores' --enable-ltdl-convenience If you need any further clarification, please let me know. Best, Niki