Re: [squid-users] Reg: Migration of squid cache
On 08/28/2013 11:16 PM, Nehal J Wani wrote: I am using CentOS 6.4 and want to set up a new proxy server using squid. Can anybody help me with the steps to be followed for migrating cache? What do you have now? What do you aim for?? I have compiled new versions of squid for CentOS which is kind of an upgrade to the current squid package that do exist. did you ever used squid before?? Eliezer
Re: [squid-users] Reg: Migration of squid cache
On Thu, Aug 29, 2013 at 2:32 AM, Eliezer Croitoru elie...@ngtech.co.il wrote: On 08/28/2013 11:16 PM, Nehal J Wani wrote: I am using CentOS 6.4 and want to set up a new proxy server using squid. Can anybody help me with the steps to be followed for migrating cache? What do you have now? What do you aim for?? I have compiled new versions of squid for CentOS which is kind of an upgrade to the current squid package that do exist. Oh no. I am using standard packages. Both are squid 3.1.10 did you ever used squid before?? Basically, I have to reconfigure the old server to add new partitions for extra space for logs and cache. Hence. temporarily, I am shifting to a new proxy. Therefore, I need to migrate the proxy cache. Eliezer -- Nehal J Wani UG3, BTech CS+MS(CL) IIIT-Hyderabad http://commandlinewani.blogspot.com
Re: [squid-users] Reg: Migration of squid cache
On 08/29/2013 12:07 AM, Nehal J Wani wrote: On Thu, Aug 29, 2013 at 2:32 AM, Eliezer Croitoru elie...@ngtech.co.il wrote: On 08/28/2013 11:16 PM, Nehal J Wani wrote: I am using CentOS 6.4 and want to set up a new proxy server using squid. Can anybody help me with the steps to be followed for migrating cache? What do you have now? What do you aim for?? I have compiled new versions of squid for CentOS which is kind of an upgrade to the current squid package that do exist. Oh no. I am using standard packages. Both are squid 3.1.10 OK did you ever used squid before?? Basically, I have to reconfigure the old server to add new partitions for extra space for logs and cache. Hence. temporarily, I am shifting to a new proxy. Therefore, I need to migrate the proxy cache. OK so if you do have a new server then it depends on the load of the server but if it's more then a basic server use a dedicated disk for squid logs... for many admins these logs are important so a raid volume\disk will might be better. LVM can ensure you the option to extend the size of the partition in a case of a loaded server. For the cache_dir it's another story since there is a limit for the size by the ram etc. Can you share the current proxy specs and squid.conf?? you can strip down the confidential info and I will try my best to understand and help you if I can.. Eliezer
Re: [squid-users] Reg: Migration of squid cache
Whoah there cowboy, I think hes just asking how to move cache and use with a new squid installation.
Re: [squid-users] Reg: Migration of squid cache
On Thu, Aug 29, 2013 at 3:01 AM, Eliezer Croitoru elie...@ngtech.co.il wrote:
On 08/29/2013 12:07 AM, Nehal J Wani wrote:
On Thu, Aug 29, 2013 at 2:32 AM, Eliezer Croitoru elie...@ngtech.co.il
wrote:
On 08/28/2013 11:16 PM, Nehal J Wani wrote:
I am using CentOS 6.4 and want to set up a new proxy server using
squid. Can anybody help me with the steps to be followed for migrating
cache?
What do you have now?
What do you aim for??
I have compiled new versions of squid for CentOS which is kind of an
upgrade to the current squid package that do exist.
Oh no. I am using standard packages. Both are squid 3.1.10
OK
did you ever used squid before??
Basically, I have to reconfigure the old server to add new partitions
for extra space for logs and cache.
Hence. temporarily, I am shifting to a new proxy. Therefore, I need to
migrate the proxy cache.
OK so if you do have a new server then it depends on the load of the
server but if it's more then a basic server use a dedicated disk for
squid logs...
for many admins these logs are important so a raid volume\disk will
might be better.
LVM can ensure you the option to extend the size of the partition in a
case of a loaded server.
For the cache_dir it's another story since there is a limit for the size
by the ram etc.
Can you share the current proxy specs and squid.conf??
you can strip down the confidential info and I will try my best to
understand and help you if I can..
Eliezer
I haven't migrated the cache completely as of yet. While copying the
cache2 directory from old server,
the RAID5 partition committed suicide. I have attached the output of
fpaste --sysinfo.
--
Nehal J Wani
UG3, BTech CS+MS(CL)
IIIT-Hyderabad
=== fpaste 0.3.7.1 System Information (fpaste --sysinfo) ===
* OS Release (lsb_release -ds):
CentOS release 6.4 (Final)
* Kernel (uname -r ; cat /proc/cmdline):
2.6.32-358.14.1.el6.x86_64
ro root=/dev/mapper/vg_proxy-lv_root rd_LVM_LV=vg_proxy/lv_root
rd_LVM_LV=vg_proxy/lv_swap rd_NO_LUKS rd_NO_MD rd_NO_DM LANG=en_US.UTF-8
SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us crashkernel=129M@0M rhgb
quiet
* Desktop(s) Running (failed: ps -eo comm= | egrep
'(gnome-session|kdeinit|xfce.?-session|fluxbox|blackbox|hackedbox|ratpoison|enlightenment|icewm-session|od-session|wmaker|wmx|openbox-lxde|openbox-gnome-session|openbox-kde-session|mwm|e16|fvwm|xmonad|sugar-session)'
):
N/A
* Desktop(s) Installed (failed: ls -m /usr/share/xsessions/ | sed
's/\.desktop//g' ):
N/A
* SELinux Status (sestatus):
SELinux status: disabled
* SELinux Error Count (failed: selinuxenabled (grep avc: /var/log/messages;
ausearch -m avc -ts today)2/dev/null|egrep -o comm=\[^ ]+|sort|uniq -c|sort
-rn):
N/A
* CPU Model (grep 'model name' /proc/cpuinfo | awk -F: '{print $2}' | uniq -c |
sed -re 's/^ +//' ):
4 Intel(R) Xeon(R) CPU X5450 @ 3.00GHz
* 64-bit Support (grep -q ' lm ' /proc/cpuinfo echo Yes || echo No):
Yes
* Hardware Virtualization Support (egrep -q '(vmx|svm)' /proc/cpuinfo echo
Yes || echo No):
Yes
* Load average (uptime):
03:05:08 up 8:01, 4 users, load average: 1.46, 0.53, 0.37
* Memory usage (free -m):
total used free sharedbuffers cached
Mem: 7872 7635237 0374 4409
-/+ buffers/cache: 2852 5020
Swap:10063 53 10010
* Top 5 CPU hogs (ps axuScnh | awk '$2!=5995' | sort -rnk3 | head -5):
23 2369 88.5 29.6 2489416 2389076 ? Sl Aug28 421:57 squid
0 5850 6.0 0.0 107680 2972 pts/0Ss 03:02 0:07 bash
0 1786 2.7 0.0 64360 556 ?Ss Aug28 13:24 sshd
0 1 0.8 0.0 19360 1000 ?Ss Aug28 4:08 init
0 1049 0.5 0.0 0 0 ?SAug28 2:42
flush-253:3
* Top 5 Memory hogs (ps axuScnh | sort -rnk4 | head -5):
23 2369 88.5 29.6 2489416 2389076 ? Sl Aug28 421:57 squid
0 5995 0.0 0.1 184020 8528 pts/0S+ 03:05 0:00 python
81 1613 0.0 0.0 21536 912 ?Ss Aug28 0:00
dbus-daemon
70 1625 0.0 0.0 27668 152 ?Ss Aug28 0:00
avahi-daemon
70 1624 0.0 0.0 27792 1160 ?SAug28 0:00
avahi-daemon
* Disk space usage (df -hT):
FilesystemTypeSize Used Avail Use% Mounted on
/dev/mapper/vg_proxy-lv_root
ext4 85G 60G 21G 75% /
tmpfstmpfs3.9G 0 3.9G 0% /dev/shm
/dev/sda1 ext4485M 148M 313M 33% /boot
/dev/mapper/vg_proxy-lv_squid1
ext4366G 333G 15G 96% /var/squid/cache1
root@ss1://digilib/data1/proxysquidlogs
fuse.sshfs 1008G 39G 918G 5% /var/log/squid/squidlogs
* Block devices
Re: [squid-users] Reg: Migration of squid cache
On 08/29/2013 12:42 AM, Nehal J Wani wrote: On Thu, Aug 29, 2013 at 3:01 AM, Eliezer Croitoru elie...@ngtech.co.il wrote: On 08/29/2013 12:07 AM, Nehal J Wani wrote: On Thu, Aug 29, 2013 at 2:32 AM, Eliezer Croitoru elie...@ngtech.co.il wrote: On 08/28/2013 11:16 PM, Nehal J Wani wrote: I am using CentOS 6.4 and want to set up a new proxy server using squid. Can anybody help me with the steps to be followed for migrating cache? What do you have now? What do you aim for?? I have compiled new versions of squid for CentOS which is kind of an upgrade to the current squid package that do exist. Oh no. I am using standard packages. Both are squid 3.1.10 OK did you ever used squid before?? Basically, I have to reconfigure the old server to add new partitions for extra space for logs and cache. Hence. temporarily, I am shifting to a new proxy. Therefore, I need to migrate the proxy cache. OK so if you do have a new server then it depends on the load of the server but if it's more then a basic server use a dedicated disk for squid logs... for many admins these logs are important so a raid volume\disk will might be better. LVM can ensure you the option to extend the size of the partition in a case of a loaded server. For the cache_dir it's another story since there is a limit for the size by the ram etc. Can you share the current proxy specs and squid.conf?? you can strip down the confidential info and I will try my best to understand and help you if I can.. Eliezer I haven't migrated the cache completely as of yet. While copying the cache2 directory from old server, the RAID5 partition committed suicide. I have attached the output of fpaste --sysinfo. OK this is pretty understandable.. it seems to me like more then just a small cache but I might be wrong... there is a basic calculation of RAM VS DISK cache. I dont remember now the calculation but I do remeber it's a FAQ. take a peek at: http://wiki.squid-cache.org/SquidFaq/ConfiguringSquid#What_.27.27cache_dir.27.27_size_should_I_use.3F There is the question and calculation of how much RAM you can use.. The CPU is not an issue but squid DB which is residing in RAM takes up space per number of objects. I do Hope that AMOS later on will respond with the right calculation. This is the right time to actually write a small script that will help linux users calculate it. There is a windows squidcalc if i'm right Program that can instruct you how to configure the cache_dir size. By the way take into account that the proxy might have been configured wrongly for a very long period of time and this is indeed a good time to tweak the right settings. Yet I have seen the squid.conf so I was only referring to what I have understood from the plain system info. Eliezer
Re: [squid-users] Reg: Migration of squid cache
Your reply is definitely helpful in re-configuring the new proxy server. But I am more interested in the correct way of migrating/copying cache Will a simple rsync -vertp --partial --progress /var/squid/cache1/* dst do? What are the remaining steps? Will there be any permissions issue? What if those copied files just stay there and not be used? On Thu, Aug 29, 2013 at 3:30 AM, Eliezer Croitoru elie...@ngtech.co.il wrote: On 08/29/2013 12:42 AM, Nehal J Wani wrote: On Thu, Aug 29, 2013 at 3:01 AM, Eliezer Croitoru elie...@ngtech.co.il wrote: On 08/29/2013 12:07 AM, Nehal J Wani wrote: On Thu, Aug 29, 2013 at 2:32 AM, Eliezer Croitoru elie...@ngtech.co.il wrote: On 08/28/2013 11:16 PM, Nehal J Wani wrote: I am using CentOS 6.4 and want to set up a new proxy server using squid. Can anybody help me with the steps to be followed for migrating cache? What do you have now? What do you aim for?? I have compiled new versions of squid for CentOS which is kind of an upgrade to the current squid package that do exist. Oh no. I am using standard packages. Both are squid 3.1.10 OK did you ever used squid before?? Basically, I have to reconfigure the old server to add new partitions for extra space for logs and cache. Hence. temporarily, I am shifting to a new proxy. Therefore, I need to migrate the proxy cache. OK so if you do have a new server then it depends on the load of the server but if it's more then a basic server use a dedicated disk for squid logs... for many admins these logs are important so a raid volume\disk will might be better. LVM can ensure you the option to extend the size of the partition in a case of a loaded server. For the cache_dir it's another story since there is a limit for the size by the ram etc. Can you share the current proxy specs and squid.conf?? you can strip down the confidential info and I will try my best to understand and help you if I can.. Eliezer I haven't migrated the cache completely as of yet. While copying the cache2 directory from old server, the RAID5 partition committed suicide. I have attached the output of fpaste --sysinfo. OK this is pretty understandable.. it seems to me like more then just a small cache but I might be wrong... there is a basic calculation of RAM VS DISK cache. I dont remember now the calculation but I do remeber it's a FAQ. take a peek at: http://wiki.squid-cache.org/SquidFaq/ConfiguringSquid#What_.27.27cache_dir.27.27_size_should_I_use.3F There is the question and calculation of how much RAM you can use.. The CPU is not an issue but squid DB which is residing in RAM takes up space per number of objects. I do Hope that AMOS later on will respond with the right calculation. This is the right time to actually write a small script that will help linux users calculate it. There is a windows squidcalc if i'm right Program that can instruct you how to configure the cache_dir size. By the way take into account that the proxy might have been configured wrongly for a very long period of time and this is indeed a good time to tweak the right settings. Yet I have seen the squid.conf so I was only referring to what I have understood from the plain system info. Eliezer -- Nehal J Wani UG3, BTech CS+MS(CL) IIIT-Hyderabad http://commandlinewani.blogspot.com
Re: [squid-users] Reg: Migration of squid cache
On 29/08/2013 10:26 a.m., Nehal J Wani wrote: Your reply is definitely helpful in re-configuring the new proxy server. But I am more interested in the correct way of migrating/copying cache Will a simple rsync -vertp --partial --progress /var/squid/cache1/* dst do? What are the remaining steps? Will there be any permissions issue? What if those copied files just stay there and not be used? If you are in doubt just copy the cache contents (rsync is fine), erase the swap.state journal, run squid -z. A full rebuild of the index will happen on next startup. ... the rest of this is is about trouble you may get into by doing it that way ... NOTE first that the cache is just a cache. You can rebuild it from nothing just by running any Squid with an empty cache_dir. If you have a very large cache now this is in fact probably the best way to go. You will have a few days with low HIT rate as it fills up again but the growth curve back to normal HITs is exponential and client traffic is never bogged down completely. If you want to go ahead with the upgrade on a huge cache and/or if it needs to re-scan while in production, be aware that it means some hours/days depending on size where you will have 0 HIT rate anyway. You an do the startup and resulting rebuild offline in a proxy with dummy config file pointing at the cache_dir, while the old one (or dummy with no cache) continues to run. This will result in cache which is some time out of data and have a burst of revalidations and removals when swapping the cache into production use. The growth back to normal HIT rate starts further along teh growth curve and is exponential, but slightly slower than for an empty cache. Squid (all versions) will check the swap.state file and load it normally if possible. If there is any problems Squid will re-scan the entire cache_dir and rebuild it from the disk contents (see above). Be aware there is a swap.state corruption bug fixed and format upgrade happened in 3.2 so Squid-3.3 will most likely opt to re-scan regardless of what you do. Amos
