Re: [squid-users] Yet another store_id question HIT MISS
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 21.08.2016 19:23, Eliezer Croitoru пишет: > Well it's pretty simple, I have used squd 4.0.13 with default settings and > the rules you provided ie: > squid.conf: > store_id_program /usr/lib64/squid/storeid_file_rewrite /etc/squid/storeid-rules > store_id_children 40 startup=10 idle=5 concurrency=0 > store_id_access allow GET > store_id_access deny all > > /etc/squid/storeid-rules: > ^http:\/\/igcdn-.*\.akamaihd\.net/hphotos-ak-.*/(t5.*?)(?:\?|$) http://instagramcdn.squid.internal/$1 > > But I am not sure how you decided on the StoreID structure. I am pretty sure you can replace the ".*" with a much more accurate rule such as in: > ^http:\/\/igcdn-(photos|videos)-([a-z]+)-a\.akamaihd\.net/hphotos-ak-([a-z0-9]+)/(t5[\.\-\_\/a-z0-9]+?)(?:\?|$) http://instagramcdn.squid.internal/$4 I suggests here is required efficiency agains accuracy. ".*" in some cases requires less steps. > > > Eliezer > > > Eliezer Croitoru > Linux System Administrator > Mobile: +972-5-28704261 > Email: elie...@ngtech.co.il > > > -Original Message- > From: squid-users [mailto:squid-users-boun...@lists.squid-cache.org] On Behalf Of Omid Kosari > Sent: Saturday, August 20, 2016 4:41 PM > To: squid-users@lists.squid-cache.org > Subject: Re: [squid-users] Yet another store_id question HIT MISS > > I have also tested with several browsers, PCs etc . Also i have disabled > every refresh pattern except default squid rules as like as yours . Same > result . > > The only way to get hits are the way i mentioned in > http://squid-web-proxy-cache.1019090.n4.nabble.com/Yet-another-store-id-question-HIT-MISS-tp4678972p4679025.html > > > Do you tested when enabled my storeid config (mentioned in my first email ) > was there ? > > > > -- > View this message in context: http://squid-web-proxy-cache.1019090.n4.nabble.com/Yet-another-store-id-question-HIT-MISS-tp4678972p4679067.html > Sent from the Squid - Users mailing list archive at Nabble.com. > ___ > squid-users mailing list > squid-users@lists.squid-cache.org > http://lists.squid-cache.org/listinfo/squid-users > > ___ > squid-users mailing list > squid-users@lists.squid-cache.org > http://lists.squid-cache.org/listinfo/squid-users -BEGIN PGP SIGNATURE- Version: GnuPG v2 iQEcBAEBCAAGBQJXudvGAAoJENNXIZxhPexGO8EH/2aZEapcMeO3q4MOeP2/hVz4 /8hc3xwjai72Uk0hQ2LWt/jB34GEs7FUPYi3i51nKpI6+qYHcvDbGwTGn6DXdTdV 8VSR/NIyrDTNSBKoZDNvkyWDm45H4W0HYwRRrG0XMVagyZpDFj8JzNwt6iXvlJdY mSHkxP5wslbhpXG/+cD4nqKEjLGBvSjr+7Jp4/py4D1KXd8BFsg1wh78wwCiAUUp WnCVsdvsHIW3TDUNYQuQE8d4ONktklm6aRo62McYr3VGnb4w0dI5IpnXKnDOzfUP 7rIq0kfykWkQSn/MN5ibz/AInrxyCe7Tp7TFiufbr+aFPFbltmA6obBW+qRxU24= =dxxA -END PGP SIGNATURE- 0x613DEC46.asc Description: application/pgp-keys ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] Yet another store_id question HIT MISS
Well it's pretty simple, I have used squd 4.0.13 with default settings and the rules you provided ie: squid.conf: store_id_program /usr/lib64/squid/storeid_file_rewrite /etc/squid/storeid-rules store_id_children 40 startup=10 idle=5 concurrency=0 store_id_access allow GET store_id_access deny all /etc/squid/storeid-rules: ^http:\/\/igcdn-.*\.akamaihd\.net/hphotos-ak-.*/(t5.*?)(?:\?|$) http://instagramcdn.squid.internal/$1 But I am not sure how you decided on the StoreID structure. I am pretty sure you can replace the ".*" with a much more accurate rule such as in: ^http:\/\/igcdn-(photos|videos)-([a-z]+)-a\.akamaihd\.net/hphotos-ak-([a-z0-9]+)/(t5[\.\-\_\/a-z0-9]+?)(?:\?|$) http://instagramcdn.squid.internal/$4 Eliezer Eliezer Croitoru Linux System Administrator Mobile: +972-5-28704261 Email: elie...@ngtech.co.il -Original Message- From: squid-users [mailto:squid-users-boun...@lists.squid-cache.org] On Behalf Of Omid Kosari Sent: Saturday, August 20, 2016 4:41 PM To: squid-users@lists.squid-cache.org Subject: Re: [squid-users] Yet another store_id question HIT MISS I have also tested with several browsers, PCs etc . Also i have disabled every refresh pattern except default squid rules as like as yours . Same result . The only way to get hits are the way i mentioned in http://squid-web-proxy-cache.1019090.n4.nabble.com/Yet-another-store-id-question-HIT-MISS-tp4678972p4679025.html Do you tested when enabled my storeid config (mentioned in my first email ) was there ? -- View this message in context: http://squid-web-proxy-cache.1019090.n4.nabble.com/Yet-another-store-id-question-HIT-MISS-tp4678972p4679067.html Sent from the Squid - Users mailing list archive at Nabble.com. ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] Yet another store_id question HIT MISS
On 21/08/2016 2:10 a.m., Omid Kosari wrote:
> Amos Jeffries wrote
>> On 19/08/2016 2:15 a.m., Omid Kosari wrote:
>>> I was correct .
>>>
>>> If one of following conditions happens then the mentioned urls will not
>>> cache .
>>>
>>> 1-in squid.conf have this line
>>> acl storeiddomainregex dstdom_regex
>>> ^igcdn(\-photos|\-videos)[a-z0-9\-]{0,9}\.akamaihd\.net$
>>>
>>
>> How you use that ACL in your access rule logics is just as important as
>> what it contains / matches.
>
> In my first email i have pasted other parts of config .
>
>
>
> Amos Jeffries wrote
>>> 2-in storeid_db have this line
>>> ^http:\/\/igcdn-.*\.akamaihd\.net/hphotos-ak-.*/(t5.*?)(?:\?|$)
>>> http://instagramcdn.squid.internal/$1
>>
>>
>> The pattern is malformed. Several parts of it are impossible to match,
>> and just make it look complicated when its not.
>>
>> It is better written as:
>>
>> ^http:\/\/igcdn-[a-z0-9\-\.]+\.akamaihd\.net/hphotos-ak-[a-z0-9]+/(t5.*)
>> http://instagramcdn.squid.internal/$2
>
> Thanks for suggest but this one seems incorrect . At least i think the $2
> should be $1 . Apart from that i will use yours .
>
Right - should be $1. Sorry, thats leftover from my own first attempt to
clean it up quickly.
Amos
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] Yet another store_id question HIT MISS
Amos Jeffries wrote
> On 19/08/2016 2:15 a.m., Omid Kosari wrote:
>> I was correct .
>>
>> If one of following conditions happens then the mentioned urls will not
>> cache .
>>
>> 1-in squid.conf have this line
>> acl storeiddomainregex dstdom_regex
>> ^igcdn(\-photos|\-videos)[a-z0-9\-]{0,9}\.akamaihd\.net$
>>
>
> How you use that ACL in your access rule logics is just as important as
> what it contains / matches.
In my first email i have pasted other parts of config .
Amos Jeffries wrote
>> 2-in storeid_db have this line
>> ^http:\/\/igcdn-.*\.akamaihd\.net/hphotos-ak-.*/(t5.*?)(?:\?|$)
>> http://instagramcdn.squid.internal/$1
>
>
> The pattern is malformed. Several parts of it are impossible to match,
> and just make it look complicated when its not.
>
> It is better written as:
>
> ^http:\/\/igcdn-[a-z0-9\-\.]+\.akamaihd\.net/hphotos-ak-[a-z0-9]+/(t5.*)
> http://instagramcdn.squid.internal/$2
Thanks for suggest but this one seems incorrect . At least i think the $2
should be $1 . Apart from that i will use yours .
Amos Jeffries wrote
>>
>> Now the problem is with my mentioned squid.conf and store_id rules , the
>> mentioned urls will not cache at all.Even if same url reopens many times
>> .
>>
>
> Store-ID does not force caching. All it does is provide Squid with a
> different ID under which the item would be stored *IF* (and only if) it
> would be stored normally.
>
> I suggest removing the "ignore-no-store ignore-private" from your
> .squid.internal refresh_pattern. They trigger actions that break the
> image objects you actually want to store at that Store-ID location.
>
> Amos
I have completely deleted that refresh_pattern line . But still same results
.
--
View this message in context:
http://squid-web-proxy-cache.1019090.n4.nabble.com/Yet-another-store-id-question-HIT-MISS-tp4678972p4679068.html
Sent from the Squid - Users mailing list archive at Nabble.com.
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] Yet another store_id question HIT MISS
I have also tested with several browsers, PCs etc . Also i have disabled every refresh pattern except default squid rules as like as yours . Same result . The only way to get hits are the way i mentioned in http://squid-web-proxy-cache.1019090.n4.nabble.com/Yet-another-store-id-question-HIT-MISS-tp4678972p4679025.html Do you tested when enabled my storeid config (mentioned in my first email ) was there ? -- View this message in context: http://squid-web-proxy-cache.1019090.n4.nabble.com/Yet-another-store-id-question-HIT-MISS-tp4678972p4679067.html Sent from the Squid - Users mailing list archive at Nabble.com. ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] Yet another store_id question HIT MISS
On 19/08/2016 2:15 a.m., Omid Kosari wrote:
> I was correct .
>
> If one of following conditions happens then the mentioned urls will not
> cache .
>
> 1-in squid.conf have this line
> acl storeiddomainregex dstdom_regex
> ^igcdn(\-photos|\-videos)[a-z0-9\-]{0,9}\.akamaihd\.net$
>
How you use that ACL in your access rule logics is just as important as
what it contains / matches.
> 2-in storeid_db have this line
> ^http:\/\/igcdn-.*\.akamaihd\.net/hphotos-ak-.*/(t5.*?)(?:\?|$)
> http://instagramcdn.squid.internal/$1
The pattern is malformed. Several parts of it are impossible to match,
and just make it look complicated when its not.
It is better written as:
^http:\/\/igcdn-[a-z0-9\-\.]+\.akamaihd\.net/hphotos-ak-[a-z0-9]+/(t5.*)
http://instagramcdn.squid.internal/$2
>
> Now the problem is with my mentioned squid.conf and store_id rules , the
> mentioned urls will not cache at all.Even if same url reopens many times .
>
Store-ID does not force caching. All it does is provide Squid with a
different ID under which the item would be stored *IF* (and only if) it
would be stored normally.
I suggest removing the "ignore-no-store ignore-private" from your
.squid.internal refresh_pattern. They trigger actions that break the
image objects you actually want to store at that Store-ID location.
Amos
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] Yet another store_id question HIT MISS
Well it works for me locally:
1471532262.544 3 192.168.10.131 TCP_MEM_HIT/200 72214 GET
http://igcdn-photos-a-a.akamaihd.net/hphotos-ak-xap1/t51.2885-15/s640x640/sh0.08/e35/13702999_1008425479275495_76276919_n.jpg
- HIER_NONE/- image/jpeg
1471532274.903 9 192.168.10.131 TCP_MEM_HIT/200 72215 GET
http://igcdn-photos-h-a.akamaihd.net/hphotos-ak-xap1/t51.2885-15/s640x640/sh0.08/e35/13702999_1008425479275495_76276919_n.jpg
- HIER_NONE/- image/jpeg
Try to open a "private" or incognito window and see what happens.
Also try to download the file using wget or curl via the proxy.
See what happens..
I am using the default refresh_pattern rules of squid ie :
refresh_pattern ^ftp: 144020% 10080
refresh_pattern ^gopher:14400% 1440
refresh_pattern -i (/cgi-bin/|\?) 0 0% 0
refresh_pattern . 0 20% 4320
Eliezer
Eliezer Croitoru
Linux System Administrator
Mobile: +972-5-28704261
Email: elie...@ngtech.co.il
-Original Message-
From: squid-users [mailto:squid-users-boun...@lists.squid-cache.org] On Behalf
Of Omid Kosari
Sent: Thursday, August 18, 2016 5:15 PM
To: squid-users@lists.squid-cache.org
Subject: Re: [squid-users] Yet another store_id question HIT MISS
I was correct .
If one of following conditions happens then the mentioned urls will not
cache .
1-in squid.conf have this line
acl storeiddomainregex dstdom_regex
^igcdn(\-photos|\-videos)[a-z0-9\-]{0,9}\.akamaihd\.net$
2-in storeid_db have this line
^http:\/\/igcdn-.*\.akamaihd\.net/hphotos-ak-.*/(t5.*?)(?:\?|$)
http://instagramcdn.squid.internal/$1
IF 1 OR 2 THEN
http://igcdn-photos-h-a.akamaihd.net/hphotos-ak-xap1/t51.2885-15/s640x640/sh0.08/e35/13702999_1008425479275495_76276919_n.jpg
will not cache at all even if we open that url for many times .
But if i remove 1 and 2 then the url will be cached .
My first email was incorrect because i realized that first url is hit but
second miss . The hit was from before store_id rules added .
Now the problem is with my mentioned squid.conf and store_id rules , the
mentioned urls will not cache at all.Even if same url reopens many times .
Thanks
--
View this message in context:
http://squid-web-proxy-cache.1019090.n4.nabble.com/Yet-another-store-id-question-HIT-MISS-tp4678972p4679025.html
Sent from the Squid - Users mailing list archive at Nabble.com.
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] Yet another store_id question HIT MISS
I was correct .
If one of following conditions happens then the mentioned urls will not
cache .
1-in squid.conf have this line
acl storeiddomainregex dstdom_regex
^igcdn(\-photos|\-videos)[a-z0-9\-]{0,9}\.akamaihd\.net$
2-in storeid_db have this line
^http:\/\/igcdn-.*\.akamaihd\.net/hphotos-ak-.*/(t5.*?)(?:\?|$)
http://instagramcdn.squid.internal/$1
IF 1 OR 2 THEN
http://igcdn-photos-h-a.akamaihd.net/hphotos-ak-xap1/t51.2885-15/s640x640/sh0.08/e35/13702999_1008425479275495_76276919_n.jpg
will not cache at all even if we open that url for many times .
But if i remove 1 and 2 then the url will be cached .
My first email was incorrect because i realized that first url is hit but
second miss . The hit was from before store_id rules added .
Now the problem is with my mentioned squid.conf and store_id rules , the
mentioned urls will not cache at all.Even if same url reopens many times .
Thanks
--
View this message in context:
http://squid-web-proxy-cache.1019090.n4.nabble.com/Yet-another-store-id-question-HIT-MISS-tp4678972p4679025.html
Sent from the Squid - Users mailing list archive at Nabble.com.
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] Yet another store_id question HIT MISS
Simply open following url in firefox http://igcdn-photos-h-a.akamaihd.net/hphotos-ak-xap1/t51.2885-15/s640x640/sh0.08/e35/13702999_1008425479275495_76276919_n.jpg then rename h to a,b,c,d,e,f for example http://igcdn-photos-a-a.akamaihd.net/hphotos-ak-xap1/t51.2885-15/s640x640/sh0.08/e35/13702999_1008425479275495_76276919_n.jpg According to my store_id rules it should be hit but it is not . Even i am uncertain about if we open same exact url 2 times , is it hit in second time ? -- View this message in context: http://squid-web-proxy-cache.1019090.n4.nabble.com/Yet-another-store-id-question-HIT-MISS-tp4678972p4679021.html Sent from the Squid - Users mailing list archive at Nabble.com. ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] Yet another store_id question HIT MISS
Hey Omid, I will try to test here on my local squid to see if there is a well understood reason for the MISS. What have you tried until now to test the issue? Did you tried to reproduce from your PC? Eliezer Eliezer Croitoru Linux System Administrator Mobile: +972-5-28704261 Email: elie...@ngtech.co.il -Original Message- From: squid-users [mailto:squid-users-boun...@lists.squid-cache.org] On Behalf Of Omid Kosari Sent: Wednesday, August 17, 2016 5:36 PM To: squid-users@lists.squid-cache.org Subject: Re: [squid-users] Yet another store_id question HIT MISS Amos Jeffries wrote > On 18/08/2016 1:43 a.m., Omid Kosari wrote: >> Why following link is HIT >> >> X-Cache:"HIT from cache1" >> X-Cache-Lookup:"HIT from cache1:3128" >> >> >> http://igcdn-photos-c-a.akamaihd.net/hphotos-ak-xaf1/t51.2885-15/s150 >> x150/e35/13649137_1547514802224163_950421795_n.jpg >> >> but this one is MISS >> >> http://igcdn-photos-a-a.akamaihd.net/hphotos-ak-xaf1/t51.2885-15/s150 >> x150/e35/13649137_1547514802224163_950421795_n.jpg >> > > Because your override-expire tells Squid to ignore the > Cache-Control:max-age expiry details - instead to treat anything 10080 > seconds old as expired. > > Then the override-last-mod tells Squid to ignore the Last-Modified > header, instead to treat everything arriving as being 10080 seconds old. > > When combined they mean that objects are only cacheable for the > remainder of the first second after arrival. When that 0-1 second is > over they are identified as being stale and require revalidation to be > used. > > Your options also tell Squid to ignore revalidation ... > You see where this is going? > > > It is most often a bad idea to override cache controls coming from the > server. refresh_pattern is a way to work around broken services or > those which do not send sufficient cache-controls to cache properly. > Use with great care and caution. > > Amos > > ___ > squid-users mailing list > squid-users@.squid-cache > http://lists.squid-cache.org/listinfo/squid-users Thanks . i have completely removed that refresh_pattern line but still same result . -- View this message in context: http://squid-web-proxy-cache.1019090.n4.nabble.com/Yet-another-store-id-question-HIT-MISS-tp4678972p4678983.html Sent from the Squid - Users mailing list archive at Nabble.com. ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] Yet another store_id question HIT MISS
Amos Jeffries wrote > On 18/08/2016 1:43 a.m., Omid Kosari wrote: >> Why following link is HIT >> >> X-Cache:"HIT from cache1" >> X-Cache-Lookup:"HIT from cache1:3128" >> >> >> http://igcdn-photos-c-a.akamaihd.net/hphotos-ak-xaf1/t51.2885-15/s150x150/e35/13649137_1547514802224163_950421795_n.jpg >> >> but this one is MISS >> >> http://igcdn-photos-a-a.akamaihd.net/hphotos-ak-xaf1/t51.2885-15/s150x150/e35/13649137_1547514802224163_950421795_n.jpg >> > > Because your override-expire tells Squid to ignore the > Cache-Control:max-age expiry details - instead to treat anything 10080 > seconds old as expired. > > Then the override-last-mod tells Squid to ignore the Last-Modified > header, instead to treat everything arriving as being 10080 seconds old. > > When combined they mean that objects are only cacheable for the > remainder of the first second after arrival. When that 0-1 second is > over they are identified as being stale and require revalidation to be > used. > > Your options also tell Squid to ignore revalidation ... > You see where this is going? > > > It is most often a bad idea to override cache controls coming from the > server. refresh_pattern is a way to work around broken services or those > which do not send sufficient cache-controls to cache properly. Use with > great care and caution. > > Amos > > ___ > squid-users mailing list > squid-users@.squid-cache > http://lists.squid-cache.org/listinfo/squid-users Thanks . i have completely removed that refresh_pattern line but still same result . -- View this message in context: http://squid-web-proxy-cache.1019090.n4.nabble.com/Yet-another-store-id-question-HIT-MISS-tp4678972p4678983.html Sent from the Squid - Users mailing list archive at Nabble.com. ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] Yet another store_id question HIT MISS
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 17.08.2016 20:28, Amos Jeffries пишет: > On 18/08/2016 1:43 a.m., Omid Kosari wrote: >> Why following link is HIT >> >> X-Cache:"HIT from cache1" >> X-Cache-Lookup:"HIT from cache1:3128" >> >> >> http://igcdn-photos-c-a.akamaihd.net/hphotos-ak-xaf1/t51.2885-15/s150x150/e35/13649137_1547514802224163_950421795_n.jpg >> >> but this one is MISS >> >> http://igcdn-photos-a-a.akamaihd.net/hphotos-ak-xaf1/t51.2885-15/s150x150/e35/13649137_1547514802224163_950421795_n.jpg >> > > Because your override-expire tells Squid to ignore the > Cache-Control:max-age expiry details - instead to treat anything 10080 > seconds old as expired. > > Then the override-last-mod tells Squid to ignore the Last-Modified > header, instead to treat everything arriving as being 10080 seconds old. > > When combined they mean that objects are only cacheable for the > remainder of the first second after arrival. When that 0-1 second is > over they are identified as being stale and require revalidation to be used. > > Your options also tell Squid to ignore revalidation ... > You see where this is going? > > > It is most often a bad idea to override cache controls coming from the > server. refresh_pattern is a way to work around broken services or those > which do not send sufficient cache-controls to cache properly. Use with > great care and caution. I do not agree. Many webmasters intentionally set the value in the "no cache" in their own selfish interests. For example, when displaying advertising. Or, even worse, when watching videos - like Youtube. Which leads to the monstrous traffic overspending. Why should we respect them? While they actively oppose us? > > > Amos > > ___ > squid-users mailing list > squid-users@lists.squid-cache.org > http://lists.squid-cache.org/listinfo/squid-users -BEGIN PGP SIGNATURE- Version: GnuPG v2 iQEcBAEBCAAGBQJXtHXiAAoJENNXIZxhPexGueMIAIWGnKWYsEkMPKmCqotCAcxo N+raeePYsVaur/PZFsZgm2I9Dh+K0x2E8s5cYo5kcAcJl+q2Kfe9M1PvTJdvc6AG zOSHzYtZ6e3n/w0f/WkUunSXOX0gWOlqHXvZZTjfm6R1zH8Qw+yx3jWIKQG8TuFP EwSk1fPHfk77sfBXeYcgHPOYHrpGr8ubLsUmfQmrLWTWd5LP6rpFtGvm/pnreX9j ZTOoNjYcqRdC+90Hwg6p6KDCyxSJOkd00fOoCclp4toAiNEwq2mUOOVwlvuL6pTZ Vz97vrus0c2ntcojYPlHmX9/ywzs4vlR1y8J2QAg5HeITvnlv64ZI0Mmt3Enjmc= =l7rO -END PGP SIGNATURE- 0x613DEC46.asc Description: application/pgp-keys ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] Yet another store_id question HIT MISS
On 18/08/2016 1:43 a.m., Omid Kosari wrote: > Why following link is HIT > > X-Cache:"HIT from cache1" > X-Cache-Lookup:"HIT from cache1:3128" > > > http://igcdn-photos-c-a.akamaihd.net/hphotos-ak-xaf1/t51.2885-15/s150x150/e35/13649137_1547514802224163_950421795_n.jpg > > but this one is MISS > > http://igcdn-photos-a-a.akamaihd.net/hphotos-ak-xaf1/t51.2885-15/s150x150/e35/13649137_1547514802224163_950421795_n.jpg > Because your override-expire tells Squid to ignore the Cache-Control:max-age expiry details - instead to treat anything 10080 seconds old as expired. Then the override-last-mod tells Squid to ignore the Last-Modified header, instead to treat everything arriving as being 10080 seconds old. When combined they mean that objects are only cacheable for the remainder of the first second after arrival. When that 0-1 second is over they are identified as being stale and require revalidation to be used. Your options also tell Squid to ignore revalidation ... You see where this is going? It is most often a bad idea to override cache controls coming from the server. refresh_pattern is a way to work around broken services or those which do not send sufficient cache-controls to cache properly. Use with great care and caution. Amos ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] Yet another store_id question HIT MISS
Eliezer Croitoru-2 wrote
> StoreID is not the only thing which can affect a HIT or a MISS.
> A nice tool which was written to understand the subject is RedBot at:
> https://redbot.org/
>
> From a simple inspection of the file it seems that it should get hit but,
> why are you using StoreID for this object?
Already tested with redbot and then asked here .
This url belongs to instagram and it uses a lot of such urls for same file .
Eliezer Croitoru-2 wrote
> Also why are you using:
> refresh_pattern -i ^http:\/\/[a-zA-Z0-9\-\_\.]+\.squid\.internal\/.* 10080
> 95% 86400 override-lastmod override-expire reload-into-ims ignore-reload
> ignore-must-revalidate ignore-no-store ignore-private
>
> ??
> You would only need this for the specific case which the hostname is
> "dynamic".
Thanks removed it .
Eliezer Croitoru-2 wrote
> This url seems by default cache friendly and only if you have enough
> details on their cdn network you should try to use StoreID.
> Something that may help you is the next log format settings:
> logformat cache_headers %ts.%03tu %6tr %>a %Ss/%03>Hs %
>
> h" "%{Cache-Control}>ha" Q-P: "%{Pragma}>h" "%{Pragma}>ha" REP-CC:
> "%{Cache-Control} "%{Vary}eui
> access_log daemon:/var/log/squid/access.log cache_headers
>
> Try to see how the requests for these looks like in the logs.
Yes we have enough details. Now i am investigating on hit miss problem with
logformat you provided .I will inform you the result .
Thanks
--
View this message in context:
http://squid-web-proxy-cache.1019090.n4.nabble.com/Yet-another-store-id-question-HIT-MISS-tp4678972p4678979.html
Sent from the Squid - Users mailing list archive at Nabble.com.
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] Yet another store_id question HIT MISS
Hey Omid,
StoreID is not the only thing which can affect a HIT or a MISS.
A nice tool which was written to understand the subject is RedBot at:
https://redbot.org/
From a simple inspection of the file it seems that it should get hit but, why
are you using StoreID for this object?
Also why are you using:
refresh_pattern -i ^http:\/\/[a-zA-Z0-9\-\_\.]+\.squid\.internal\/.* 10080 95%
86400 override-lastmod override-expire reload-into-ims ignore-reload
ignore-must-revalidate ignore-no-store ignore-private
??
You would only need this for the specific case which the hostname is "dynamic".
This url seems by default cache friendly and only if you have enough details on
their cdn network you should try to use StoreID.
Something that may help you is the next log format settings:
logformat cache_headers %ts.%03tu %6tr %>a %Ss/%03>Hs %h" "%{Cache-Control}>ha" Q-P: "%{Pragma}>h"
"%{Pragma}>ha" REP-CC: "%{Cache-Control}eui
access_log daemon:/var/log/squid/access.log cache_headers
Try to see how the requests for these looks like in the logs.
Eliezer
Eliezer Croitoru
Linux System Administrator
Mobile: +972-5-28704261
Email: elie...@ngtech.co.il
-Original Message-
From: squid-users [mailto:squid-users-boun...@lists.squid-cache.org] On Behalf
Of Omid Kosari
Sent: Wednesday, August 17, 2016 4:44 PM
To: squid-users@lists.squid-cache.org
Subject: [squid-users] Yet another store_id question HIT MISS
Why following link is HIT
X-Cache:"HIT from cache1"
X-Cache-Lookup:"HIT from cache1:3128"
http://igcdn-photos-c-a.akamaihd.net/hphotos-ak-xaf1/t51.2885-15/s150x150/e35/13649137_1547514802224163_950421795_n.jpg
but this one is MISS
http://igcdn-photos-a-a.akamaihd.net/hphotos-ak-xaf1/t51.2885-15/s150x150/e35/13649137_1547514802224163_950421795_n.jpg
store_id_program "/usr/lib/squid/storeid_file_rewrite"
"/etc/squid/storeid_db"
store_id_children 50 startup=10 idle=5 concurrency=50
acl storeiddomainregex dstdom_regex
^igcdn(\-photos|\-videos)[a-z0-9\-]{0,9}\.akamaihd\.net$
store_id_access allow storeiddomainregex
store_id_access deny all
refresh_pattern -i ^http:\/\/[a-zA-Z0-9\-\_\.]+\.squid\.internal\/.* 10080
95% 86400 override-lastmod override-expire reload-into-ims ignore-reload
ignore-must-revalidate ignore-no-store ignore-private
storeid_db content
^http:\/\/igcdn-.*\.akamaihd\.net/hphotos-ak-.*/(t5.*?)(?:\?|$)
http://instagramcdn.squid.internal/$1
root@cache:~# echo
'http://igcdn-photos-c-a.akamaihd.net/hphotos-ak-xaf1/t51.2885-15/s150x150/e35/13649137_1547514802224163_950421795_n.jpg'
| /usr/lib/squid/storeid_file_rewrite /etc/squid/storeid_db
OK
store-id=http://instagramcdn.squid.internal/t51.2885-15/s150x150/e35/13649137_1547514802224163_950421795_n.jpg
root@cache:~# echo
'http://igcdn-photos-a-a.akamaihd.net/hphotos-ak-xaf1/t51.2885-15/s150x150/e35/13649137_1547514802224163_950421795_n.jpg'
| /usr/lib/squid/storeid_file_rewrite /etc/squid/storeid_db
OK
store-id=http://instagramcdn.squid.internal/t51.2885-15/s150x150/e35/13649137_1547514802224163_950421795_n.jpg
StoreId helper Statistics:
program: /usr/lib/squid/storeid_file_rewrite
number active: 10 of 50 (0 shutting down)
requests sent: 1755734
replies received: 1755734
queue length: 0
avg service time: 0 msec
Number of requests bypassed because all StoreId helpers were busy: 0
--
View this message in context:
http://squid-web-proxy-cache.1019090.n4.nabble.com/Yet-another-store-id-question-HIT-MISS-tp4678972.html
Sent from the Squid - Users mailing list archive at Nabble.com.
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users
[squid-users] Yet another store_id question HIT MISS
Why following link is HIT
X-Cache:"HIT from cache1"
X-Cache-Lookup:"HIT from cache1:3128"
http://igcdn-photos-c-a.akamaihd.net/hphotos-ak-xaf1/t51.2885-15/s150x150/e35/13649137_1547514802224163_950421795_n.jpg
but this one is MISS
http://igcdn-photos-a-a.akamaihd.net/hphotos-ak-xaf1/t51.2885-15/s150x150/e35/13649137_1547514802224163_950421795_n.jpg
store_id_program "/usr/lib/squid/storeid_file_rewrite"
"/etc/squid/storeid_db"
store_id_children 50 startup=10 idle=5 concurrency=50
acl storeiddomainregex dstdom_regex
^igcdn(\-photos|\-videos)[a-z0-9\-]{0,9}\.akamaihd\.net$
store_id_access allow storeiddomainregex
store_id_access deny all
refresh_pattern -i ^http:\/\/[a-zA-Z0-9\-\_\.]+\.squid\.internal\/.* 10080
95% 86400 override-lastmod override-expire reload-into-ims ignore-reload
ignore-must-revalidate ignore-no-store ignore-private
storeid_db content
^http:\/\/igcdn-.*\.akamaihd\.net/hphotos-ak-.*/(t5.*?)(?:\?|$)
http://instagramcdn.squid.internal/$1
root@cache:~# echo
'http://igcdn-photos-c-a.akamaihd.net/hphotos-ak-xaf1/t51.2885-15/s150x150/e35/13649137_1547514802224163_950421795_n.jpg'
| /usr/lib/squid/storeid_file_rewrite /etc/squid/storeid_db
OK
store-id=http://instagramcdn.squid.internal/t51.2885-15/s150x150/e35/13649137_1547514802224163_950421795_n.jpg
root@cache:~# echo
'http://igcdn-photos-a-a.akamaihd.net/hphotos-ak-xaf1/t51.2885-15/s150x150/e35/13649137_1547514802224163_950421795_n.jpg'
| /usr/lib/squid/storeid_file_rewrite /etc/squid/storeid_db
OK
store-id=http://instagramcdn.squid.internal/t51.2885-15/s150x150/e35/13649137_1547514802224163_950421795_n.jpg
StoreId helper Statistics:
program: /usr/lib/squid/storeid_file_rewrite
number active: 10 of 50 (0 shutting down)
requests sent: 1755734
replies received: 1755734
queue length: 0
avg service time: 0 msec
Number of requests bypassed because all StoreId helpers were busy: 0
--
View this message in context:
http://squid-web-proxy-cache.1019090.n4.nabble.com/Yet-another-store-id-question-HIT-MISS-tp4678972.html
Sent from the Squid - Users mailing list archive at Nabble.com.
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users
