Re: A question regarding rock cache_dir.
Thanks Alex for the detailed reposonse.
I have been reading the RockStore wiki and at start I couldn't
understand the description since the strucutre was a bit too much for a
second.
I wanted to write some basic code that would do couple "actions" in the
DB and this will make sure that I understood it.
By the way Rock ROCKS!!
once I will understand it a bit more I hope to write a small helper that
will do something in the DB file.
Eliezer
On 09/20/2013 06:47 PM, Alex Rousskov wrote:
> On 09/19/2013 07:50 PM, Eliezer Croitoru wrote:
>
>> 1. how is the rock organized compared to ufs?
>
> Rock store uses a single database-style file that holds all information
> about stored entries (as opposed to one entry per file approach used by
> ufs). In the future, Rock store may start to use raw disk partitions as
> well.
>
>
>> 2. what is the rock file structure?binary I assume..?
>
> The basic structure is a fixed-size database header followed by a
> sequence of fixed-size database records or "slots". Besides the source
> code itself, Rock store is documented at
>
> http://wiki.squid-cache.org/Features/RockStore
> http://wiki.squid-cache.org/Features/LargeRockStore
>
>
>> 3. assuming we want to *purge* from the rock cache_dir an object when
>> the server is offline what would be the pseudo code to do that?
>
> calculate the anchor slot position using the entry key and slot size
> load the entry meta data from the anchor slot on disk
> if the anchor slot is occupied by the entry you want to purge {
> mark the slot as empty
> write the slot [metadata] back to disk
> optional: update other slots occupied by the entry (Large Rock)
> }
>
> Note that the above can be done both on a disconnected or live store.
> When purging a live store, you would need to use/update the store maps
> that Squid loads into shared memory (Ipc::StoreMap and its kids).
>
> For a disconnected store, the overall algorithm is very similar to ufs
> if you replace "file" with "[anchor] slot". In the official code, anchor
> slot is the only slot used for the entry. Large Rock uses a chain of
> slots to store additional entry data if needed, removing the "all
> entries must fit into one-slot" limitation.
>
>
> HTH,
>
> Alex.
>
Re: A question regarding rock cache_dir.
On 09/19/2013 07:50 PM, Eliezer Croitoru wrote:
> 1. how is the rock organized compared to ufs?
Rock store uses a single database-style file that holds all information
about stored entries (as opposed to one entry per file approach used by
ufs). In the future, Rock store may start to use raw disk partitions as
well.
> 2. what is the rock file structure?binary I assume..?
The basic structure is a fixed-size database header followed by a
sequence of fixed-size database records or "slots". Besides the source
code itself, Rock store is documented at
http://wiki.squid-cache.org/Features/RockStore
http://wiki.squid-cache.org/Features/LargeRockStore
> 3. assuming we want to *purge* from the rock cache_dir an object when
> the server is offline what would be the pseudo code to do that?
calculate the anchor slot position using the entry key and slot size
load the entry meta data from the anchor slot on disk
if the anchor slot is occupied by the entry you want to purge {
mark the slot as empty
write the slot [metadata] back to disk
optional: update other slots occupied by the entry (Large Rock)
}
Note that the above can be done both on a disconnected or live store.
When purging a live store, you would need to use/update the store maps
that Squid loads into shared memory (Ipc::StoreMap and its kids).
For a disconnected store, the overall algorithm is very similar to ufs
if you replace "file" with "[anchor] slot". In the official code, anchor
slot is the only slot used for the entry. Large Rock uses a chain of
slots to store additional entry data if needed, removing the "all
entries must fit into one-slot" limitation.
HTH,
Alex.
A question regarding rock cache_dir.
I have seen the basic idea of ufs\aufs cache_dir which contains the DB that holds the dynamic data and the cache_dir files structure(TLV etc). Before I am even trying to understand rock basic structure I am looking for a tiny briefing about it. 1. how is the rock organized compared to ufs? 2. what is the rock file structure?binary I assume..? 3. assuming we want to *purge* from the rock cache_dir an object when the server is offline what would be the pseudo code to do that? If you can start with something from the above it would be good and then we can try to go on. Once I will have couple hints I will try to read some of the code to make sure I understood it right. Thanks, Eliezer
Re: I have a question about the md5 hash.
On 09/13/2012 12:12 PM, Henrik Nordström wrote: Is it a 200 OK response. Suspect it's a 304? Thanks, WIll check maybe it is. ..checked.. not 302 a fully 200 response. 1347525419.744116 127.0.0.1 TCP_MISS/200 767 GET http://www1.ngtech.co.il/302.html - HIER_DIRECT/79.181.232.109 text/html 1347525421.880 12 127.0.0.1 TCP_MISS/200 767 GET http://www1.ngtech.co.il/302.html - HIER_DIRECT/79.181.232.109 text/html 1347525870.740143 127.0.0.1 TCP_MISS/200 767 GET http://www1.ngtech.co.il/302.html - HIER_DIRECT/79.181.232.109 text/html 1347525873.614 9 127.0.0.1 TCP_MISS/200 767 GET http://www1.ngtech.co.il/302.html - HIER_DIRECT/79.181.232.109 text/html 1347526129.689311 127.0.0.1 TCP_MISS/200 767 GET http://www1.ngtech.co.il/302.html - HIER_DIRECT/79.181.232.109 text/html 1347526131.798 8 127.0.0.1 TCP_MISS/200 767 GET http://www1.ngtech.co.il/302.html - HIER_DIRECT/79.181.232.109 text/html these are all the access.log records in squid. (The delay there is because it's on wireless.) The sources are from the 3.trunk from a week ago. Eliezer
Re: I have a question about the md5 hash.
tor 2012-09-13 klockan 12:07 +0300 skrev Eliezer Croitoru: > but in the logs is see a lot: > 2012/09/13 11:48:51.798 kid1| StoreEntry::checkCachable: NO: not cachable Is it a 200 OK response. Suspect it's a 304? Regards Henrik
Re: I have a question about the md5 hash.
On 09/13/2012 10:07 AM, Henrik Nordström wrote: We might need to make this the actual method string instead of internal >code to preserve store keys across squid versions. Well I was wondering about it myself. and I wanted to ask a thing but still didnt finished investigating about it. I have seen in debug ALL,3 some weird thing about the hash key and This is the reason I asked about it. I'm still not sure what I have seen so dont take yet anything as granted from me about it yet. I have tried to digest a url with a method to see if I can predict what I will expect while watching the logs for a set of tests. I saw squid does a double lookup for a HEAD and GET so 001(1) and 100 (4) and I got a string using a struct. Request uint8 :method string :url = http://www1.ngtech.co.il/302.html which resulted with a binary string with size of 34 bytes: "\x01http://www1.ngtech.co.il/302.html"; the \x01 is hex octet for 1. and I can predict it pretty nicely and smoothly for the "storeGet:key" which is good since i can predict which key will be used to lookup the store. 2012/09/13 11:37:01.875 kid1| storeGet: looking up AE59C69F629FE4253FBCC7DCC4D924E1 but in the logs is see a lot: 2012/09/13 11:48:51.798 kid1| StoreEntry::checkCachable: NO: not cachable but if you will check the url itself it's cachable as hell and allows the cache to decide the freshness period. as an example: http://redbot.org/?uri=http%3A%2F%2Fwww1.ngtech.co.il%2F302.html and it's a regular forward proxy so not a thing about validating the host should cause this. The browser sure cache it well and is not fetching using the proxy unless I force it a refresh. I'm still testing on where exactly we must use the store_url and not the original_url. I have two main point of url origin which are: at StoreEntry::url >tranformed into---> StoreEntry::OriginalUrl and at MemObject->url --->which tranformed into --> MemObject->original_url (In a lot of places ) this is only canonical for now but it's in-line with other variables and methods at these points. I was thinking: instead of using If else different statement how about using: (store_url ? store_url : origianal_url, ..) the disadvantage is the I cant put there a debugs... but I can throw it inside the related method that will state the url. Eliezer
Re: I have a question about the md5 hash.
tor 2012-09-13 klockan 12:05 +1200 skrev Amos Jeffries: > Just had a thought. I wonder if this is related to the releases which > people suddenly started having cache MISS for a period with no visible > reason. > That could be the releases where we added/removed methods from the > registered set. Only if the new methods were added before METHOD_GET. > We might need to make this the actual method string instead of internal > code to preserve store keys across squid versions. Yes. Regards Henrik
Re: I have a question about the md5 hash.
On 13.09.2012 04:51, Henrik Nordström wrote: ons 2012-09-12 klockan 10:26 +0300 skrev Eliezer Croitoru: if it was a string what will be the string structure? It's not a string. The first hashed octet is the Squid internal method representation in binary integer form. Followed by the requested URL. Just had a thought. I wonder if this is related to the releases which people suddenly started having cache MISS for a period with no visible reason. That could be the releases where we added/removed methods from the registered set. We might need to make this the actual method string instead of internal code to preserve store keys across squid versions. See storeKeyPublic() function for details http://bazaar.launchpad.net/~squid/squid/3-trunk/view/head:/src/store_key_md5.cc#L118 And for method number assignment see HttpRequestMethod http://bazaar.launchpad.net/~squid/squid/3-trunk/view/head:/src/HttpRequestMethod.h#L41 Regards Henrik Amos
Re: I have a question about the md5 hash.
On 9/12/2012 7:51 PM, Henrik Nordström wrote: ons 2012-09-12 klockan 10:26 +0300 skrev Eliezer Croitoru: if it was a string what will be the string structure? It's not a string. The first hashed octet is the Squid internal method representation in binary integer form. Followed by the requested URL. See storeKeyPublic() function for details http://bazaar.launchpad.net/~squid/squid/3-trunk/view/head:/src/store_key_md5.cc#L118 And for method number assignment see HttpRequestMethod http://bazaar.launchpad.net/~squid/squid/3-trunk/view/head:/src/HttpRequestMethod.h#L41 Regards Henrik Thanks Henrik, Sharp as a blade! I have seen the storekeypublic and was over it but i couldn't figure-out the method thing unless I would dig up more and since some people have the answer already. Thanks, it gave me a lot. Eliezer -- Eliezer Croitoru https://www1.ngtech.co.il IT consulting for Nonprofit organizations eliezer ngtech.co.il
Re: I have a question about the md5 hash.
ons 2012-09-12 klockan 10:26 +0300 skrev Eliezer Croitoru: > if it was a string what will be the string structure? It's not a string. The first hashed octet is the Squid internal method representation in binary integer form. Followed by the requested URL. See storeKeyPublic() function for details http://bazaar.launchpad.net/~squid/squid/3-trunk/view/head:/src/store_key_md5.cc#L118 And for method number assignment see HttpRequestMethod http://bazaar.launchpad.net/~squid/squid/3-trunk/view/head:/src/HttpRequestMethod.h#L41 Regards Henrik
Re: I have a question about the md5 hash.
On 9/12/2012 12:57 AM, Alex Rousskov wrote: On 09/11/2012 10:41 AM, Eliezer Croitoru wrote: I do not know Ruby, but googling suggests that Ruby equivalent of incremental MD5 hash creation would be something like this: incr_digest = Digest::MD5.new() incr_digest << url incr_digest << method return incr_digest.hexdigest Thanks Alex. As a matter of fact I know that already. the problem is that this is equal to: req = url + method incr_digest << req and actually is like: incr_digest << "http://www.yahoo.com/GET"; ^^^ url method and this is more like my question. if it was a string what will be the string structure? I think i can easily try to extract from the buffer the exact line but if someone knows how it suppose to be done I will be happy. Thanks, Eliezer HTH, Alex. -- Eliezer Croitoru https://www1.ngtech.co.il IT consulting for Nonprofit organizations eliezer ngtech.co.il
Re: I have a question about the md5 hash.
On 09/11/2012 10:41 AM, Eliezer Croitoru wrote:
> I have seen some of the code but I dont understand on what the
> calculation being done in literal works.
> to explain myself.
>
> I do know that it uses the url and the method to hash for the key.
> so there is a public and private keys.
> private is method+url+id
> public is url+method.
> the method\function uses this api:
> SquidMD5Update(&M, (unsigned char *) url, strlen(url));
> the internals of the function is a bit high for me so to translate it
> for me:
> let say in ruby i'm using:
> digest = Digest::MD5.hexdigest("Hello World\n")
> => "e59ff97941044f85df5297e1c302d260"
>
> is there any way to translate a url + method hash into ruby?
> a specific example is:
> 2012/09/11 18:37:45.949| storeKeypublicwith URL and method: GET
> http://laptope:3128/squid-internal-static/icons/silk/page_world.png
> 2012/09/11 18:37:45.949| storeGet: looking up
> E3F08214F440DC108204575CF98E2054
>
> I can try to do some thinks in the code that will show me each time the
> two options but I prefer to understand and plan what i'm suppose to see
> which will help me.
I do not know Ruby, but googling suggests that Ruby equivalent of
incremental MD5 hash creation would be something like this:
incr_digest = Digest::MD5.new()
incr_digest << url
incr_digest << method
return incr_digest.hexdigest
HTH,
Alex.
I have a question about the md5 hash.
I have seen some of the code but I dont understand on what the
calculation being done in literal works.
to explain myself.
I do know that it uses the url and the method to hash for the key.
so there is a public and private keys.
private is method+url+id
public is url+method.
the method\function uses this api:
SquidMD5Update(&M, (unsigned char *) url, strlen(url));
the internals of the function is a bit high for me so to translate it
for me:
let say in ruby i'm using:
digest = Digest::MD5.hexdigest("Hello World\n")
=> "e59ff97941044f85df5297e1c302d260"
is there any way to translate a url + method hash into ruby?
a specific example is:
2012/09/11 18:37:45.949| storeKeypublicwith URL and method: GET
http://laptope:3128/squid-internal-static/icons/silk/page_world.png
2012/09/11 18:37:45.949| storeGet: looking up
E3F08214F440DC108204575CF98E2054
I can try to do some thinks in the code that will show me each time the
two options but I prefer to understand and plan what i'm suppose to see
which will help me.
Thanks in advance,
Eliezer
--
Eliezer Croitoru
https://www1.ngtech.co.il
IT consulting for Nonprofit organizations
eliezer ngtech.co.il
Re: A question regarding url_rewrite_program
On Sat, 2008-09-20 at 04:51 +0530, Kulbir Saini wrote: > I want to know what signal is sent by the squid to the url rewrite > programs when squid service is restarted or reload? Is it documented > somewhere? AFAICT, Squid just closes the file descriptor that is used to communicate with the helper program. The program is supposed to quit when it can no longer read from or write to Squid. For details, please see redirectShutdown() which calls helperShutdown(). HTH, Alex.
A question regarding url_rewrite_program
Hi list, I want to know what signal is sent by the squid to the url rewrite programs when squid service is restarted or reload? Is it documented somewhere? I hope my question qualifies to be asked on devel list. --- Thank you, Kulbir Saini, Computer Science and Engineering, International Institute of Information Technology, Hyderbad, India - 500032. My Home-Page: http://saini.co.in/ My Web-Blog: http://fedora.co.in/ IRC nick : generalBordeaux Channels : #fedora, #fedora-devel, #yum on freenode -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean.
Re: A question regarding Squid authentication
mån 2006-06-12 klockan 21:23 +0300 skrev Tsachi: > Thanks for the information I will try to do that as well. > Can you point me to why, When working with WWW authenticate there is a > valid value to "httpState->request->pinned_connection" in > "HttpReadReplay()" > , But, when working with parent Proxy the value is nil and it seems > that "httpState->orig_request->pinned_connection" holds the right > value? > > This is relevent for "pinning connection patch" that does mark the > proxy connection as pinning in the "clientBuildReplyHeader()". orig_request should always be used when looking for the pinned_connection field. request is the proxy request when talking to a proxy. when going direct they are equal. Regards Henrik signature.asc Description: Detta är en digitalt signerad meddelandedel
Re: A question....
Hi, Please ignore this stupid question. Was made while I was under panic :-( > Hi all, > > I have a question about clientHttpRequest structure. > It has the field "clientHttpRequest *next". > Where the hell used this field? > ... > What am I loosing here? Everything Sory, Christos
A question....
Hi all,
I have a question about clientHttpRequest structure.
It has the field "clientHttpRequest *next".
Where the hell used this field?
I am showing code like that:
while (*H) {
if (*H == http)
break;
H = &(*H)->next;
}
But I can not find where two or more clientHttpRequest structures
connected to a list.
Is it possible for the clientHttpRequest->next to have a value other
than NULL?
What am I loosing here?
Just tell me if needed or not and I will find the rest...
Thanks,
Christos
