Re: Store_url_rewrite for squid 3+
On 9/24/2012 11:46 AM, Amos Jeffries wrote: Haven't heard anything about this is a while. How is this project going? Well it's holidays here now so I took a small stop to breath. we had one just ended and 2\3 in the way so it leaves small amount of time. and some family stuff on the way so I hope I will sit some more this week. I had some progress while refactoring and re-reviewing the code. I found one problem of handling the helper which leaves the helper without any place(STOUT) to write and there for will have trouble and crash. I am now working on the points which each original_url is being requested and making sure this is the right point to replace it with store_url. I managed to predict what keys squid should lookup(storeget) and create couple times In the code but I'm still missing something since while I managed to change the digest I get in the store.log a lot of question mark which leaves me to think that probably there is a point which the referencing variable is not the right one. I also had a bit time to look over the Link headers and metalinks files since I remember something about it from the past. Eliezer
Re: Store_url_rewrite for squid 3+
On 09/08/2012 01:36 AM, Alex Rousskov wrote:
On 09/07/2012 01:32 PM, Eliezer Croitoru wrote:
On 09/07/2012 05:10 PM, Alex Rousskov wrote:
I am not sure what option you are referring to in the above. The
Store::get(key) API I have described is not optional -- it is the
primary way of detecting a hit.
+1 to use the api.
are we talking about the Store::Root().get ?
That is where it starts, but other Store kids implement the get() method
as well IIRC. From the caller perspective, you should also look at
storeGetPublic*() and similar functions.
basic review Done already.
The URL rewriting must happen before or during Store key calculation.
The problem is that if I am rewriting the url, unlike
url_rewrite\redirect the connection should be initiated based on the
original url and not the rewritten one.
Sure, but that is not a problem you need to worry about much. Since you
are not going to alter the original URL, the connection initiation code
will not be affected by your changes (except for some minor adjustments
if some data members get renamed or Store::get() API-related calls get
adjusted as discussed below).
Got my answers during the review.
What I do not know yet is what and where and based on what the request
is done.
For this project, you do not need to know much about connection
initiation code. You need to make a copy of the original URL, rewrite
that copy using the new helper, and put it in a new HttpRequest data
member (for example). This should probably happen (or be initiated from)
client_side.cc or client_side_request.cc.
Well after the review it seems like there is no need to copy the request
but to use the char * storeurl.
it makes more sense since you never touch any other part of the object
while you are getting it.
You then need to make sure that every time Store calculates a store key,
the new member (if any) is used instead of the original URL.
this is the original reason I wanted to know\find about the initiation
code.
but I managed to find what I need for now.
Since the current code is using the original request URL where you want
the rewritten/store URL to be used, I recommend _renaming_ all members
that keep or extract the original/request URL (e.g., MemObject::url).
This will force you to go through the relevant callers and make sure
that the right URL is used, depending on the caller context.
Exactly what I was getting into.
Once you are done, we can review whether the vast majority of changes in
the diff are a simple renaming of url to requestUrl. If yes, you can
revert that renaming change. If there are many cases where old url
became storeUrl and many cases where it became requestUrl then the
change should stay.
Please note that the above is just a sketch. I am not suggesting any
specific new names, and I am using existing class names only as an example.
Will look into it more.
as far as I can tell most of the current code MemObject::url calls are
from debug.
so we need to store the original url and the store_url at least for the
period of time the request is being served.
Yes, we need to keep both URLs around during the transaction lifetime.
When I was talking about storing a URL, I was only talking about Store
(i.e., storing something in the cache).
A reason I can think of is to make it possible to use
mgr:store_url_option that will maybe give some data about in cache
objects such as urls etc.
For most cases the access.log is what will be used but if there is a
need it's for it it's for that.
by the way the Purge tool uses the URL part of the metadata to find urls
and purge them in a ufs cache_dir.
since I stored objects in my cache with URL meta data as the rewritten
one I could do lookups on the cache objects using purge.
It gives you the ability to find out exactly at this moment what objects
urls are in the cache and in my case also rewritten ones.
(answer about storing or not the store_url in meta)
That is a separate question. I cannot answer it, unfortunately. I do not
know why Squid2 stored either of those URLs. I know Squid3 does not
store the original/request URL.
It does in ufs cahe_dir as a fact. (see the below quotation from a cache
file)
url_rewrite api gets the http requests as the background to any action
so it cannot be used for store_url since any change to the request will
lead to the change we are trying to not do exactly.
mem_object contains:
{...
char *url;
HttpRequest *request;
...}
and the url is being used to make the hash but I still dont know what is
being taken from the mem_object to do the request.
HttpStateData::buildRequestPrefix() builds request headers. It may use
mem_object::url. I do not think you should change that code (apart from
renaming and Store::Get() API changes discussed elsewhere).
You should only be concerned with calls to storeGetPublic*() and such.
Make sure the callers do not use mem_object::url without checking
whether there is a rewritten store URL as
Re: Store_url_rewrite for squid 3+
On 9/09/2012 4:19 a.m., Alex Rousskov wrote: On 09/07/2012 09:13 PM, Amos Jeffries wrote: Also, any revalidation requests done later must be done on the original request URL. Not the stored URL nor the potentially different current client request URL. This sounds like a very important point that could justify storing the original request URL -- exactly the kind of information I was asking for, thank you! Why do we have to use the original request URL for revalidation instead of the current one? We use current, not original request headers (we do not store the original ones), right? Is it better to combine current headers with the original URL than it is to use the current URL with current headers? Revalidation requires very precise variant targeting to ensure updated headers received from the revalidation is not corrupting the cached object copy. Regardless what people may think the YouTube URLs and other sites being de-duplicated with store-url *are* actually pointing at different files on different servers with potentially different hashes or encoding details. Particularly in the cases where the HD and standard definition variants of a video are store-url mapped to the same cache object. The URL and ETag are both critical details to preserve here. Also, anything else which is used for specific Squid-upstream identification of the resource being revalidated. The store URL rewriting feature essentially assumes that any request URL that maps to URL X is equivalent and, hence, any response to any request URL that maps to URL X is equivalent. Why not use that assumption when revalidating? If we receive a 304, we can keep using the stored content. If we receive new response content, should not we assume that the stored content [under the original URL] is stale as well? Assumes is the right word. They are equivalent only in the proxy administrators thoughts. Which may be wrong or right. We have to let them be wrong sometimes and cause clients display problems, but we should not let them cause local cache corruption with revalidation updating cached objects meta data from incorrect variant sources. Again, I am not trying to say that using original URL for revalidation is wrong -- I am just trying to understand what the design constraints are. We could simply re-fetch and store a new copy from the new client request details. Revalidation is an optimization, but requires correct identification of the particular resource and variant we have in cache. That goes for anything in cache, store-url is just tricky in that the client-side request can't present us the accurate details for server-side. Thank you, Alex. P.S. The above still does not justify storing the rewritten URL(s), of course. No. I think those are only useful for key purposes and can be discarded once the object in cache is located for a HTI, or stored fro a MISS. Amos
Re: Store_url_rewrite for squid 3+
On 09/06/2012 03:58 AM, Amos Jeffries wrote:
I don't think there is anything which needs new cache_cf.cc code. The
parsing side if things is identical for url_rewrite_*. The different
defaults and locations are all coded in cf.data.pre ...
yes indeed but the actual effect comes from the code in cache_cf.cc
an example is:
if (Config.Program.redirect) {
if (Config.redirectChildren.n_max 1) {
Config.redirectChildren.n_max = 0;
wordlistDestroy(Config.Program.redirect);
}
}it's specific for the redirect program.
I have tried to use the helper but it seems like without the cache_cf.cc
code it wil assume the number of default helpers is 0/20 and will not
start any.
which is werid...
Eliezer
Re: Store_url_rewrite for squid 3+
On 10.09.2012 04:56, Eliezer Croitoru wrote:
On 09/06/2012 03:58 AM, Amos Jeffries wrote:
I don't think there is anything which needs new cache_cf.cc code.
The parsing side if things is identical for url_rewrite_*. The
different defaults and locations are all coded in cf.data.pre ... yes
indeed but the actual effect comes from the code in cache_cf.cc
an example is:
if (Config.Program.redirect) {
if (Config.redirectChildren.n_max 1) {
Config.redirectChildren.n_max = 0;
wordlistDestroy(Config.Program.redirect);
}
}it's specific for the redirect program.
I have tried to use the helper but it seems like without the
cache_cf.cc code it wil assume the number of default helpers is 0/20
and will not start any.
which is werid...
Ah. Okay I had forgotten that hack. Yes it is needed for store-url as
well.
It is for the case where a program name is configured but there are no
children to be started. Or on reconfigure where a program used to be
configured but is now removed from the config.
0/20 with none started straight away is the default. One will be
started on first request through the proxy that needs the helper.
Amos
Re: Store_url_rewrite for squid 3+
On 09/07/2012 09:13 PM, Amos Jeffries wrote: Also, any revalidation requests done later must be done on the original request URL. Not the stored URL nor the potentially different current client request URL. This sounds like a very important point that could justify storing the original request URL -- exactly the kind of information I was asking for, thank you! Why do we have to use the original request URL for revalidation instead of the current one? We use current, not original request headers (we do not store the original ones), right? Is it better to combine current headers with the original URL than it is to use the current URL with current headers? The store URL rewriting feature essentially assumes that any request URL that maps to URL X is equivalent and, hence, any response to any request URL that maps to URL X is equivalent. Why not use that assumption when revalidating? If we receive a 304, we can keep using the stored content. If we receive new response content, should not we assume that the stored content [under the original URL] is stale as well? Again, I am not trying to say that using original URL for revalidation is wrong -- I am just trying to understand what the design constraints are. Thank you, Alex. P.S. The above still does not justify storing the rewritten URL(s), of course.
Re: Store_url_rewrite for squid 3+
On 09/06/2012 09:04 PM, Alex Rousskov wrote: On 09/05/2012 06:58 PM, Amos Jeffries wrote: FWIW, I have not reviewed the store_url_rewrite code in Squid2 so I cannot answer the questions related to how it was done. I can suggest ways of doing this in Squid3, but since somebody already investigated all the alternatives, it would be better to hear the summary of the Squid2 implementation (as it relates to Store) before diving into Squid3 development. squid2 form makes sense for me by the minute. The biggest question for me is why Squid2 code was storing multiple URLs with the cached object (if it was). Why cannot Store just work with the [rewritten] URL given to it and ignore the fact that some [store] URLs originated from some other [real] URLs? I found the answer in the documentation at: 03_major_componenets: The Storage Manager is the glue between client and server sides. Every object saved in the cache is allocated a StoreEntry structure. While the object is being accessed, it also has a MemObject structure. so I think the duplication was to preserve this structure and prevent major api changes. as far I get into the depth of the code I see how it's reasonable to make this decision if you compare the loss and benefits . loosing a few single(sometimes it's not) bytes of GB is cheap compared to depth development time. some calculations: youtube url will be about then 600 ascii letters and before it was object = video but now it's about 1.7MB per chunk. so the loss of about 600 bytes of space(am i right?) compares to 20 Million bytes gain? well on 1.7MB it's something else but we are talking about 96Kb loss for a video file. Are we trying to support going from a store_url_rewrite config back to regular config without losing some of the cached objects? Since we are talking about a try for solution to solve a static objects de-duplication I think it's not our case. and even for a more dynamic one the case is always many-1 to achieve better cache and it's a case that makes no sense in rolling back to a dynamic url acquired per unique IP+COOKIE+TIME+other stuff. It will make sense if there was a plan for rebuilding the cached objects and the storeurl in store but it's seems like too intensive task to hand it like a reasonable regular usage case. If someone have this kind of cases he should migrate from generic cache-proxy to a more specific task customized proxy.(still not seems like a reasonable request for anything unless you want to be the CIA\KGB\FBI and collect data) SNIP ? question how cachemgr gets the list of urls in memory? You might be confusing cache manager (the thing that responds to squidclient mgr:info requests) with Store. Also, you should not think in terms of memory (RAM) because some objects are only cached on disk. It is best to think of Store as a collection of stored objects, ignoring their particular location (memory or disk) to the extent possible. no no i was talking about mgr:info .. this not related in straight connection but I had a question about it in the past and will leave that to somewhere in the future. but related to to store_url there was something that prevented mgr:what-ever-gives-data-on-cached-objects that it wont show the store_url objects. after understanding how it was coded it's pretty obviates how and why this happened and it can be prevented during new development while structuring the data correctly. Store can get a list of cached objects by iterating through store_table and other store indexes. In general, you should not assume that it is possible to get a list of all cached URLs in any efficient/practical fashion because not all in-RAM indexes store URLs. It is only possible to get an answer to the following question: * Is a response with cache key K likely to be in Squid cache now? Where cache key is a hash computed over the request method, request URI, and other properties. this is what I remember. so for the new development there should be an option to do that but also do the rewriting of the url before checking that or more practically to change the cache key calculations if there is a store_url present for the request. SNIP The Store is too big and complex of an API to accurately describe in an email IMO. I would be happy to answer specific questions about the stuff I know, but you may have to research how things work as there is no comprehensive documentation yet. just need major points in the process mentioned before. but will take sometime until I will review more code to not make any basic stand on the subject. I do have an approach that I want to check and Its' based on finding all\specific mem_object operations and object creation in the code. now I'm struggling to juggle find and mark the points. answering the double url thing In 2.7 the feature used the storeswap_meta to add the storeurl string and it's simplified things. it's a TLV struct so you are safe in to
Re: Store_url_rewrite for squid 3+
On 09/07/2012 06:20 AM, Eliezer Croitoru wrote: On 09/06/2012 09:04 PM, Alex Rousskov wrote: The biggest question for me is why Squid2 code was storing multiple URLs with the cached object (if it was). Why cannot Store just work with the [rewritten] URL given to it and ignore the fact that some [store] URLs originated from some other [real] URLs? I found the answer in the documentation at: 03_major_componenets: The Storage Manager is the glue between client and server sides. Every object saved in the cache is allocated a StoreEntry structure. While the object is being accessed, it also has a MemObject structure. I do not understand how the above quote answers my question. Moreover, the above text is stale -- Squid3 (Rock Store and shared memory caches specifically) does not allocate StoreEntry for every object saved in the cache. Furthermore, as far as I can tell, current Squid3 code does not store the request URL at all (please correct me if I am wrong). This may imply that there is no [pressing] need to store the rewritten URL either (or at least we should have a clear understanding of why it needs to be stored, and storing it may be viewed as a separate project/improvement). so I think the duplication was to preserve this structure and prevent major api changes. as far I get into the depth of the code I see how it's reasonable to make this decision if you compare the loss and benefits . loosing a few single(sometimes it's not) bytes of GB is cheap compared to depth development time. I do not know what benefits you are talking about. Why do we need to store the rewritten URL? In other words, what will break if we do not store the rewritten URL? Again, I am not saying that storing URLs is wrong -- I just want to understand why we need to do that. One possible use is consistency checks. If we store the request URL, the hit serving code can double check that the current request URL is the same as the stored request URL. However, those checks do not explain why we need to store two URLs (rewritten and original) and they should be viewed as a separate improvement/project outside of your work scope. Store can get a list of cached objects by iterating through store_table and other store indexes. In general, you should not assume that it is possible to get a list of all cached URLs in any efficient/practical fashion because not all in-RAM indexes store URLs. It is only possible to get an answer to the following question: * Is a response with cache key K likely to be in Squid cache now? Where cache key is a hash computed over the request method, request URI, and other properties. this is what I remember. so for the new development there should be an option to do that but also do the rewriting of the url before checking that or more practically to change the cache key calculations if there is a store_url present for the request. I am not sure what option you are referring to in the above. The Store::get(key) API I have described is not optional -- it is the primary way of detecting a hit. The URL rewriting must happen before or during Store key calculation. I do have an approach that I want to check and Its' based on finding all\specific mem_object operations and object creation in the code. now I'm struggling to juggle find and mark the points. answering the double url thing In 2.7 the feature used the storeswap_meta to add the storeurl string and it's simplified things. Yes, if we need to store URL(s), we should use the StoreMeta API. Why do we need to store URL(s)? I noticed STORE_META_STOREURL still in the TLV headers (probably to support older cache object structure versions) so i will try to use it.. for some testing purposes. STORE_META_STOREURL is unused in Squid3 AFAICT. I have tested and added basic tests to make sure that the storeurl is being written and used and it's not hearts the any current cache objects or stuff. any thoughts? Why do we need to store it? Thank you, Alex.
Re: Store_url_rewrite for squid 3+
On 09/07/2012 05:10 PM, Alex Rousskov wrote:
I am not sure what option you are referring to in the above. The
Store::get(key) API I have described is not optional -- it is the
primary way of detecting a hit.
+1 to use the api.
are we talking about the Store::Root().get ?
I was just getting into it now and it seems to confused me a bit.
and I found my answer for couple things about purge and other stuff on
the way.
as I see if we want to make it work we need to take another logic then
then in 2.7 (obviously).
I will check it and make sure I can make it stick and not break anything
in *store* on the way.
The URL rewriting must happen before or during Store key calculation.
The problem is that if I am rewriting the url, unlike
url_rewrite\redirect the connection should be initiated based on the
original url and not the rewritten one.
What I do not know yet is what and where and based on what the request
is done.
if it's based on the original request or the url.
so we need to store the original url and the store_url at least for the
period of time the request is being served.(answer about storing or not
the store_url in meta)
url_rewrite api gets the http requests as the background to any action
so it cannot be used for store_url since any change to the request will
lead to the change we are trying to not do exactly.
mem_object contains:
{...
char *url;
HttpRequest *request;
...}
and the url is being used to make the hash but I still dont know what is
being taken from the mem_object to do the request.
when I will know what exact object and exact point in code is being used
to fetch the request I think I can be more clever on a way to implement
my idea.
I had the end of the redirector but I lost it somewhere in couple too
much K*lines.
STORE_META_STOREURL is unused in Squid3 AFAICT.
Ok, but it still there and can be used with just 2-3 lines of code.
I have tested it and the data is written if needed but is not in use
when reading from store.
Why do we need to store it?
I do not know why we need to store it and i think that the only things
that should be saved is the full request full reply and some data\meta
the being will be used by the replacement policy.
since I dont know how everything works and about all the api in the code
after reviewing the 2.7 code related to store_url it's seems like the
approach wasn't that bad using the meta data.
since couple hours ago I noticed that it was done in order to make
things possible and not really to think in a more wide angle about the
effect of it.
the main problem I think the STORE_META_STOREURL was maybe used for is
if and while rebuilding the cache_dir data to be able not pass the url
to the store_url helper.
from this point I think it's costs space but when squid rebuilds the
cache_dir I still dont now if it's good to pass url into the helper.
And for the question that pops out: if a cache_dir swap.data get's
corrupted, what the fate of the cache in the squid3.head ? (I have never
had such a bad luck to be able feel what it's like).
Eliezer
Re: Store_url_rewrite for squid 3+
On 09/07/2012 01:32 PM, Eliezer Croitoru wrote:
On 09/07/2012 05:10 PM, Alex Rousskov wrote:
I am not sure what option you are referring to in the above. The
Store::get(key) API I have described is not optional -- it is the
primary way of detecting a hit.
+1 to use the api.
are we talking about the Store::Root().get ?
That is where it starts, but other Store kids implement the get() method
as well IIRC. From the caller perspective, you should also look at
storeGetPublic*() and similar functions.
The URL rewriting must happen before or during Store key calculation.
The problem is that if I am rewriting the url, unlike
url_rewrite\redirect the connection should be initiated based on the
original url and not the rewritten one.
Sure, but that is not a problem you need to worry about much. Since you
are not going to alter the original URL, the connection initiation code
will not be affected by your changes (except for some minor adjustments
if some data members get renamed or Store::get() API-related calls get
adjusted as discussed below).
What I do not know yet is what and where and based on what the request
is done.
For this project, you do not need to know much about connection
initiation code. You need to make a copy of the original URL, rewrite
that copy using the new helper, and put it in a new HttpRequest data
member (for example). This should probably happen (or be initiated from)
client_side.cc or client_side_request.cc.
You then need to make sure that every time Store calculates a store key,
the new member (if any) is used instead of the original URL.
Since the current code is using the original request URL where you want
the rewritten/store URL to be used, I recommend _renaming_ all members
that keep or extract the original/request URL (e.g., MemObject::url).
This will force you to go through the relevant callers and make sure
that the right URL is used, depending on the caller context.
Once you are done, we can review whether the vast majority of changes in
the diff are a simple renaming of url to requestUrl. If yes, you can
revert that renaming change. If there are many cases where old url
became storeUrl and many cases where it became requestUrl then the
change should stay.
Please note that the above is just a sketch. I am not suggesting any
specific new names, and I am using existing class names only as an example.
so we need to store the original url and the store_url at least for the
period of time the request is being served.
Yes, we need to keep both URLs around during the transaction lifetime.
When I was talking about storing a URL, I was only talking about Store
(i.e., storing something in the cache).
(answer about storing or not the store_url in meta)
That is a separate question. I cannot answer it, unfortunately. I do not
know why Squid2 stored either of those URLs. I know Squid3 does not
store the original/request URL.
url_rewrite api gets the http requests as the background to any action
so it cannot be used for store_url since any change to the request will
lead to the change we are trying to not do exactly.
mem_object contains:
{...
char *url;
HttpRequest *request;
...}
and the url is being used to make the hash but I still dont know what is
being taken from the mem_object to do the request.
HttpStateData::buildRequestPrefix() builds request headers. It may use
mem_object::url. I do not think you should change that code (apart from
renaming and Store::Get() API changes discussed elsewhere).
You should only be concerned with calls to storeGetPublic*() and such.
Make sure the callers do not use mem_object::url without checking
whether there is a rewritten store URL as well. You will probably want
to change profile of most of those calls so that the caller does not
need to worry about picking the right URL to use (the decision would
happen inside storeGetPublic() and friends).
Why do we need to store it?
I do not know why we need to store it and i think that the only things
that should be saved is the full request full reply and some data\meta
the being will be used by the replacement policy.
since I dont know how everything works and about all the api in the code
after reviewing the 2.7 code related to store_url it's seems like the
approach wasn't that bad using the meta data.
How is that metadata used in Squid2? What is it used for? Nobody is
suggesting that Squid2 does something wrong here, of course. We just
need to understand _what_ it does so that we can either mimic what it
does in Squid3 or do something better.
since couple hours ago I noticed that it was done in order to make
things possible and not really to think in a more wide angle about the
effect of it.
What things does it make possible?
I hope we will not have to commit some code without understanding its
effects.
the main problem I think the STORE_META_STOREURL was maybe used for is
if and while rebuilding the cache_dir data to
Re: Store_url_rewrite for squid 3+
On 8/09/2012 7:38 a.m., Eliezer Croitoru wrote:
On 09/07/2012 05:10 PM, Alex Rousskov wrote:
I am not sure what option you are referring to in the above. The
Store::get(key) API I have described is not optional -- it is the
primary way of detecting a hit.
+1 to use the api.
are we talking about the Store::Root().get ?
I was just getting into it now and it seems to confused me a bit.
and I found my answer for couple things about purge and other stuff on
the way.
as I see if we want to make it work we need to take another logic then
then in 2.7 (obviously).
I will check it and make sure I can make it stick and not break
anything in *store* on the way.
The URL rewriting must happen before or during Store key calculation.
The problem is that if I am rewriting the url, unlike
url_rewrite\redirect the connection should be initiated based on the
original url and not the rewritten one.
What I do not know yet is what and where and based on what the request
is done.
if it's based on the original request or the url.
so we need to store the original url and the store_url at least for
the period of time the request is being served.(answer about storing
or not the store_url in meta)
Correct. Also, any revalidation requests done later must be done on the
original request URL. Not the stored URL nor the potentially different
current client request URL.
Thus we need to store the object with key being the store-url and
preserve the original for server-side use.
url_rewrite api gets the http requests as the background to any action
so it cannot be used for store_url since any change to the request
will lead to the change we are trying to not do exactly.
mem_object contains:
{...
char *url;
HttpRequest *request;
...}
and the url is being used to make the hash but I still dont know what
is being taken from the mem_object to do the request.
when I will know what exact object and exact point in code is being
used to fetch the request I think I can be more clever on a way to
implement my idea.
I had the end of the redirector but I lost it somewhere in couple too
much K*lines.
STORE_META_STOREURL is unused in Squid3 AFAICT.
Ok, but it still there and can be used with just 2-3 lines of code.
I have tested it and the data is written if needed but is not in use
when reading from store.
Why do we need to store it?
I do not know why we need to store it and i think that the only things
that should be saved is the full request full reply and some data\meta
the being will be used by the replacement policy.
since I dont know how everything works and about all the api in the
code after reviewing the 2.7 code related to store_url it's seems like
the approach wasn't that bad using the meta data.
since couple hours ago I noticed that it was done in order to make
things possible and not really to think in a more wide angle about the
effect of it.
the main problem I think the STORE_META_STOREURL was maybe used for is
if and while rebuilding the cache_dir data to be able not pass the
url to the store_url helper.
from this point I think it's costs space but when squid rebuilds the
cache_dir I still dont now if it's good to pass url into the helper.
It will be costly to process maybe several millions of objects through
the helper on startup/restart.
And for the question that pops out: if a cache_dir swap.data get's
corrupted, what the fate of the cache in the squid3.head ? (I have
never had such a bad luck to be able feel what it's like).
Amos
Re: Store_url_rewrite for squid 3+, on what branch to start working?
On 6/09/2012 11:00 p.m., Eliezer Croitoru wrote: I had a *smalll* issue with my storage so I'm forced to start from almost scratch. I previously worked on the 3.2.1 latest stable sources and I am wondering on where to start now? start on head? stable? Eliezer 3.HEAD please. Amos
Re: Store_url_rewrite for squid 3+, on what branch to start working?
I had a *smalll* issue with my storage so I'm forced to start from almost scratch. I previously worked on the 3.2.1 latest stable sources and I am wondering on where to start now? start on head? stable? Eliezer
Re: Store_url_rewrite for squid 3+
On 09/05/2012 06:58 PM, Amos Jeffries wrote: On 06.09.2012 11:58, Eliezer Croitoru wrote: We can pause there for the infrastructure to look fine before moving on to the store details. I've been waiting on assistance from Henrik or Alex on that for a while. They are the ones who know the answers to your questions below AFAIK. FWIW, I have not reviewed the store_url_rewrite code in Squid2 so I cannot answer the questions related to how it was done. I can suggest ways of doing this in Squid3, but since somebody already investigated all the alternatives, it would be better to hear the summary of the Squid2 implementation (as it relates to Store) before diving into Squid3 development. The biggest question for me is why Squid2 code was storing multiple URLs with the cached object (if it was). Why cannot Store just work with the [rewritten] URL given to it and ignore the fact that some [store] URLs originated from some other [real] URLs? Are we trying to support going from a store_url_rewrite config back to regular config without losing some of the cached objects? 2. Research the workflow of storing objects in memory and store and introduce psudo for a new workflow of storing objects to avoid bad effects on cache objects usage in any form that can be. - I do know that squid uses some hash look-up and I have seen in the things about it. - as far I understood from the code: client_request builds the request of the http object. creates a mem-object and on the way creates a checksum. a transfer from of the mem-object to a store happens. if a store rebuild happens it takes all of the data from the file in the store. ? question how cachemgr gets the list of urls in memory? You might be confusing cache manager (the thing that responds to squidclient mgr:info requests) with Store. Also, you should not think in terms of memory (RAM) because some objects are only cached on disk. It is best to think of Store as a collection of stored objects, ignoring their particular location (memory or disk) to the extent possible. Store can get a list of cached objects by iterating through store_table and other store indexes. In general, you should not assume that it is possible to get a list of all cached URLs in any efficient/practical fashion because not all in-RAM indexes store URLs. It is only possible to get an answer to the following question: * Is a response with cache key K likely to be in Squid cache now? Where cache key is a hash computed over the request method, request URI, and other properties. I will look at it later but if someone have solid knowledge on how the store routing was or implemented before i'm rushing into the code every piece of info will help me when looking into it. The Store is too big and complex of an API to accurately describe in an email IMO. I would be happy to answer specific questions about the stuff I know, but you may have to research how things work as there is no comprehensive documentation yet. Another complication is that such fundamental Squid2 Store feature as store_table needs to be removed but it has not been completely removed from Squid3 yet, so there is some [older] code that relies on it and some [newer] code that tries hard to stay away from it, all while doing the same kind of operations. Finally, the whole Store class hierarchy is ugly to a fault. It needs to be split into more independent classes instead of everybody and the kitchen sink inheriting from Store, hiding the intended boundaries among store manager, memory storage manager, disk storage manager, cache_dir manager, etc. Good luck, Alex.
Re: Store_url_rewrite for squid 3+
On 9/5/2012 8:00 AM, Amos Jeffries wrote:
On 5/09/2012 4:10 p.m., Eliezer Croitoru wrote:
I'm reading some code (will take a while) to maybe get a functional
store_url_rewrite for squid3+.
Actually i was thinking about it a lot and the process should be very
simple:
use some stdin+stdout like for url_rewrite interface for starter.
Yes. The redirect.cc interface can be used. The caller code runs
*_access checks and provides a callback function which uses the
resulting URL to do things to store keys.
i think this is can be done pretty fast if someone knows the current
code.
it's kind of replicate the url_rewrite and change all the directives
names from url_rewrite to store_url_rewrite and let someone write the
functions.
how is this as a starter to maybe make it work?
Exactly what I was planning. Just a lack of store code knowledge getting
in the way.
It should be done before the request is being done and i think that
before url_rewrite helper.
At the point where cache key is being decided I think. Which is just
before HIT lookup. After adaptations like url_rewrite.
the next step after that is to take the to be cached entry and
change the the key of the stored object. as it will be stored at in
the cache and not like the old store_url_rewrite that was saved in a
formart of old+new url.
how about just start with the basics of putting the whole redirector
as in with store_url_rewrite?
how hard can it be?
Want to give it a try?
Well I would like to but it seems i'm lacking of knowledge in c\c++ and
squid structure.
I do recognize that there is a language there with some structure since
I did wrote using java and ruby.
(just wrote a small proxy for specific protocol that peers two tcp
incoming connections after verification of identity and it seems to rock)
I was thinking to start with creating a store_url_rewrite (fake) that
will debug and all but will not do any changes except logging.
If I\we can make this stage it will benefit anyone who will want to
integrate some new interface for squid.
So I started playing with the sources but since squid is a big puzzle
for me I dont know where and what I should put or look for pieces of
code related to the redirect.cc .
what I did as a starter is to take redirect.cc (3.2.1)and change all the
methods names I could find\think off to a srewriter syntax such as:
redirectRegisterWithCacheManager();
to
srewriteRegisterWithCacheManager();
file and diff attached.
after that I tried to find anything related to redirect.cc in the
makefiles and have seen:
at:
./src/Makefile.am
./src/Makefile.in
so i suppose i should add to every place of redirect.cc also srewrite.cc
(but i'm dont know...)
I didnt made any changed to the srewrite.cc that will make it harmless
and fake yet.
now the hard part for me is: how would I find any redirect.cc in the
rest of squid and make sure to add the needed code for srewrite.cc?
I can parse the srewrite.cc but since I dont know any of the code
structure I wouldnt know what to extract and look for.
I noticed kinkie did pretty things while parsing the code.
I took a peer at the old store_url_rewrite and it seems to be divided
into client_side_storeurl_rewrite.c, store_rewrite.c
and has some struct in structs.h for children concurrency and command.
On 3.2.1 it's seems to be organized better and only in couple files such
as the redirect.cc and others.
any leads,?
Eliezer
--
Eliezer Croitoru
https://www1.ngtech.co.il
IT consulting for Nonprofit organizations
eliezer at ngtech.co.il
#diff redirect.cc srewriter.cc
5c5
* DEBUG: section 61Redirector
---
* DEBUG: section 61Srewriter
55c55
/// url maximum lengh + extra informations passed to redirector
---
/// url maximum lengh + extra informations passed to srewriter
66c66
} redirectStateData;
---
} srewriterStateData;
68,71c68,71
static HLPCB redirectHandleReply;
static void redirectStateFree(redirectStateData * r);
static helper *redirectors = NULL;
static OBJH redirectStats;
---
static HLPCB srewriteHandleReply;
static void srewriterStateFree(srewriterStateData * r);
static helper *srewriters = NULL;
static OBJH srewriterStats;
73c73
CBDATA_TYPE(redirectStateData);
---
CBDATA_TYPE(srewriterStateData);
76c76
redirectHandleReply(void *data, char *reply)
---
srewriteHandleReply(void *data, char *reply)
78c78
redirectStateData *r = static_castredirectStateData *(data);
---
srewriterStateData *r = static_castsrewriterStateData *(data);
81c81
debugs(61, 5, redirectHandleRead: { (reply *reply != '\0' ? reply
: NULL) });
---
debugs(61, 5, srewriteHandleRead: { (reply *reply != '\0' ? reply
: NULL) });
94c94
redirectStateFree(r);
---
srewriterStateFree(r);
98c98
redirectStateFree(redirectStateData * r)
---
srewriterStateFree(srewriterStateData * r)
105c105
redirectStats(StoreEntry * sentry)
---
srewriterStats(StoreEntry * sentry)
107,108c107,108
if (redirectors == NULL) {
Re: Store_url_rewrite for squid 3+
On 9/5/2012 9:56 AM, Eliezer Croitoru wrote: any leads,? Well there is a nice progress. I reviewed the 2.7 store_url_rewrite and I divided the task into more detailed smaller tasks. 1. Research the url_rewrite interface code and Introduce a modified version of url_rewrite as url_store_rewrite_program. - this task is kind of done(passed compiling and running on 3.2.1) by now and I want to get some ideas on naming conventions for the code to fit the project amazing code looks. list of changed files and code: create a mimic file of redirect.cc in ./store_rewrite.cc and change all the methods and variables to fit store_rewrite. strip all the url_rewrite data manipulating actions. change the debugging info. (after the store related planning tasks get back here to redo) ./structs.h adding the proper variables for: helper naming, bypass(on\off), acl_access namespace, child configs the ??_rewrites_host of url_rewrite dosnt belong for store_rewrite process at all. ./cache_cf.cc state the default configuration for the helper ./cf.data.pre stating all config directives for the the helper (copy and modify from url_rewrite_program) ./ClientRequestContext.h adding int for state adding bool for done ./client_side_request.h stating the start method as squidexternal something ./client_side_request.cc adding calls and callouts ./protos.h stating the start init and shutdown methods. ./main.cc: calling init and shutdown methods at start/reconfigure etc.. ./Makefile.am ./Makefile.in adding the source ?.cc file into the commands 2. Research the workflow of storing objects in memory and store and introduce psudo for a new workflow of storing objects to avoid bad effects on cache objects usage in any form that can be. - I do know that squid uses some hash look-up and I have seen in the things about it. - as far I understood from the code: client_request builds the request of the http object. creates a mem-object and on the way creates a checksum. a transfer from of the mem-object to a store happens. if a store rebuild happens it takes all of the data from the file in the store. ? question how cachemgr gets the list of urls in memory? so probable points of failure: using the wrong url to fetch the object. wrong arguments for checksum. storing with wrong arguments\url leading to faulty rebuild. I do remember that when I looked at a stored old store_url_rewrite cahce file long time ago there were two urls in the file what leads me (it's a bit fogy) to think that the stored file was the memobject cache rather then a set of arguments such as refresh time related info,method,url,request,response. I will look at it later but if someone have solid knowledge on how the store routing was or implemented before i'm rushing into the code every piece of info will help me when looking into it. Eliezer -- Eliezer Croitoru https://www1.ngtech.co.il IT consulting for Nonprofit organizations eliezer at ngtech.co.il
Re: Store_url_rewrite for squid 3+
On 06.09.2012 11:58, Eliezer Croitoru wrote: On 9/5/2012 9:56 AM, Eliezer Croitoru wrote: any leads,? Well there is a nice progress. I reviewed the 2.7 store_url_rewrite and I divided the task into more detailed smaller tasks. FTR: squid-2.7 ports are exempted as suitable in most cases for back-porting to stable despite our no new features policy. I am happy for this to be done as a series of patches instead of a singular change. It can be assembled in trunk and back-ported as a singular later. 1. Research the url_rewrite interface code and Introduce a modified version of url_rewrite as url_store_rewrite_program. - this task is kind of done(passed compiling and running on 3.2.1) by now and I want to get some ideas on naming conventions for the code to fit the project amazing code looks. list of changed files and code: create a mimic file of redirect.cc in ./store_rewrite.cc and change No need. What I meant earlier about src/redirect.cc being usable is that most of the code is an exact duplicate. You should only need to: * write a new start() function ie storeurlRewriteStart() * add new storeurl_rewriters global * adapt redirectRegisterWithCacheManager() to register a new storeurl_rewrite report (when that part is written) * adapt redirectInit() to setup both url_rewrite and storeurl-rewrite helpers. * adapt redirectShutdown() to cleanup both url_rewrite and storeurl-rewrite helpers. The new storeurlRewriteStart() to be used by store code for this re-writing and sets up the redirect interface using the new store helpers and whatever callback the changed URL is to be sent to. The entire rest of the code for helper management is identical. all the methods and variables to fit store_rewrite. strip all the url_rewrite data manipulating actions. change the debugging info. (after the store related planning tasks get back here to redo) ./structs.h adding the proper variables for: helper naming, bypass(on\off), acl_access namespace, child configs the ??_rewrites_host of url_rewrite dosnt belong for store_rewrite process at all. ./cache_cf.cc state the default configuration for the helper What do you mean by this? doConfigure() post-configuration checking? I don't think there is anything which needs new cache_cf.cc code. The parsing side if things is identical for url_rewrite_*. The different defaults and locations are all coded in cf.data.pre ... ./cf.data.pre stating all config directives for the the helper (copy and modify from url_rewrite_program) Okay. However, if merging the stages to trunk separately this will need to be the final step, since it makes the directives publicly visible. We want to make these changes and doc/release-notes/release-3.*.sgml at the same time when it is suitable for public use. Also, remove the storeurl_* entries at the top marking them as obsolete/unavilable type. ./ClientRequestContext.h adding int for state adding bool for done ./client_side_request.h stating the start method as squidexternal something Just extern will do. We are killing the SQUIDCEXTERN mess. ./client_side_request.cc adding calls and callouts ./protos.h stating the start init and shutdown methods. We are in the process of killing protos.h. Please create a store_urlrewrite.h header for these definitions instead. However, see the comments above about redirect.cc. There should not need to be any new files created for this. Which removes the protos.h, main.cc and Makefile.am changes. ./main.cc: calling init and shutdown methods at start/reconfigure etc.. ./Makefile.am ./Makefile.in adding the source ?.cc file into the commands Other than the notes above. Okay. If you have a patch for that please submit for audit :-) We can pause there for the infrastructure to look fine before moving on to the store details. I've been waiting on assistance from Henrik or Alex on that for a while. They are the ones who know the answers to your questions below AFAIK. 2. Research the workflow of storing objects in memory and store and introduce psudo for a new workflow of storing objects to avoid bad effects on cache objects usage in any form that can be. - I do know that squid uses some hash look-up and I have seen in the things about it. - as far I understood from the code: client_request builds the request of the http object. creates a mem-object and on the way creates a checksum. a transfer from of the mem-object to a store happens. if a store rebuild happens it takes all of the data from the file in the store. ? question how cachemgr gets the list of urls in memory? so probable points of failure: using the wrong url to fetch the object. wrong arguments for checksum. storing with wrong arguments\url leading to faulty rebuild. I do remember that when I looked at a stored old store_url_rewrite cahce file long time ago there were two urls in the file what leads me (it's a bit fogy) to think that the stored file was the memobject
Re: Store_url_rewrite for squid 3+
OK, it seems we are getting to somewhere. i know how to patch using command but what are the proper one to get a patch file to be run later? will look into it. Eliezer On 9/6/2012 3:58 AM, Amos Jeffries wrote: On 06.09.2012 11:58, Eliezer Croitoru wrote: On 9/5/2012 9:56 AM, Eliezer Croitoru wrote: any leads,? Well there is a nice progress. I reviewed the 2.7 store_url_rewrite and I divided the task into more detailed smaller tasks. FTR: squid-2.7 ports are exempted as suitable in most cases for back-porting to stable despite our no new features policy. I am happy for this to be done as a series of patches instead of a singular change. It can be assembled in trunk and back-ported as a singular later. 1. Research the url_rewrite interface code and Introduce a modified version of url_rewrite as url_store_rewrite_program. - this task is kind of done(passed compiling and running on 3.2.1) by now and I want to get some ideas on naming conventions for the code to fit the project amazing code looks. list of changed files and code: create a mimic file of redirect.cc in ./store_rewrite.cc and change No need. What I meant earlier about src/redirect.cc being usable is that most of the code is an exact duplicate. You should only need to: * write a new start() function ie storeurlRewriteStart() * add new storeurl_rewriters global * adapt redirectRegisterWithCacheManager() to register a new storeurl_rewrite report (when that part is written) * adapt redirectInit() to setup both url_rewrite and storeurl-rewrite helpers. * adapt redirectShutdown() to cleanup both url_rewrite and storeurl-rewrite helpers. fair The new storeurlRewriteStart() to be used by store code for this re-writing and sets up the redirect interface using the new store helpers and whatever callback the changed URL is to be sent to. The entire rest of the code for helper management is identical. all the methods and variables to fit store_rewrite. strip all the url_rewrite data manipulating actions. change the debugging info. (after the store related planning tasks get back here to redo) ./structs.h adding the proper variables for: helper naming, bypass(on\off), acl_access namespace, child configs the ??_rewrites_host of url_rewrite dosnt belong for store_rewrite process at all. ./cache_cf.cc state the default configuration for the helper What do you mean by this? doConfigure() post-configuration checking? I don't think there is anything which needs new cache_cf.cc code. The parsing side if things is identical for url_rewrite_*. The different defaults and locations are all coded in cf.data.pre ... ok so later we will see how ot optimize the conf file. but there are coupe arguments there that are crucial for compilation and parsing the config file ./cf.data.pre stating all config directives for the the helper (copy and modify from url_rewrite_program) Okay. However, if merging the stages to trunk separately this will need to be the final step, since it makes the directives publicly visible. We want to make these changes and doc/release-notes/release-3.*.sgml at the same time when it is suitable for public use. Also, remove the storeurl_* entries at the top marking them as obsolete/unavilable type. ./ClientRequestContext.h adding int for state adding bool for done ./client_side_request.h stating the start method as squidexternal something Just extern will do. We are killing the SQUIDCEXTERN mess. ./client_side_request.cc adding calls and callouts ./protos.h stating the start init and shutdown methods. We are in the process of killing protos.h. Please create a store_urlrewrite.h header for these definitions instead. will be done However, see the comments above about redirect.cc. There should not need to be any new files created for this. Which removes the protos.h, main.cc and Makefile.am changes. ./main.cc: calling init and shutdown methods at start/reconfigure etc.. ./Makefile.am ./Makefile.in adding the source ?.cc file into the commands Other than the notes above. Okay. If you have a patch for that please submit for audit :-) We can pause there for the infrastructure to look fine before moving on to the store details. I've been waiting on assistance from Henrik or Alex on that for a while. They are the ones who know the answers to your questions below AFAIK. 2. Research the workflow of storing objects in memory and store and introduce psudo for a new workflow of storing objects to avoid bad effects on cache objects usage in any form that can be. - I do know that squid uses some hash look-up and I have seen in the things about it. - as far I understood from the code: client_request builds the request of the http object. creates a mem-object and on the way creates a checksum. a transfer from of the mem-object to a store happens. if a store rebuild happens it takes all of the data from the file in the store. ? question how cachemgr gets the list of urls in memory? so
Re: Store_url_rewrite for squid 3+
On 06.09.2012 13:23, Eliezer Croitoru wrote: OK, it seems we are getting to somewhere. i know how to patch using command but what are the proper one to get a patch file to be run later? will look into it. If you are using diff: diff -u orig_code/ new_code/ output_file.patch If you are using bzr: bzr diff output_file.patch Amos
Re: Store_url_rewrite for squid 3+
On 9/6/2012 4:37 AM, Amos Jeffries wrote: On 06.09.2012 13:23, Eliezer Croitoru wrote: OK, it seems we are getting to somewhere. i know how to patch using command but what are the proper one to get a patch file to be run later? will look into it. If you are using diff: diff -u orig_code/ new_code/ output_file.patch If you are using bzr: bzr diff output_file.patch Amos Thanks, and I hoped to sleep but i'm not tired :( anyway I will post couple patches later with the basic thing. i'm now mangling my working version into the ridrect.cc file and moving on. -- Eliezer Croitoru https://www1.ngtech.co.il IT consulting for Nonprofit organizations eliezer at ngtech.co.il
Store_url_rewrite for squid 3+
I'm reading some code (will take a while) to maybe get a functional store_url_rewrite for squid3+. Actually i was thinking about it a lot and the process should be very simple: use some stdin+stdout like for url_rewrite interface for starter. i think this is can be done pretty fast if someone knows the current code. it's kind of replicate the url_rewrite and change all the directives names from url_rewrite to store_url_rewrite and let someone write the functions. how is this as a starter to maybe make it work? It should be done before the request is being done and i think that before url_rewrite helper. the next step after that is to take the to be cached entry and change the the key of the stored object. as it will be stored at in the cache and not like the old store_url_rewrite that was saved in a formart of old+new url. how about just start with the basics of putting the whole redirector as in with store_url_rewrite? how hard can it be? Thanks, Eliezer -- Eliezer Croitoru https://www1.ngtech.co.il IT consulting for Nonprofit organizations eliezer at ngtech.co.il
Re: Store_url_rewrite for squid 3+
On 5/09/2012 4:10 p.m., Eliezer Croitoru wrote: I'm reading some code (will take a while) to maybe get a functional store_url_rewrite for squid3+. Actually i was thinking about it a lot and the process should be very simple: use some stdin+stdout like for url_rewrite interface for starter. Yes. The redirect.cc interface can be used. The caller code runs *_access checks and provides a callback function which uses the resulting URL to do things to store keys. i think this is can be done pretty fast if someone knows the current code. it's kind of replicate the url_rewrite and change all the directives names from url_rewrite to store_url_rewrite and let someone write the functions. how is this as a starter to maybe make it work? Exactly what I was planning. Just a lack of store code knowledge getting in the way. It should be done before the request is being done and i think that before url_rewrite helper. At the point where cache key is being decided I think. Which is just before HIT lookup. After adaptations like url_rewrite. the next step after that is to take the to be cached entry and change the the key of the stored object. as it will be stored at in the cache and not like the old store_url_rewrite that was saved in a formart of old+new url. how about just start with the basics of putting the whole redirector as in with store_url_rewrite? how hard can it be? Want to give it a try? Amos
Re: [PATCH] Solaris /dev/poll support for Squid 3 (how can I contribute)
On 10/19/10 18:40, Alex Rousskov wrote: On 10/13/2010 04:21 AM, Peter Payne wrote: BENCHMARKS Firstly, I wanted to address the benchmark questions as it made me curious as to whether there really was an advantage in using /dev/poll. I used the Apache Bench tool (that comes with HTTPD) to do my benchmarks. I compiled a 32-bit Solaris 5.10 x86 of Squid version 3.1.8, and another Squid version 3.1.8 with my patches. I shall call them unpatched Squid and patched Squid respectively. Note that unpatched Squid was using ordinary poll() and patched Squid was using /dev/poll. Where I used Apache bench with 1 simultaneous connection or even 8,000 simultaneous connections there was little difference between unpatched Squid and patched Squid. When all connections are busy there is no performance gain. However the test that proved the most striking difference was pre-establishing 8,000 TCP socket connections to Squid using a basic Perl script. Then running Apache Bench with 1 connection making 100,000 keep-alive'd requests. unpatched Squid (using poll): 2min23sec CPU user-time 171sec to process 10 Apache Bench queries patched Squid (using /dev/poll): 0min22sec CPU user-time 25sec to process 10 Apache Bench queries Clearly /dev/poll has a significant performance gain where there are many idle TCP socket connections. On 10/13/2010 04:25 PM, Amos Jeffries wrote: Oooh... Speed. Not really, at least not if you define performance or speed as ability to sustain an X requests/second load. The above test is a classic best-effort test that, by design, cannot measure proxy's ability to handle load because its offered request rate depends on measured response time. During such a test, Squid drives the benchmark rather than the benchmark driving Squid. There are many real-world examples where a faster product (for any practical definition of faster) would show drastically worse results during such a test. What the test does tell you is that an idle Squid became faster at processing a single transaction after the patch. Significant performance gain and similar conclusions are premature after a best-effort test like this one. Please do _not_ interpret my comments as negative towards the patch itself. I just want to minimize the chance that others will start running similar tests and jumping to the wrong conclusions regarding their own optimization ideas. Thank you, Alex. Alex, I completely agree. During testing there was little or no performance gain when processing n busy connections (all busy). The only performance benefit arose out of servicing few busy connections when many were idle. As for the realism or effectiveness in the real world it may be negligible. However there are situations when a connection is idle: when waiting for a proxy to fetch a page from somewhere else, when holding a connection open using keep-alive. At any rate, it was a desired feature on the Squid 3 roadmap and was already provided for Squid 2, and given that epoll and kqueue support exists, it is added for completeness (with a LOT more comments in the source too, initially I found reading the older code harder going). Thanks, Peter
Re: [PATCH] Solaris /dev/poll support for Squid 3 (how can I contribute)
On 10/13/2010 04:21 AM, Peter Payne wrote: BENCHMARKS Firstly, I wanted to address the benchmark questions as it made me curious as to whether there really was an advantage in using /dev/poll. I used the Apache Bench tool (that comes with HTTPD) to do my benchmarks. I compiled a 32-bit Solaris 5.10 x86 of Squid version 3.1.8, and another Squid version 3.1.8 with my patches. I shall call them unpatched Squid and patched Squid respectively. Note that unpatched Squid was using ordinary poll() and patched Squid was using /dev/poll. Where I used Apache bench with 1 simultaneous connection or even 8,000 simultaneous connections there was little difference between unpatched Squid and patched Squid. When all connections are busy there is no performance gain. However the test that proved the most striking difference was pre-establishing 8,000 TCP socket connections to Squid using a basic Perl script. Then running Apache Bench with 1 connection making 100,000 keep-alive'd requests. unpatched Squid (using poll): 2min23sec CPU user-time 171sec to process 10 Apache Bench queries patched Squid (using /dev/poll): 0min22sec CPU user-time 25sec to process 10 Apache Bench queries Clearly /dev/poll has a significant performance gain where there are many idle TCP socket connections. On 10/13/2010 04:25 PM, Amos Jeffries wrote: Oooh... Speed. Not really, at least not if you define performance or speed as ability to sustain an X requests/second load. The above test is a classic best-effort test that, by design, cannot measure proxy's ability to handle load because its offered request rate depends on measured response time. During such a test, Squid drives the benchmark rather than the benchmark driving Squid. There are many real-world examples where a faster product (for any practical definition of faster) would show drastically worse results during such a test. What the test does tell you is that an idle Squid became faster at processing a single transaction after the patch. Significant performance gain and similar conclusions are premature after a best-effort test like this one. Please do _not_ interpret my comments as negative towards the patch itself. I just want to minimize the chance that others will start running similar tests and jumping to the wrong conclusions regarding their own optimization ideas. Thank you, Alex.
Re: [PATCH] Solaris /dev/poll support for Squid 3 (how can I contribute)
On 14/10/10 13:58, Henrik Nordström wrote: ons 2010-10-13 klockan 22:25 + skrev Amos Jeffries: +1 from me with merge tweaks. Unless anyone has objections I will commit with tweaks at the next opportunity. No objection from me. But have not reviewed the changes outside comm_devpoll.cc. Regards Henrik The rest being minimal basic plumbing. I'm taking this as a second vote. :) Applied to Squid-3 as r10946. Thank you Peter. Amos -- Please be using Current Stable Squid 2.7.STABLE9 or 3.1.8 Beta testers wanted for 3.2.0.2
Re: [PATCH] Solaris /dev/poll support for Squid 3 (how can I contribute)
Hello Amos, in response to your comments. BENCHMARKS Firstly, I wanted to address the benchmark questions as it made me curious as to whether there really was an advantage in using /dev/poll. I used the Apache Bench tool (that comes with HTTPD) to do my benchmarks. I compiled a 32-bit Solaris 5.10 x86 of Squid version 3.1.8, and another Squid version 3.1.8 with my patches. I shall call them unpatched Squid and patched Squid respectively. Note that unpatched Squid was using ordinary poll() and patched Squid was using /dev/poll. Where I used Apache bench with 1 simultaneous connection or even 8,000 simultaneous connections there was little difference between unpatched Squid and patched Squid. When all connections are busy there is no performance gain. However the test that proved the most striking difference was pre-establishing 8,000 TCP socket connections to Squid using a basic Perl script. Then running Apache Bench with 1 connection making 100,000 keep-alive'd requests. unpatched Squid (using poll): 2min23sec CPU user-time 171sec to process 10 Apache Bench queries patched Squid (using /dev/poll): 0min22sec CPU user-time 25sec to process 10 Apache Bench queries Clearly /dev/poll has a significant performance gain where there are many idle TCP socket connections. I have now also optimised the comm_devpoll.cc routine to ensure unnecessary calls to /dev/poll with the POLLREMOVE flag were removed. CODE AUDIT Most of the style used was a result of directly copying the styles used in the existing comm_devpoll.c from Squid v2.7 and comm_epoll.cc from Squid v3.1.8. However where these conflict with the requests made most recently I will adopt the recommendations in the e-mail. 1. used #if USE_DEVPOLL instead of #ifdef 2. moved #includes except squid.h inside USE_DEVPOLL wrapper 3. moved comment to own line 4. removed big XXX..XXX lines (note comm_epoll.cc uses this notation) 5. made big doxygen effort!!! Enjoy. 6. (did not remove no-op else cases, I think they are clear) 7. used HERE macro as requested SQUID-COMPAT.DIFF /dev/poll doesn't need to be hard coded, that was a mistake. Please migrate USE_DEVPOLL conditional to same line as USE_EPOLL. SQUID-ROOT.DIFF Pretty much all beyond me, am happy with whatever changes you decide to make. I just wanted to add a /dev/poll auto-detect routine and use that in preference to poll() if available. Note that epoll/kqueue is assumed to be unavailable on any Solaris system with /dev/poll. You might want to thus prioritise /dev/poll support to BELOW that of epoll and kqueue (in case there's any confusion). But of course /dev/poll should be by default selected if available before defaulting to standard poll(). Kind regards, Peter Payne On 10/12/10 00:36, Amos Jeffries wrote: On Mon, 11 Oct 2010 14:58:34 +0100, Peter Paynesourcefo...@pirosa.co.uk wrote: Hello Amos, thank you for your response. Please find attached the four files in question. Companies to mention as sponsors are BBC (UK), Siemens IT Solutions and Services (UK). Credit to Peter Payne, Pirosa Limited UK (no e-mail please). Sure. No problem. Squid bug 3057 was raised by another member of Siemens IT Solutions and Services during testing of Squid 3.1.8 on Solaris - that bug was not produced on the 32-bit compile, just the 64-bit compile. It is not thought this is related to the /dev/poll support added here. Okay. On to the audit. This is just a 3.x style and consistency check. I'll take it as tested and working. comm_devpoll.cc: please make the following updates - use #if USE_DEVPOLL instead of #ifdef. - move all #includes except squid.h inside the USE_DEVPOLL wrapper. - move the /* Solaris /dev/poll support */ to its own line or remove completely. - remove the big ... lines. - use /** for the function documentation. If you know doxygen syntax please markup the text in any other relevant ways as well. - replace all debugs DEBUG_DEVPOLL ? 0 : 8 with 5 or 7 to match the other comm loops display density. - replace comm_update_fd the #if DEBUG_DEVPOLL with (?:) constructs in the debugs() ie (events POLLIN ? IN:) - in commSetSelect please remove the no-op else cases. // else: we want ... comments are fine. - comm_select please use HERE macro in debugs() instead of the text name of the function. squid-include.diff all done by autoconf during packaging. This can be dropped. squid-src.diff: - automake changes need to be made in Makefile.am. This can be done on commit. - unlinkd.cc changes okay. squid-compat.diff: - changes okay. - but can you mention why /dev/poll limit is hard-coded? compat area needs enough details so we can track the when and why for each hack. The aim being to remove them when the problems have been resolved. squid-root.diff: - okay as-is for 3.1. - for 3.2 and later we have simpler configure.in logics with some helper macros. That can be
Re: [PATCH] Solaris /dev/poll support for Squid 3 (how can I contribute)
Hello Amos, apologies to the dev list for what must appear to be spamming. Please use the attached version of the file. It has a minor performance tweak (only set events that have changed if not clearing any events). Compiled using CC=/tool/sunstudio12/bin/cc and tested again using Apache Bench and some manual queries through a web browser. Peter On 10/13/10 11:21, Peter Payne wrote: Hello Amos, in response to your comments. BENCHMARKS Firstly, I wanted to address the benchmark questions as it made me curious as to whether there really was an advantage in using /dev/poll. I used the Apache Bench tool (that comes with HTTPD) to do my benchmarks. I compiled a 32-bit Solaris 5.10 x86 of Squid version 3.1.8, and another Squid version 3.1.8 with my patches. I shall call them unpatched Squid and patched Squid respectively. Note that unpatched Squid was using ordinary poll() and patched Squid was using /dev/poll. Where I used Apache bench with 1 simultaneous connection or even 8,000 simultaneous connections there was little difference between unpatched Squid and patched Squid. When all connections are busy there is no performance gain. However the test that proved the most striking difference was pre-establishing 8,000 TCP socket connections to Squid using a basic Perl script. Then running Apache Bench with 1 connection making 100,000 keep-alive'd requests. unpatched Squid (using poll): 2min23sec CPU user-time 171sec to process 10 Apache Bench queries patched Squid (using /dev/poll): 0min22sec CPU user-time 25sec to process 10 Apache Bench queries Clearly /dev/poll has a significant performance gain where there are many idle TCP socket connections. I have now also optimised the comm_devpoll.cc routine to ensure unnecessary calls to /dev/poll with the POLLREMOVE flag were removed. CODE AUDIT Most of the style used was a result of directly copying the styles used in the existing comm_devpoll.c from Squid v2.7 and comm_epoll.cc from Squid v3.1.8. However where these conflict with the requests made most recently I will adopt the recommendations in the e-mail. 1. used #if USE_DEVPOLL instead of #ifdef 2. moved #includes except squid.h inside USE_DEVPOLL wrapper 3. moved comment to own line 4. removed big XXX..XXX lines (note comm_epoll.cc uses this notation) 5. made big doxygen effort!!! Enjoy. 6. (did not remove no-op else cases, I think they are clear) 7. used HERE macro as requested SQUID-COMPAT.DIFF /dev/poll doesn't need to be hard coded, that was a mistake. Please migrate USE_DEVPOLL conditional to same line as USE_EPOLL. SQUID-ROOT.DIFF Pretty much all beyond me, am happy with whatever changes you decide to make. I just wanted to add a /dev/poll auto-detect routine and use that in preference to poll() if available. Note that epoll/kqueue is assumed to be unavailable on any Solaris system with /dev/poll. You might want to thus prioritise /dev/poll support to BELOW that of epoll and kqueue (in case there's any confusion). But of course /dev/poll should be by default selected if available before defaulting to standard poll(). Kind regards, Peter Payne On 10/12/10 00:36, Amos Jeffries wrote: On Mon, 11 Oct 2010 14:58:34 +0100, Peter Paynesourcefo...@pirosa.co.uk wrote: Hello Amos, thank you for your response. Please find attached the four files in question. Companies to mention as sponsors are BBC (UK), Siemens IT Solutions and Services (UK). Credit to Peter Payne, Pirosa Limited UK (no e-mail please). Sure. No problem. Squid bug 3057 was raised by another member of Siemens IT Solutions and Services during testing of Squid 3.1.8 on Solaris - that bug was not produced on the 32-bit compile, just the 64-bit compile. It is not thought this is related to the /dev/poll support added here. Okay. On to the audit. This is just a 3.x style and consistency check. I'll take it as tested and working. comm_devpoll.cc: please make the following updates - use #if USE_DEVPOLL instead of #ifdef. - move all #includes except squid.h inside the USE_DEVPOLL wrapper. - move the /* Solaris /dev/poll support */ to its own line or remove completely. - remove the big ... lines. - use /** for the function documentation. If you know doxygen syntax please markup the text in any other relevant ways as well. - replace all debugs DEBUG_DEVPOLL ? 0 : 8 with 5 or 7 to match the other comm loops display density. - replace comm_update_fd the #if DEBUG_DEVPOLL with (?:) constructs in the debugs() ie (events POLLIN ? IN:) - in commSetSelect please remove the no-op else cases. // else: we want ... comments are fine. - comm_select please use HERE macro in debugs() instead of the text name of the function. squid-include.diff all done by autoconf during packaging. This can be dropped. squid-src.diff: - automake changes need to be made in Makefile.am. This can be done on commit. -
Re: [PATCH] Solaris /dev/poll support for Squid 3 (how can I contribute)
ons 2010-10-13 klockan 15:40 +0100 skrev Peter Payne: Hello Amos, apologies to the dev list for what must appear to be spamming. No apologies needed. We are all for release early an often, and discussing code is what this list is for. Regards Henrik
Re: [PATCH] Solaris /dev/poll support for Squid 3 (how can I contribute)
On Wed, 13 Oct 2010 15:40:22 +0100, Peter Payne sourcefo...@pirosa.co.uk wrote: Hello Amos, apologies to the dev list for what must appear to be spamming. Completely on topic. No apologies needed. Please use the attached version of the file. It has a minor performance tweak (only set events that have changed if not clearing any events). Compiled using CC=/tool/sunstudio12/bin/cc and tested again using Apache Bench and some manual queries through a web browser. Peter Will do. Thanks. On 10/13/10 11:21, Peter Payne wrote: Hello Amos, in response to your comments. BENCHMARKS Firstly, I wanted to address the benchmark questions as it made me curious as to whether there really was an advantage in using /dev/poll. I used the Apache Bench tool (that comes with HTTPD) to do my benchmarks. I compiled a 32-bit Solaris 5.10 x86 of Squid version 3.1.8, and another Squid version 3.1.8 with my patches. I shall call them unpatched Squid and patched Squid respectively. Note that unpatched Squid was using ordinary poll() and patched Squid was using /dev/poll. Where I used Apache bench with 1 simultaneous connection or even 8,000 simultaneous connections there was little difference between unpatched Squid and patched Squid. When all connections are busy there is no performance gain. However the test that proved the most striking difference was pre-establishing 8,000 TCP socket connections to Squid using a basic Perl script. Then running Apache Bench with 1 connection making 100,000 keep-alive'd requests. unpatched Squid (using poll): 2min23sec CPU user-time 171sec to process 10 Apache Bench queries patched Squid (using /dev/poll): 0min22sec CPU user-time 25sec to process 10 Apache Bench queries Oooh... Speed. Clearly /dev/poll has a significant performance gain where there are many idle TCP socket connections. I have now also optimised the comm_devpoll.cc routine to ensure unnecessary calls to /dev/poll with the POLLREMOVE flag were removed. CODE AUDIT Most of the style used was a result of directly copying the styles used in the existing comm_devpoll.c from Squid v2.7 and comm_epoll.cc from Squid v3.1.8. However where these conflict with the requests made most recently I will adopt the recommendations in the e-mail. 1. used #if USE_DEVPOLL instead of #ifdef 2. moved #includes except squid.h inside USE_DEVPOLL wrapper 3. moved comment to own line 4. removed big XXX..XXX lines (note comm_epoll.cc uses this notation) 5. made big doxygen effort!!! Enjoy. 6. (did not remove no-op else cases, I think they are clear) 7. used HERE macro as requested SQUID-COMPAT.DIFF /dev/poll doesn't need to be hard coded, that was a mistake. Please migrate USE_DEVPOLL conditional to same line as USE_EPOLL. K. will do SQUID-ROOT.DIFF Pretty much all beyond me, am happy with whatever changes you decide to make. I just wanted to add a /dev/poll auto-detect routine and use that in preference to poll() if available. Note that epoll/kqueue is assumed to be unavailable on any Solaris system with /dev/poll. You might want to thus prioritise /dev/poll support to BELOW that of epoll and kqueue (in case there's any confusion). But of course /dev/poll should be by default selected if available before defaulting to standard poll(). Okay. Thanks for that. Kind regards, Peter Payne +1 from me with merge tweaks. Unless anyone has objections I will commit with tweaks at the next opportunity. Amos
Re: [PATCH] Solaris /dev/poll support for Squid 3 (how can I contribute)
ons 2010-10-13 klockan 22:25 + skrev Amos Jeffries: +1 from me with merge tweaks. Unless anyone has objections I will commit with tweaks at the next opportunity. No objection from me. But have not reviewed the changes outside comm_devpoll.cc. Regards Henrik
[PATCH] Solaris /dev/poll support for Squid 3 (how can I contribute)
Hello Amos, thank you for your response. Please find attached the four files in question. Companies to mention as sponsors are BBC (UK), Siemens IT Solutions and Services (UK). Credit to Peter Payne, Pirosa Limited UK (no e-mail please). Squid bug 3057 was raised by another member of Siemens IT Solutions and Services during testing of Squid 3.1.8 on Solaris - that bug was not produced on the 32-bit compile, just the 64-bit compile. It is not thought this is related to the /dev/poll support added here. Note that while porting comm_devpoll.cc I have suspicions that the comm_epoll.cc commResetSelect() function does nothing (i.e., it calls commSetSelect with no type flag, which I suspect is a no-op for all intents and purposes). I haven't confirmed this, however, so haven't raised it as a bug. Someone may want to double-check. Kind regards, Peter Payne On 10/09/10 02:26, Amos Jeffries wrote: On 08/10/10 23:13, Peter Payne wrote: Dear Mailing List, I have a contribution to make, a C++ file (and autoconf patches) for /dev/poll Solaris support, ported from Squid-2 and in use on the BBC (UK) estate managed by Siemens UK with permission from the BBC to publish to the Squid open-source project. I've not contributed to an open source project before and unsure how to deliver the patch. We (at the BBC/Siemens) would like to see the patch included so that future releases of Squid shall have the Solaris /dev/poll support supplied (reducing the need for us to patch each following release manually). If one of you were to e-mail me directly I would be happy to provide the following: comm_devpoll.cc - /dev/poll support file to add to /src/ subdir squid-root.diff - unified diff to update configure.in in the / subdir squid-include.diff - unified diff to update autoconf.h.in in the /include/ subdir squid-src.diff - unified diff to update files Makefile.in and unlinkd.cc in the /src/ subdir The support for /dev/poll will be automatically detected by configure and conditionally compiled in. Greetings Peter, Thank you very much for this. Our procedure for patch inclusion is relatively simple: * Post the unified-diff format patches here to the squid-dev list. * with a subject starting [PATCH] * who to credit as author(s), default is the senders name and email. * message text describing hat it does for the public commit record and to assist people understand the code. * companies where relevant mention as thanks or sponsors (these last two are covered by the email you already sent). One or more of us will do an audit that it fits on 3.HEAD then reply with any updates that may be needed. Most of that you have done already, we just need the patches posted with author details to start the audit cycle. On the administrative side. I'm not sure this one is /dev/poll related, but with the info that BBC is using your patch this reporters address and build options looks a little suspicious: http://bugs.squid-cache.org/show_bug.cgi?id=3057 If you can sight it without the patch or can't sight it at all thats fine, but if introduced it will need fixing. Amos Jeffries Squid HTTP Proxy Project /* * $Id$ * * DEBUG: section 05Socket Functions * * SQUID Web Proxy Cache http://www.squid-cache.org/ * -- * * Squid is the result of efforts by numerous individuals from * the Internet community; see the CONTRIBUTORS file for full * details. Many organizations have provided support for Squid's * development; see the SPONSORS file for full details. Squid is * Copyrighted (C) 2001 by the Regents of the University of * California; see the COPYRIGHT file for full details. Squid * incorporates software developed and/or copyrighted by other * sources; see the CREDITS file for full details. * * This program is free software; you can redistribute it and/or modify * it under the terms of the GNU General Public License as published by * the Free Software Foundation; either version 2 of the License, or * (at your option) any later version. * * This program is distributed in the hope that it will be useful, * but WITHOUT ANY WARRANTY; without even the implied warranty of * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the * GNU General Public License for more details. * * You should have received a copy of the GNU General Public License * along with this program; if not, write to the Free Software * Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111, USA. * */ /* * This is a very simple driver for Solaris /dev/poll. * * The updates are batched, one trip through the comm loop. * (like libevent.) We keep a pointer into the structs so we * can zero out an entry in the poll list if its active. * * Ported by Peter Payne from Squid 2.7.STABLE9 comm_devpoll.c * on August 11, 2010 at 3pm (GMT+0100 Europe/London). * * Last modified 2010-10-08 */ #include
Re: [PATCH] Solaris /dev/poll support for Squid 3 (how can I contribute)
On Mon, 11 Oct 2010 14:58:34 +0100, Peter Payne sourcefo...@pirosa.co.uk wrote: Hello Amos, thank you for your response. Please find attached the four files in question. Companies to mention as sponsors are BBC (UK), Siemens IT Solutions and Services (UK). Credit to Peter Payne, Pirosa Limited UK (no e-mail please). Sure. No problem. Squid bug 3057 was raised by another member of Siemens IT Solutions and Services during testing of Squid 3.1.8 on Solaris - that bug was not produced on the 32-bit compile, just the 64-bit compile. It is not thought this is related to the /dev/poll support added here. Okay. On to the audit. This is just a 3.x style and consistency check. I'll take it as tested and working. comm_devpoll.cc: please make the following updates - use #if USE_DEVPOLL instead of #ifdef. - move all #includes except squid.h inside the USE_DEVPOLL wrapper. - move the /* Solaris /dev/poll support */ to its own line or remove completely. - remove the big ... lines. - use /** for the function documentation. If you know doxygen syntax please markup the text in any other relevant ways as well. - replace all debugs DEBUG_DEVPOLL ? 0 : 8 with 5 or 7 to match the other comm loops display density. - replace comm_update_fd the #if DEBUG_DEVPOLL with (?:) constructs in the debugs() ie (events POLLIN ? IN:) - in commSetSelect please remove the no-op else cases. // else: we want ... comments are fine. - comm_select please use HERE macro in debugs() instead of the text name of the function. squid-include.diff all done by autoconf during packaging. This can be dropped. squid-src.diff: - automake changes need to be made in Makefile.am. This can be done on commit. - unlinkd.cc changes okay. squid-compat.diff: - changes okay. - but can you mention why /dev/poll limit is hard-coded? compat area needs enough details so we can track the when and why for each hack. The aim being to remove them when the problems have been resolved. squid-root.diff: - okay as-is for 3.1. - for 3.2 and later we have simpler configure.in logics with some helper macros. That can be cut-n-pasted on commit. - RUNIFELSE is a bigger problem. It breaks cross-compiling and I notice its not present in 2.7. Is the configure-time run test marked Verify that /dev/poll really works actually needed? What will happen to builds made on non-Solaris with forced devpoll and the kernel headers present? In configure.in the magic if statement (hunk: @@ -3013,6 +3066,8 @@) has a specific order from fastest to slowest. Can you please point me/us at any reliable /dev/poll benchmarking comparisons with epoll/kqueue to justify placing it where you did? Thank you. Note that while porting comm_devpoll.cc I have suspicions that the comm_epoll.cc commResetSelect() function does nothing (i.e., it calls commSetSelect with no type flag, which I suspect is a no-op for all intents and purposes). I haven't confirmed this, however, so haven't raised it as a bug. Someone may want to double-check. Hmm. Yes. It stops the event FD being polled, but does not unset the handler like all the other loops. I've not yet learned the details of epoll well, if you think doing a no-op there is incorrect please make the bug report about it. Amos
Solaris /dev/poll support for Squid 3 (how can I contribute)
Dear Mailing List, I have a contribution to make, a C++ file (and autoconf patches) for /dev/poll Solaris support, ported from Squid-2 and in use on the BBC (UK) estate managed by Siemens UK with permission from the BBC to publish to the Squid open-source project. I've not contributed to an open source project before and unsure how to deliver the patch. We (at the BBC/Siemens) would like to see the patch included so that future releases of Squid shall have the Solaris /dev/poll support supplied (reducing the need for us to patch each following release manually). If one of you were to e-mail me directly I would be happy to provide the following: comm_devpoll.cc - /dev/poll support file to add to /src/ subdir squid-root.diff - unified diff to update configure.in in the / subdir squid-include.diff - unified diff to update autoconf.h.in in the /include/ subdir squid-src.diff - unified diff to update files Makefile.in and unlinkd.cc in the /src/ subdir The support for /dev/poll will be automatically detected by configure and conditionally compiled in. Kind regards, Peter Payne Pirosa Limited, under contract to Siemens UK, who operate BBC media estate sourcefo...@pirosa.co.uk
Re: Solaris /dev/poll support for Squid 3 (how can I contribute)
On 08/10/10 23:13, Peter Payne wrote: Dear Mailing List, I have a contribution to make, a C++ file (and autoconf patches) for /dev/poll Solaris support, ported from Squid-2 and in use on the BBC (UK) estate managed by Siemens UK with permission from the BBC to publish to the Squid open-source project. I've not contributed to an open source project before and unsure how to deliver the patch. We (at the BBC/Siemens) would like to see the patch included so that future releases of Squid shall have the Solaris /dev/poll support supplied (reducing the need for us to patch each following release manually). If one of you were to e-mail me directly I would be happy to provide the following: comm_devpoll.cc - /dev/poll support file to add to /src/ subdir squid-root.diff - unified diff to update configure.in in the / subdir squid-include.diff - unified diff to update autoconf.h.in in the /include/ subdir squid-src.diff - unified diff to update files Makefile.in and unlinkd.cc in the /src/ subdir The support for /dev/poll will be automatically detected by configure and conditionally compiled in. Greetings Peter, Thank you very much for this. Our procedure for patch inclusion is relatively simple: * Post the unified-diff format patches here to the squid-dev list. * with a subject starting [PATCH] * who to credit as author(s), default is the senders name and email. * message text describing hat it does for the public commit record and to assist people understand the code. * companies where relevant mention as thanks or sponsors (these last two are covered by the email you already sent). One or more of us will do an audit that it fits on 3.HEAD then reply with any updates that may be needed. Most of that you have done already, we just need the patches posted with author details to start the audit cycle. On the administrative side. I'm not sure this one is /dev/poll related, but with the info that BBC is using your patch this reporters address and build options looks a little suspicious: http://bugs.squid-cache.org/show_bug.cgi?id=3057 If you can sight it without the patch or can't sight it at all thats fine, but if introduced it will need fixing. Amos Jeffries Squid HTTP Proxy Project
storeurl_rewriter squid 3.x
I read a while ago that this would only be available in squid 3.2, but wonder if there is already some implementation because I was thinking about doing that but was wondering if there is already some effort. -- osmano807 Joaquim Pedro
Re: storeurl_rewriter squid 3.x
Joaquim Pedro França Simão wrote: I read a while ago that this would only be available in squid 3.2, but wonder if there is already some implementation because I was thinking about doing that but was wondering if there is already some effort. It was planned to be ported along with the other 2.7 features. It is one which has not yet happened. The deadline for 3.2 feature submission is running low (branching and release of 3.2.0.0 happens 30th July if snapshot releases are fixed), so any effort you can make towards a port patch in that time would be very welcome. You can find the basic 2.7 code changes and a few additional bug fixes at: http://www.squid-cache.org/Versions/v2/2.7/changesets/port.html plus one more recent bug fix at: http://www.squid-cache.org/Versions/v2/HEAD/changesets/12695.patch Amos -- Please be using Current Stable Squid 2.7.STABLE9 or 3.1.4
Re: Squid-3 and HTTP/1.1
Hello, HTTP compliance results for Squid 3.0.23, 3.1.16, and trunk r10264 have been uploaded to http://wiki.squid-cache.org/Features/HTTP11 We tried to preserve the old results intact for this update and just added new stuff. Going forward, we will polish the spreadsheet and probably remove outdated results. Suggestions welcome! The updated spreadsheet contains a new U test case outcome, representing test cases that yield different results during what should be identical tests from HTTP point of view. Most of such outcomes indicate timeouts and similar non-problems, but a few probably indicate parsing bugs. Thank you, Alex. On 01/29/2010 09:45 AM, Alex Rousskov wrote: On 01/28/2010 06:34 AM, Amos Jeffries wrote: Robert Collins wrote: On Wed, 2010-01-27 at 22:49 -0700, Alex Rousskov wrote: c) Co-Advisor currently only tests MUST-level requirements. Old Robert's checklist contained some SHOULD-level requirements as well. I see that Sheet1 on the spreadsheet has SHOULDs. Are we kind of ignoring them (and Sheet1) for now, until all MUSTs on Sheet2 are satisfied? d) I do not know who created the spreadsheet. Whoever it was, thank you! Is there a script that takes Co-Advisor results and produces a spreadsheet column for cut-and-pasting? Spreadsheet by me. Item format + naming by the Co-Advisor authors. Data by Y! testing + me for the estimated column. I tried to base it on yours Rob for a historical view of the 2.x support. But the item naming and crossover was too different and too much work to be reliable and easy. half-sorry ;) Alex: the current form is effectively a XLS dump cross-test of the Co-Advisor results. Plus manual estimations for the guess column. I've forgotten what was on the Sheet2. So yes it's missed a few rounds of updates. By Sheet2, do you mean Sheet1? Sheet2 seems to be more polished and it opens first by default. Please double check. Can you explain what XLS dump cross-test of the Co-Advisor results mean? Co-Advisor produces HTML and XML output. If you have a script that converts that to XLS, please share. Thank you, Alex.
Re: Squid-3 and HTTP/1.1
On 01/28/2010 04:42 PM, Mark Nottingham wrote: FWIW, I have an XSLT stylesheet that can format the results pleasantly; it could be a starting point for something automated. Please share if you can. Thank you, Alex. On 29/01/2010, at 12:34 AM, Amos Jeffries wrote: Robert Collins wrote: On Wed, 2010-01-27 at 22:49 -0700, Alex Rousskov wrote: c) Co-Advisor currently only tests MUST-level requirements. Old Robert's checklist contained some SHOULD-level requirements as well. I see that Sheet1 on the spreadsheet has SHOULDs. Are we kind of ignoring them (and Sheet1) for now, until all MUSTs on Sheet2 are satisfied? d) I do not know who created the spreadsheet. Whoever it was, thank you! Is there a script that takes Co-Advisor results and produces a spreadsheet column for cut-and-pasting? It looks nice. It might be based on the xls spreadsheet I made, but I don't know ;) I would not worry about SHOULD's until the MUSTs are done (but if a SHOULD is in reach while doing a MUST, doing it would be good). -Rob Spreadsheet by me. Item format + naming by the Co-Advisor authors. Data by Y! testing + me for the estimated column. I tried to base it on yours Rob for a historical view of the 2.x support. But the item naming and crossover was too different and too much work to be reliable and easy. half-sorry ;) Alex: the current form is effectively a XLS dump cross-test of the Co-Advisor results. Plus manual estimations for the guess column. I've forgotten what was on the Sheet2. So yes it's missed a few rounds of updates. Amos -- Please be using Current Stable Squid 2.7.STABLE7 or 3.0.STABLE21 Current Beta Squid 3.1.0.15 -- Mark Nottingham m...@yahoo-inc.com
Re: Squid-3 and HTTP/1.1
On 01/28/2010 06:34 AM, Amos Jeffries wrote: Robert Collins wrote: On Wed, 2010-01-27 at 22:49 -0700, Alex Rousskov wrote: c) Co-Advisor currently only tests MUST-level requirements. Old Robert's checklist contained some SHOULD-level requirements as well. I see that Sheet1 on the spreadsheet has SHOULDs. Are we kind of ignoring them (and Sheet1) for now, until all MUSTs on Sheet2 are satisfied? d) I do not know who created the spreadsheet. Whoever it was, thank you! Is there a script that takes Co-Advisor results and produces a spreadsheet column for cut-and-pasting? Spreadsheet by me. Item format + naming by the Co-Advisor authors. Data by Y! testing + me for the estimated column. I tried to base it on yours Rob for a historical view of the 2.x support. But the item naming and crossover was too different and too much work to be reliable and easy. half-sorry ;) Alex: the current form is effectively a XLS dump cross-test of the Co-Advisor results. Plus manual estimations for the guess column. I've forgotten what was on the Sheet2. So yes it's missed a few rounds of updates. By Sheet2, do you mean Sheet1? Sheet2 seems to be more polished and it opens first by default. Please double check. Can you explain what XLS dump cross-test of the Co-Advisor results mean? Co-Advisor produces HTML and XML output. If you have a script that converts that to XLS, please share. Thank you, Alex.
Re: Squid-3 and HTTP/1.1
On Wed, 2010-01-27 at 22:49 -0700, Alex Rousskov wrote: c) Co-Advisor currently only tests MUST-level requirements. Old Robert's checklist contained some SHOULD-level requirements as well. I see that Sheet1 on the spreadsheet has SHOULDs. Are we kind of ignoring them (and Sheet1) for now, until all MUSTs on Sheet2 are satisfied? d) I do not know who created the spreadsheet. Whoever it was, thank you! Is there a script that takes Co-Advisor results and produces a spreadsheet column for cut-and-pasting? It looks nice. It might be based on the xls spreadsheet I made, but I don't know ;) I would not worry about SHOULD's until the MUSTs are done (but if a SHOULD is in reach while doing a MUST, doing it would be good). -Rob signature.asc Description: This is a digitally signed message part
Re: Squid-3 and HTTP/1.1
Robert Collins wrote: On Wed, 2010-01-27 at 22:49 -0700, Alex Rousskov wrote: c) Co-Advisor currently only tests MUST-level requirements. Old Robert's checklist contained some SHOULD-level requirements as well. I see that Sheet1 on the spreadsheet has SHOULDs. Are we kind of ignoring them (and Sheet1) for now, until all MUSTs on Sheet2 are satisfied? d) I do not know who created the spreadsheet. Whoever it was, thank you! Is there a script that takes Co-Advisor results and produces a spreadsheet column for cut-and-pasting? It looks nice. It might be based on the xls spreadsheet I made, but I don't know ;) I would not worry about SHOULD's until the MUSTs are done (but if a SHOULD is in reach while doing a MUST, doing it would be good). -Rob Spreadsheet by me. Item format + naming by the Co-Advisor authors. Data by Y! testing + me for the estimated column. I tried to base it on yours Rob for a historical view of the 2.x support. But the item naming and crossover was too different and too much work to be reliable and easy. half-sorry ;) Alex: the current form is effectively a XLS dump cross-test of the Co-Advisor results. Plus manual estimations for the guess column. I've forgotten what was on the Sheet2. So yes it's missed a few rounds of updates. Amos -- Please be using Current Stable Squid 2.7.STABLE7 or 3.0.STABLE21 Current Beta Squid 3.1.0.15
Re: Squid-3 and HTTP/1.1
FWIW, I have an XSLT stylesheet that can format the results pleasantly; it could be a starting point for something automated. On 29/01/2010, at 12:34 AM, Amos Jeffries wrote: Robert Collins wrote: On Wed, 2010-01-27 at 22:49 -0700, Alex Rousskov wrote: c) Co-Advisor currently only tests MUST-level requirements. Old Robert's checklist contained some SHOULD-level requirements as well. I see that Sheet1 on the spreadsheet has SHOULDs. Are we kind of ignoring them (and Sheet1) for now, until all MUSTs on Sheet2 are satisfied? d) I do not know who created the spreadsheet. Whoever it was, thank you! Is there a script that takes Co-Advisor results and produces a spreadsheet column for cut-and-pasting? It looks nice. It might be based on the xls spreadsheet I made, but I don't know ;) I would not worry about SHOULD's until the MUSTs are done (but if a SHOULD is in reach while doing a MUST, doing it would be good). -Rob Spreadsheet by me. Item format + naming by the Co-Advisor authors. Data by Y! testing + me for the estimated column. I tried to base it on yours Rob for a historical view of the 2.x support. But the item naming and crossover was too different and too much work to be reliable and easy. half-sorry ;) Alex: the current form is effectively a XLS dump cross-test of the Co-Advisor results. Plus manual estimations for the guess column. I've forgotten what was on the Sheet2. So yes it's missed a few rounds of updates. Amos -- Please be using Current Stable Squid 2.7.STABLE7 or 3.0.STABLE21 Current Beta Squid 3.1.0.15 -- Mark Nottingham m...@yahoo-inc.com
Re: Squid-3 and HTTP/1.1
On 01/09/2010 04:10 AM, Amos Jeffries wrote: Alex Rousskov wrote: On 09/12/2009 05:36 AM, Amos Jeffries wrote: Updating the checklist today I again wonder if we can repeat the step from 2.7 and enable HTTP/1.1 on requests sent to servers As far as I can see the missing bits 3.2 needs to take that step are: - reject http-Upgrade requests from clients. - reject Expect-100 requests from clients. anything else? NP: both of those are temporary measures to prevent passing something through to the server that will get us into trouble. Until we get real support for them ready. PS... Alex and co, what hardware/system requirements are needed to run one of the audits? There are pretty much no special requirements other than internet connectivity because one can use Co-Advisor hosted on the Factory server. Linux RPMs can be used if remote testing is not possible. The tool does not require a fast CPU and 512MB RAM should be sufficient. Alex. Okay, wonderful. Then could we get things moving towards an audit of 3.1.0.16 when it happens next month please? possibly with a followup for 3.1.1 when the bits that first audit finds are fixed. Will do. I should update Sheet2 of the wonderful Features/HTTP11 spreadsheet, right? Please consider it locked for a few weeks. I have a few related questions: a) We have a 3.1 patch (currently being tested in production) that forces HTTP/1.1 version string for ACL-matching responses. Should we add that and turn in on for this audit? IIRC, a few test cases fail because Squid does not claim to be HTTP/1.1 compliant and that would allow us to pass those. b) We can (and may even have to) write a patch to force HTTP/1.1 version string for ACL-matching requests. Should we write and enable that as well for this audit? Same rationale as in (a). c) Co-Advisor currently only tests MUST-level requirements. Old Robert's checklist contained some SHOULD-level requirements as well. I see that Sheet1 on the spreadsheet has SHOULDs. Are we kind of ignoring them (and Sheet1) for now, until all MUSTs on Sheet2 are satisfied? d) I do not know who created the spreadsheet. Whoever it was, thank you! Is there a script that takes Co-Advisor results and produces a spreadsheet column for cut-and-pasting? Thank you, Alex.
Re: Squid-3 and HTTP/1.1
Alex Rousskov wrote: On 09/12/2009 05:36 AM, Amos Jeffries wrote: Updating the checklist today I again wonder if we can repeat the step from 2.7 and enable HTTP/1.1 on requests sent to servers As far as I can see the missing bits 3.2 needs to take that step are: - reject http-Upgrade requests from clients. - reject Expect-100 requests from clients. anything else? NP: both of those are temporary measures to prevent passing something through to the server that will get us into trouble. Until we get real support for them ready. PS... Alex and co, what hardware/system requirements are needed to run one of the audits? There are pretty much no special requirements other than internet connectivity because one can use Co-Advisor hosted on the Factory server. Linux RPMs can be used if remote testing is not possible. The tool does not require a fast CPU and 512MB RAM should be sufficient. Alex. Okay, wonderful. Then could we get things moving towards an audit of 3.1.0.16 when it happens next month please? possibly with a followup for 3.1.1 when the bits that first audit finds are fixed. I'll do the gopher work myself if need be. Amos
Re: Squid-3 and HTTP/1.1
On 09/12/2009 05:36 AM, Amos Jeffries wrote: Updating the checklist today I again wonder if we can repeat the step from 2.7 and enable HTTP/1.1 on requests sent to servers As far as I can see the missing bits 3.2 needs to take that step are: - reject http-Upgrade requests from clients. - reject Expect-100 requests from clients. anything else? NP: both of those are temporary measures to prevent passing something through to the server that will get us into trouble. Until we get real support for them ready. PS... Alex and co, what hardware/system requirements are needed to run one of the audits? There are pretty much no special requirements other than internet connectivity because one can use Co-Advisor hosted on the Factory server. Linux RPMs can be used if remote testing is not possible. The tool does not require a fast CPU and 512MB RAM should be sufficient. Alex.
Re: Squid-3 and HTTP/1.1
Henrik Nordstrom wrote: lör 2009-09-12 klockan 23:36 +1200 skrev Amos Jeffries: Updating the checklist today I again wonder if we can repeat the step from 2.7 and enable HTTP/1.1 on requests sent to servers The default in 2.7 is 1.0 still. There is an option to enable 1.1, or actually three.. (http11 cache_peer option, server_http11 directive and http11 http(s)_port option) As far as I can see the missing bits 3.2 needs to take that step are: - reject http-Upgrade requests from clients. - reject Expect-100 requests from clients. Unfortunately there is several clients which will fail if you do that... sending Expect: 100-continue, but not expecting it to be rejected with 417 Expectation Failed... is why there is an ignore_expect_100 directive in 2.7.. default off but many of the setups using 2.7 as normal proxy have had to enable this.. Thanks to the experimentation trying to fix bug 2176 I think I see a clean way to create expect-100 handling in Squid-3. If someone with more store experience can point out to me how to reset the StoreEntry properly after receiving and passing on a reply we can accept and process the 100 reply and then reset it for the actual data reply. Turning sockets used for *100 into a temporary persistent connection with two pipelined replies regardless of the overall persistence needs. ... branch created to experiment with. Amos -- Please be using Current Stable Squid 2.7.STABLE7 or 3.0.STABLE21 Current Beta Squid 3.1.0.15
Re: [PATCH] Support ICY protocol (ICECast / ICEShout servers) in Squid-3
Amos Jeffries wrote: Since we have no central place where the headers are upgraded I've had to skip porting the upgrade_http0.9 hack in Squid-2 and go straight to accepting ICY protocol as an accepted response protocol and handling it. Somewhat primitive for now. It's limited to parsing and regenerating the status line correctly, and skipping the HTTP/1.0 version override on non-HTTP protocol replies. Since it is on port 80 I've temporarily left the HTTP/1.1 required header alterations happening. Some testing will be needed over the next few days to ensure that the client software treats unknown headers nicely. If needed overrides for those are easily done as well now. Amos Now tested with VLC and WinAmp. Looks like a success for the ICY changes. Sounds artifacts and stuttering occurring regularly 1/sec without the patch. A seamless stream of sound when it's added. The patch as given breaks HTTP with an incorrectly inverted parse test. And was omitting a bypass on the incomplete parsed reply case. Applied after tweaks. Amos -- Please be using Current Stable Squid 2.7.STABLE7 or 3.0.STABLE20 Current Beta Squid 3.1.0.14
Re: [PATCH] Support ICY protocol (ICECast / ICEShout servers) in Squid-3
On Mon, 02 Nov 2009 22:22:41 +0100, Henrik Nordstrom wrote: mån 2009-11-02 klockan 00:43 +1300 skrev Amos Jeffries: Since we have no central place where the headers are upgraded I've had to skip porting the upgrade_http0.9 hack in Squid-2 and go straight to accepting ICY protocol as an accepted response protocol and handling it. What do you mean by no central place? The parse+validate and upgrade are spread between HttpStatusLine, HttpMsg, HttpReply, and client_side. Testing the first-round patch, found the spot that was triggering the header addition to 0.9 was done in two places, first by HttpReply on clone and also by client_side on slow clients after each incomplete status line parse attempt, then by HttpReply again after the client_side. It's a little bit of a mess with correctly scoped code and incomplete conversion of legacy code to such. Internally the headers should always be upgraded. To implement the squid-2 option then all that is needed is to detect when the header have been upgraded and then skip to send the header to the client. This is merely accepting the ICY status line as a valid header. The sub-headers are still operated as normal. The only crossover between the protocols seems to be Content-Type which retains semantics. The upgrade is still done and adds Via, Date, etc normally and seems not to effect the client software. I spent a rather enjoyable evening yesterday tested with Nullsoft WinAmp and VLC against a number of radio and stream servers of various speeds and networks. Just remember to not enable chunked encoding for the response.. Oh darn. Guilty. AFAICT the keep-alive is automatically disabled because there is no content-length set or possible. Somewhat primitive for now. It's limited to parsing and regenerating the status line correctly, and Do this mean HTTP status lines is also properly preserved now? Yes. And non-ICY status lines remains upgraded to HTTP/1.0 as required by RFC. skipping the HTTP/1.0 version override on non-HTTP protocol replies. Right. What Squid-2 does is to add that header internally with HTTP/0.9 as version. The version is overridden anyway when the response is composed. Aye. I read the patch many times ;) Squid-3 stores them as: ICY-status ICY+HTTP-headers (new HTTP ones last due to appending) Code which does upgrading on purely version numbers sees ICY as version 0.0. Any which gives trouble (such as chunked encoding) can be modified slightly to check for sline.protocol==PROTO_HTTP as well as version. Since it is on port 80 I've temporarily left the HTTP/1.1 required header alterations happening. Some testing will be needed over the next few days to ensure that the client software treats unknown headers nicely. If needed overrides for those are easily done as well now. Or fall back on what squid-2 does.. internally upgrade but then skip sending an HTTP header out.. Testing succeeded. We don't need to play tricks with the other headers. It hinges entirely on the first four bytes of the reply being ICY Amos
R: R: Squid 3 build errors on Visual Studio - problem still present
Hi Henrik, Sorry again for the delay ... :-( c:\work\vc_string\vs_string.cc : see declaration of 'size_t' c:\work\vc_string\vs_string.cc(34) : error C2057: expected constant expression Good. so it seems the test case worked. now replace std with testcase and try again, both in namespace and the failing assignment just to make sure it's not tripping over something else built-in. Still failing. Regards Guido Guido Serassio Acme Consulting S.r.l. Microsoft Gold Certified Partner Via Lucia Savarino, 110098 - Rivoli (TO) - ITALY Tel. : +39.011.9530135 Fax. : +39.011.9781115 Email: guido.seras...@acmeconsulting.it WWW: http://www.acmeconsulting.it
R: Squid 3 build errors on Visual Studio - problem still present
Hi Henrik,
Odd... std::string::npos is declared as follows:
namespace std;
templateclass _Elem,
class _Traits,
class _Ax
class basic_string
: public _String_val_Elem, _Ax
{
public:
static const size_type npos;
};
templateclass _Elem,
class _Traits,
class _Alloc
const typename basic_string_Elem, _Traits,
_Alloc::size_type
basic_string_Elem, _Traits, _Alloc::npos =
(typename basic_string_Elem, _Traits,
_Alloc::size_type)(-1);
typedef basic_stringchar, char_traitschar, allocatorchar
string;
Maybe Visual Studio drops the const attribute when trying to convert
the
value from const std::string:size_type to const String::size_type
and gets lost somehow, even with the value as such being a simple
size_t
integer..
What does it say about the attached code? My attempt in isolating the
String npos = std::string::npos code with as little change as possible
(had to kill the char_traits thing as that used template syntax not
accepted by G++ which slightly changes how size_type is declared)
It fails:
vs_string.cc
c:\work\vc_string\vs_string.cc(1) : error C2371: 'size_t' :
redefinition; different basic types
c:\work\vc_string\vs_string.cc : see declaration of 'size_t'
c:\work\vc_string\vs_string.cc(34) : error C2057: expected constant
expression
A comment to line 1 fixes only the first redefinition error.
Do you need the preprocessed source ?
Regards
Guido Serassio
Acme Consulting S.r.l.
Microsoft Gold Certified Partner
Via Lucia Savarino, 110098 - Rivoli (TO) - ITALY
Tel. : +39.011.9530135 Fax. : +39.011.9781115
Email: guido.seras...@acmeconsulting.it
WWW: http://www.acmeconsulting.it
Re: R: Squid 3 build errors on Visual Studio - problem still present
tor 2009-09-17 klockan 11:15 +0200 skrev Guido Serassio: It fails: vs_string.cc c:\work\vc_string\vs_string.cc(1) : error C2371: 'size_t' : redefinition; different basic types Gah, should have been unsigned long.. but interesting that VS apparently has size_t built-in. It was declared in the preprocessed source as typedef __w64 unsigned int size_t; c:\work\vc_string\vs_string.cc : see declaration of 'size_t' c:\work\vc_string\vs_string.cc(34) : error C2057: expected constant expression Good. so it seems the test case worked. now replace std with testcase and try again, both in namespace and the failing assignment just to make sure it's not tripping over something else built-in. Do you need the preprocessed source ? No, that was preprocessed already with no includes or other preprocessor directives. Regards Henrik
R: R: R: R: R: Squid 3 build errors on Visual Studio - problem still present
Hi Henrik, Can you produce a preprocessed source? That's the output of just the preprocessor, not actual compilarion, comparable to gcc -E option. Easier to identify what the compiler actually saw that way... Done. Sorry for the delay, but I'm very busy (as usual ...) I have placed the preprocessed source of two failing files (IpIntercept.cc and QosConfig.cc) here: http://www.acmeconsulting.it/libip.zip Regards Guido Guido Serassio Acme Consulting S.r.l. Microsoft Gold Certified Partner Via Lucia Savarino, 110098 - Rivoli (TO) - ITALY Tel. : +39.011.9530135 Fax. : +39.011.9781115 Email: guido.seras...@acmeconsulting.it WWW: http://www.acmeconsulting.it
Re: R: R: R: R: R: Squid 3 build errors on Visual Studio - problem still present
mån 2009-09-14 klockan 11:27 +0200 skrev Guido Serassio:
Done. Sorry for the delay, but I'm very busy (as usual ...)
I have placed the preprocessed source of two failing files
(IpIntercept.cc and QosConfig.cc) here:
http://www.acmeconsulting.it/libip.zip
Odd... std::string::npos is declared as follows:
namespace std;
templateclass _Elem,
class _Traits,
class _Ax
class basic_string
: public _String_val_Elem, _Ax
{
public:
static const size_type npos;
};
templateclass _Elem,
class _Traits,
class _Alloc
const typename basic_string_Elem, _Traits, _Alloc::size_type
basic_string_Elem, _Traits, _Alloc::npos =
(typename basic_string_Elem, _Traits,
_Alloc::size_type)(-1);
typedef basic_stringchar, char_traitschar, allocatorchar
string;
Maybe Visual Studio drops the const attribute when trying to convert the
value from const std::string:size_type to const String::size_type
and gets lost somehow, even with the value as such being a simple size_t
integer..
What does it say about the attached code? My attempt in isolating the
String npos = std::string::npos code with as little change as possible
(had to kill the char_traits thing as that used template syntax not
accepted by G++ which slightly changes how size_type is declared)
Regards
Henrik
typedef long size_t;
namespace std {
templateclass _Ty class allocator
{
public:
typedef size_t size_type;
};
templateclass _Elem,
class _Traits,
class _Ax
class basic_string
{
public:
typedef typename _Ax::size_type size_type;
static const size_type npos;
};
templateclass _Elem,
class _Traits,
class _Alloc
const typename basic_string_Elem, _Traits, _Alloc::size_type
basic_string_Elem, _Traits, _Alloc::npos =
(typename basic_string_Elem, _Traits, _Alloc::size_type)(-1);
typedef basic_stringchar, char, allocatorchar
string;
};
class String {
typedef size_t size_type;
static const size_type npos = std::string::npos;
};
Squid-3 and HTTP/1.1
Updating the checklist today I again wonder if we can repeat the step from 2.7 and enable HTTP/1.1 on requests sent to servers As far as I can see the missing bits 3.2 needs to take that step are: - reject http-Upgrade requests from clients. - reject Expect-100 requests from clients. anything else? NP: both of those are temporary measures to prevent passing something through to the server that will get us into trouble. Until we get real support for them ready. PS... Alex and co, what hardware/system requirements are needed to run one of the audits? Amos -- Please be using Current Stable Squid 2.7.STABLE6 or 3.0.STABLE19 Current Beta Squid 3.1.0.13
Re: Squid-3 and HTTP/1.1
lör 2009-09-12 klockan 23:36 +1200 skrev Amos Jeffries: Updating the checklist today I again wonder if we can repeat the step from 2.7 and enable HTTP/1.1 on requests sent to servers The default in 2.7 is 1.0 still. There is an option to enable 1.1, or actually three.. (http11 cache_peer option, server_http11 directive and http11 http(s)_port option) As far as I can see the missing bits 3.2 needs to take that step are: - reject http-Upgrade requests from clients. - reject Expect-100 requests from clients. Unfortunately there is several clients which will fail if you do that... sending Expect: 100-continue, but not expecting it to be rejected with 417 Expectation Failed... is why there is an ignore_expect_100 directive in 2.7.. default off but many of the setups using 2.7 as normal proxy have had to enable this.. anything else? I don't think so. Regards Henrik
R: R: R: R: Squid 3 build errors on Visual Studio - problem still present
Hi, OK, but what next ? Regards Guido Serassio Acme Consulting S.r.l. Microsoft Gold Certified Partner Via Lucia Savarino, 110098 - Rivoli (TO) - ITALY Tel. : +39.011.9530135 Fax. : +39.011.9781115 Email: guido.seras...@acmeconsulting.it WWW: http://www.acmeconsulting.it -Messaggio originale- Da: Amos Jeffries [mailto:squ...@treenet.co.nz] Inviato: martedì 1 settembre 2009 10.43 A: Henrik Nordstrom Cc: Guido Serassio; Robert Collins; squid-dev@squid-cache.org Oggetto: Re: R: R: R: Squid 3 build errors on Visual Studio - problem still present Henrik Nordstrom wrote: mån 2009-08-31 klockan 13:40 +0200 skrev Guido Serassio: Hi, I have found this definition: http://msdn.microsoft.com/en-us/library/tbbk8hs6(VS.80).aspx Hoping that helps. Matches the standard definition from what I can tell, and should not cause any errors... Regards Henrik Ah, well. It seems that npos may not work if its int or unsigned We do this: include/config.h:typedef unsigned int size_t; src/SquidString.h:typedef size_t size_type; src/SquidString.h:const static size_type npos = std::string::npos; making SquidString::size_type == size_t == unsigned int But, the complaint is about constant expression being expected? Amos -- Please be using Current Stable Squid 2.7.STABLE6 or 3.0.STABLE18 Current Beta Squid 3.1.0.13
Re: R: R: R: R: Squid 3 build errors on Visual Studio - problem still present
ons 2009-09-02 klockan 21:11 +0200 skrev Guido Serassio: Hi, OK, but what next ? Can you produce a preprocessed source? That's the output of just the preprocessor, not actual compilarion, comparable to gcc -E option. Easier to identify what the compiler actually saw that way... Regards Henrik
R: R: R: Squid 3 build errors on Visual Studio - problem still present
Hi, I have found this definition: http://msdn.microsoft.com/en-us/library/tbbk8hs6(VS.80).aspx Hoping that helps. Regards Guido Serassio Acme Consulting S.r.l. Microsoft Gold Certified Partner Via Lucia Savarino, 110098 - Rivoli (TO) - ITALY Tel. : +39.011.9530135 Fax. : +39.011.9781115 Email: guido.seras...@acmeconsulting.it WWW: http://www.acmeconsulting.it -Messaggio originale- Da: Robert Collins [mailto:robe...@robertcollins.net] Inviato: domenica 30 agosto 2009 22.13 A: Guido Serassio Cc: Amos Jeffries; squid-dev@squid-cache.org Oggetto: Re: R: R: Squid 3 build errors on Visual Studio - problem still present On Sun, 2009-08-30 at 18:13 +0200, Guido Serassio wrote: Hi, I don't know what is std::string::npos, and so I don't know what to look for http://www.cplusplus.com/reference/string/string/npos/ It should be a static const, which is why I'm so surprised you're getting an error about it. -Rob
R: Squid 3 build errors on Visual Studio - problem still present
Hi, The patch from Amos fix the warning (not blocking problem), but the compile problem is still present, and I don't know how to fix it. Any file including SquidString.h doesn't build with following error: c:\work\nt-3.0\src\SquidString.h(98) : error C2057: expected constant expression The offending code is: const static size_type npos = std::string::npos; Regards Guido Hi Amos, Hi, Who can help me to fix the following C++ errors on Visual Studio ? 1auth_basic.cc 1c:\work\nt-3.0\src\SquidString.h(97) : error C2057: expected constant expression 1../../../src\auth/User.h(39) : warning C4099: 'AuthUserHashPointer' : type name first seen using 'struct' now seen using 'class' 1../../../src\auth/Gadgets.h(51) : see declaration of 'AuthUserHashPointer' Regards Guido I think the TODO just needs to be enacted. Changing the 'struct' to 'class'. I need some hints for the SquidString.h error: it's a blocking error. I remember you that my C++ knowledge is too much limited. Regards Guido Serassio Acme Consulting S.r.l. Microsoft Gold Certified Partner Via Lucia Savarino, 110098 - Rivoli (TO) - ITALY Tel. : +39.011.9530135 Fax. : +39.011.9781115 Email: guido.seras...@acmeconsulting.it WWW: http://www.acmeconsulting.it
Re: R: Squid 3 build errors on Visual Studio - problem still present
On Sun, 2009-08-30 at 09:48 +0200, Guido Serassio wrote: c:\work\nt-3.0\src\SquidString.h(98) : error C2057: expected constant expression The offending code is: const static size_type npos = std::string::npos; Can you find out what std::string::npos is defined as in your compiler's headers? Thanks, Rob signature.asc Description: This is a digitally signed message part
Re: R: R: Squid 3 build errors on Visual Studio - problem still present
On Sun, 2009-08-30 at 18:13 +0200, Guido Serassio wrote: Hi, I don't know what is std::string::npos, and so I don't know what to look for http://www.cplusplus.com/reference/string/string/npos/ It should be a static const, which is why I'm so surprised you're getting an error about it. -Rob signature.asc Description: This is a digitally signed message part
R: Squid 3 build errors on Visual Studio
Hi Amos, Hi, Who can help me to fix the following C++ errors on Visual Studio ? 1auth_basic.cc 1c:\work\nt-3.0\src\SquidString.h(97) : error C2057: expected constant expression 1../../../src\auth/User.h(39) : warning C4099: 'AuthUserHashPointer' : type name first seen using 'struct' now seen using 'class' 1../../../src\auth/Gadgets.h(51) : see declaration of 'AuthUserHashPointer' Regards Guido I think the TODO just needs to be enacted. Changing the 'struct' to 'class'. I need some hints for the SquidString.h error: it's a blocking error. I remember you that my C++ knowledge is too much limited. Regards Guido Guido Serassio Acme Consulting S.r.l. Microsoft Gold Certified Partner Via Lucia Savarino, 110098 - Rivoli (TO) - ITALY Tel. : +39.011.9530135 Fax. : +39.011.9781115 Email: guido.seras...@acmeconsulting.it WWW: http://www.acmeconsulting.it
Squid 3 build errors on Visual Studio
Hi, Who can help me to fix the following C++ errors on Visual Studio ? 1auth_basic.cc 1c:\work\nt-3.0\src\SquidString.h(97) : error C2057: expected constant expression 1../../../src\auth/User.h(39) : warning C4099: 'AuthUserHashPointer' : type name first seen using 'struct' now seen using 'class' 1../../../src\auth/Gadgets.h(51) : see declaration of 'AuthUserHashPointer' Regards Guido Guido Serassio Acme Consulting S.r.l. Microsoft Gold Certified Partner Via Lucia Savarino, 1 10098 - Rivoli (TO) - ITALY Tel. : +39.011.9530135 Fax. : +39.011.9781115 Email: guido.seras...@acmeconsulting.it WWW: http://www.acmeconsulting.it
Re: Squid 3 build errors on Visual Studio
Guido Serassio wrote: Hi, Who can help me to fix the following C++ errors on Visual Studio ? 1auth_basic.cc 1c:\work\nt-3.0\src\SquidString.h(97) : error C2057: expected constant expression 1../../../src\auth/User.h(39) : warning C4099: 'AuthUserHashPointer' : type name first seen using 'struct' now seen using 'class' 1../../../src\auth/Gadgets.h(51) : see declaration of 'AuthUserHashPointer' Regards Guido I think the TODO just needs to be enacted. Changing the 'struct' to 'class'. Amos -- Please be using Current Stable Squid 2.7.STABLE6 or 3.0.STABLE18 Current Beta Squid 3.1.0.13
Re: Squid-3 MacOSX IPv6 problems
vollkom...@gmx.net wrote: It appears the special split-stack option was off, according to config.log: Depends on how it says off. Squid configure tests to see if its required by --with-ipv6-split-stack option. And if the option is not seen it attempt to operate the v4-mapping ability and see if its available. What I expect to see for you is that: configure detects the --with-* option (split-stack requirement) is not found. tests stack capabilities and finds type/mode as split-stack. - Which causes split-stack to be automatically enabled anyway despite the missing --with option. If your automatic build failed to detect it I would still like to track down why in addition to knowing if the split-stack code works yet. Amos Original-Nachricht Datum: Mon, 27 Jul 2009 14:24:06 +1200 Von: Amos Jeffries squ...@treenet.co.nz An: vollkom...@gmx.net CC: squid-dev@squid-cache.org Betreff: Re: Squid-3 MacOSX IPv6 problems Oh dear. Can you provide the config.log generated during build and a full cache.log from the point of startup through a failed DNS query please? Amos -- Please be using Current Stable Squid 2.7.STABLE6 or 3.0.STABLE17 Current Beta Squid 3.1.0.12
Re: Squid-3 MacOSX IPv6 problems
Oh dear. Can you provide the config.log generated during build and a full cache.log from the point of startup through a failed DNS query please? Amos
Re: Squid-3 MacOSX IPv6 problems
Hello, I'm not see any discernable difference than before: Squid Cache: Version 3.HEAD-20090723 configure options: '--build=i686-apple-darwin' '--mandir=/usr/local/share/man' '--with-large-files' '--disable-ident-lookups' '--disable-dependency-tracking' '--enable-filters' '--enable-ssl' '--enable-icap-client' '--enable-removal-policies=heap,lru' '--enable-external-acl-helpers=ip_user,ldap_group' '--enable-auth=basic,digest,ntlm,negotiate' '--enable-basic-auth-helpers=PAM,NCSA,LDAP,multi-domain-NTLM' '--enable-digest-auth-helpers=password' '--enable-delay-pools' '--enable-multicast-miss' '--enable-default-err-language=templates' '--enable-fd-config' '--with-filedescriptors=16384' '--with-dl' '--enable-gnuregex' '--enable-zph-qos' '--enable-ltdl-convenience' '--prefix=/usr/local/squid31' 'build_alias=i686-apple-darwin' --with-squid=/Users/raymont/squid-3.HEAD-20090723 barebone squid.conf used: acl manager proto cache_object acl localhost src 127.0.0.1/32 ::1 acl to_localhost dst 127.0.0.0/8 ::1 acl localnet src 10.0.0.0/8 # RFC1918 possible internal network acl localnet src 172.16.0.0/12 # RFC1918 possible internal network acl localnet src 192.168.0.0/16 # RFC1918 possible internal network acl SSL_ports port 443 acl Safe_ports port 80 # http acl Safe_ports port 21 # ftp acl Safe_ports port 443 # https acl Safe_ports port 70 # gopher acl Safe_ports port 210 # wais acl Safe_ports port 1025-65535 # unregistered ports acl Safe_ports port 280 # http-mgmt acl Safe_ports port 488 # gss-http acl Safe_ports port 591 # filemaker acl Safe_ports port 777 # multiling http acl CONNECT method CONNECT http_access allow manager localhost http_access deny manager http_access deny !Safe_ports http_access deny CONNECT !SSL_ports http_access allow localhost http_access allow localnet http_access deny all http_port 3129 hierarchy_stoplist cgi-bin ? coredump_dir /Applications/squid cache_access_log /Applications/squid/logs/access.log cache_log /Applications/squid/logs/cache.log cache_store_log none refresh_pattern ^ftp: 144020% 10080 refresh_pattern ^gopher:14400% 1440 refresh_pattern -i (/cgi-bin/|\?) 0 0% 0 refresh_pattern . 0 20% 4320 cache.log with squid -N -X: 2009/07/26 21:43:03.923| comm_poll: time out: 1248658983. 2009/07/26 21:43:03.923| Engine 0xb588 is idle. 2009/07/26 21:43:03.923| Engine 0xb59c is idle. 2009/07/26 21:43:03.923| comm_poll: 0+0 FDs ready 2009/07/26 21:43:03.923| comm_poll: time out: 1248658983. 2009/07/26 21:43:03.923| Engine 0xb588 is idle. 2009/07/26 21:43:03.923| event.cc(251) checkEvents: checkEvents 2009/07/26 21:43:03.923| The AsyncCall idnsCheckQueue constructed, this=0x1132e20 [call56] 2009/07/26 21:43:03.923| event.cc(260) will call idnsCheckQueue() [call56] 2009/07/26 21:43:03.923| Engine 0xb59c is idle. 2009/07/26 21:43:03.923| entering idnsCheckQueue() 2009/07/26 21:43:03.923| AsyncCall.cc(32) make: make call idnsCheckQueue [call56] 2009/07/26 21:43:03.923| idnsCheckQueue: ID 0x3fe3timeout 2009/07/26 21:43:03.923| comm_udp_sendto: Attempt to send UDP packet to 18.70.0.160:53 using FD 6 using Port 49745 2009/07/26 21:43:03.923| comm_udp_sendto: FD 6, (family=2) 18.70.0.160:53: (22) Invalid argument 2009/07/26 21:43:03.923| idnsSendQuery: FD 6: sendto: (22) Invalid argument 2009/07/26 21:43:03.923| comm_udp_sendto: Attempt to send UDP packet to 151.198.0.38:53 using FD 6 using Port 49745 2009/07/26 21:43:03.923| comm_udp_sendto: FD 6, (family=2) 151.198.0.38:53: (22) Invalid argument 2009/07/26 21:43:03.923| idnsSendQuery: FD 6: sendto: (22) Invalid argument 2009/07/26 21:43:03.923| comm_udp_sendto: Attempt to send UDP packet to 216.168.224.200:53 using FD 6 using Port 49745 2009/07/26 21:43:03.923| comm_udp_sendto: FD 6, (family=2) 216.168.224.200:53: (22) Invalid argument 2009/07/26 21:43:03.923| idnsSendQuery: FD 6: sendto: (22) Invalid argument 2009/07/26 21:43:03.923| comm_udp_sendto: Attempt to send UDP packet to 204.127.198.4:53 using FD 6 using Port 49745 2009/07/26 21:43:03.923| comm_udp_sendto: FD 6, (family=2) 204.127.198.4:53: (22) Invalid argument 2009/07/26 21:43:03.923| idnsSendQuery: FD 6: sendto: (22) Invalid argument 2009/07/26 21:43:03.923| event.cc(343) schedule: schedule: Adding 'idnsCheckQueue', in 1.00 seconds 2009/07/26 21:43:03.923| leaving idnsCheckQueue() 2009/07/26 21:43:03.923| Engine 0xb588 is idle. 2009/07/26 21:43:03.923| Engine 0xb59c is idle. 2009/07/26 21:43:04.890| comm_poll: 0+0 FDs ready 2009/07/26 21:43:04.890| comm_poll: time out: 1248658984. 2009/07/26 21:43:04.890| Engine 0xb588 is idle. Original-Nachricht Datum: Thu, 23 Jul 2009 01:50:08 +1200 Von: Amos Jeffries squ...@treenet.co.nz An: vollkom...@gmx.net, Squid Developers squid-dev@squid-cache.org Betreff: Squid-3 MacOSX IPv6 problems Hello again and apologies
Squid-3 MacOSX IPv6 problems
Hello again and apologies for the long delays. If you are still interested in getting Squid-3 to build with IPv6 read on. The current Squid-3.HEAD snapshots have been stabilized as much as I can do for now and include a few patches which I believe get split-stack support you need to build and run. The support is theoretical, I have had it working here for me for some basic web access operations. But my systems are natively hybrid stacks so the emulation may have holes I'm unaware of. Please test it out and let me know how things go. I'd like to get it at least confirmed usable before I back-port the patches to a 3.1 release. Cheers Amos -- Please be using Current Stable Squid 2.7.STABLE6 or 3.0.STABLE16 Current Beta Squid 3.1.0.10 or 3.1.0.11
Squid 3 build failure on MinGW
Hi, I have a little of free time, and I'm trying to build Squid 3 on MinGW, but I'm getting the following error: if /bin/sh ../../libtool --tag=CXX --mode=compile g++ -DHAVE_CONFIG_H -I../.. -I../../include -I../../src -I../../include -I/usr/include/libxml2 -Werror -Wall -Wpointer-arith -Wwrite-strings -Wcomments -D_FILE_OFFSET_BITS=64 -g -O2 -mthreads -MT IntRange.lo -MD -MP -MF .deps/IntRange.Tpo -c -o IntRange.lo IntRange.cc; \ then mv -f .deps/IntRange.Tpo .deps/IntRange.Plo; else rm -f .deps/IntRange.Tpo; exit 1; fi g++ -DHAVE_CONFIG_H -I../.. -I../../include -I../../src -I../../include -I/usr/include/libxml2 -Werror -Wall -Wpointer-arith -Wwrite-strings -Wcomments -D_FILE_OFFSET_BITS=64 -g -O2 -mthreads -MT IntRange.lo -MD -MP -MF .deps/IntRange.Tpo -c IntRange.cc -DDLL_EXPORT -DPIC -o .libs/IntRange.o ../../include/Range.h: In member function `RangeC RangeC::intersection(const RangeC) const [with C = int]': IntRange.cc:92: instantiated from here ../../include/Range.h:77: error: `max' was not declared in this scope ../../include/Range.h:77: error: `min' was not declared in this scope ../../include/Range.h:77: warning: unused variable 'max' ../../include/Range.h:77: warning: unused variable 'min' make[3]: *** [IntRange.lo] Error 1 make[3]: Leaving directory `/c/work/mgw-3.nt/src/acl' make[2]: *** [all-recursive] Error 1 make[2]: Leaving directory `/c/work/mgw-3.nt/src' make[1]: *** [all] Error 2 make[1]: Leaving directory `/c/work/mgw-3.nt/src' make: *** [all-recursive] Error 1 Someone could help me ? Regards Guido - Guido Serassio Acme Consulting S.r.l. - Microsoft Certified Partner Via Lucia Savarino, 1 10098 - Rivoli (TO) - ITALY Tel. : +39.011.9530135 Fax. : +39.011.9781115 Email: guido.seras...@acmeconsulting.it WWW: http://www.acmeconsulting.it/
Re: Squid 3 build failure on MinGW
Serassio Guido wrote: Hi, I have a little of free time, and I'm trying to build Squid 3 on MinGW, but I'm getting the following error: if /bin/sh ../../libtool --tag=CXX --mode=compile g++ -DHAVE_CONFIG_H -I../.. -I../../include -I../../src -I../../include -I/usr/include/libxml2 -Werror -Wall -Wpointer-arith -Wwrite-strings -Wcomments -D_FILE_OFFSET_BITS=64 -g -O2 -mthreads -MT IntRange.lo -MD -MP -MF .deps/IntRange.Tpo -c -o IntRange.lo IntRange.cc; \ then mv -f .deps/IntRange.Tpo .deps/IntRange.Plo; else rm -f .deps/IntRange.Tpo; exit 1; fi g++ -DHAVE_CONFIG_H -I../.. -I../../include -I../../src -I../../include -I/usr/include/libxml2 -Werror -Wall -Wpointer-arith -Wwrite-strings -Wcomments -D_FILE_OFFSET_BITS=64 -g -O2 -mthreads -MT IntRange.lo -MD -MP -MF .deps/IntRange.Tpo -c IntRange.cc -DDLL_EXPORT -DPIC -o .libs/IntRange.o ../../include/Range.h: In member function `RangeC RangeC::intersection(const RangeC) const [with C = int]': IntRange.cc:92: instantiated from here ../../include/Range.h:77: error: `max' was not declared in this scope ../../include/Range.h:77: error: `min' was not declared in this scope ../../include/Range.h:77: warning: unused variable 'max' ../../include/Range.h:77: warning: unused variable 'min' make[3]: *** [IntRange.lo] Error 1 make[3]: Leaving directory `/c/work/mgw-3.nt/src/acl' make[2]: *** [all-recursive] Error 1 make[2]: Leaving directory `/c/work/mgw-3.nt/src' make[1]: *** [all] Error 2 make[1]: Leaving directory `/c/work/mgw-3.nt/src' make: *** [all-recursive] Error 1 Someone could help me ? Regards Guido They are declared in the compat/* library. Included through config.h. compat/compat_shared.h to be exact if you need the definition (but please only include config.h for the compat stuff). Amos -- Please be using Current Stable Squid 2.7.STABLE6 or 3.0.STABLE15 Current Beta Squid 3.1.0.8 or 3.0.STABLE16-RC1
Re: Squid 3 build failure on MinGW
Serassio Guido wrote: Hi Amos, At 11.45 31/05/2009, Amos Jeffries wrote: Someone could help me ? Regards Guido They are declared in the compat/* library. Included through config.h. compat/compat_shared.h to be exact if you need the definition (but please only include config.h for the compat stuff). Already seen that, and config.h is included But there is something wrong in the definition of min() and max(), the build of every file in src including Range.h is failing Regards Guido I've seen this occur under 2 cases. One was when a .h in the core section failed to include config.h outside its wrapping protection. A circular include developed and broke things. And when the namespace for something was screwed up. It wouldn't be needing that nasty using namespace Squid; hack would it? Amos -- Please be using Current Stable Squid 2.7.STABLE6 or 3.0.STABLE15 Current Beta Squid 3.1.0.8 or 3.0.STABLE16-RC1
Re: Status of Squid 3 on Windows
Hi all, I've also planned hosting a shared windows dev system on eu, but we need to secure the necessary OS and devkit licenses first. On Monday, May 4, 2009, Guido Serassio guido.seras...@acmeconsulting.it wrote: Hi Alex, Il 02.13 04/05/2009 Alex Rousskov ha scritto: Hi Guido, Thanks a lot for the update. It is sad to see the results of many years of work dissipating and I sympathize with your situation. I wonder whether the demand for Squid on Windows is just not there OR it is the current lack of Windows support that makes it difficult for Acme to quickly monetize the potential demand. I know very little about folks running Windows servers so it is difficult for me to judge what they are using and what they want to use. Squid on Windows suffers two big issues: the only available comm loop is select() and the maximum FDs number is hard coded to 2048 in the MSVCRT library. This limits the scalability and restrict its usage to low end implementation, and go over this is really not so simple, because very big changes in the comm code are needed. My feel is that Squid on Windows is mainly used on very low cost implementation, and only a few of user could be interested on very limited investment. I have compiled an experimental binary build of Squid 3.0 STABLE 13 asking on squid-users to test it and give some feedback to the list. After two months, the feedback number was NONE, really I cannot understand this ... Can you say what it would take for you and Acme to resurrect Windows support for Squid3? It might be easier for others to pitch in with time, money, and/or hardware if the requirements are known. We can post your estimates to both mailing lists and see what happens. Mainly is not a money/hardware problem, its a time/resource problem. I cannot leave my current activity out but I could try to fix the build environment and after manage the work of some other developer, but the availability of other human resource in the Windows project is essential. Regards Guido - Guido Serassio Acme Consulting S.r.l. - Microsoft Certified Partner Via Lucia Savarino, 1 10098 - Rivoli (TO) - ITALY Tel. : +39.011.9530135 Fax. : +39.011.9781115 Email: guido.seras...@acmeconsulting.it WWW: http://www.acmeconsulting.it/ -- /kinkie
Re: Status of Squid 3 on Windows
On 05/05/2009 12:51 AM, Kinkie wrote: I've also planned hosting a shared windows dev system on eu, but we need to secure the necessary OS and devkit licenses first. Securing somebody who will be working on that windows dev system should come first, I guess. If nobody is going to be responsible for Squid on Windows, there is no point in obtaining licenses. Alex. On Monday, May 4, 2009, Guido Serassio guido.seras...@acmeconsulting.it wrote: Hi Alex, Il 02.13 04/05/2009 Alex Rousskov ha scritto: Hi Guido, Thanks a lot for the update. It is sad to see the results of many years of work dissipating and I sympathize with your situation. I wonder whether the demand for Squid on Windows is just not there OR it is the current lack of Windows support that makes it difficult for Acme to quickly monetize the potential demand. I know very little about folks running Windows servers so it is difficult for me to judge what they are using and what they want to use. Squid on Windows suffers two big issues: the only available comm loop is select() and the maximum FDs number is hard coded to 2048 in the MSVCRT library. This limits the scalability and restrict its usage to low end implementation, and go over this is really not so simple, because very big changes in the comm code are needed. My feel is that Squid on Windows is mainly used on very low cost implementation, and only a few of user could be interested on very limited investment. I have compiled an experimental binary build of Squid 3.0 STABLE 13 asking on squid-users to test it and give some feedback to the list. After two months, the feedback number was NONE, really I cannot understand this ... Can you say what it would take for you and Acme to resurrect Windows support for Squid3? It might be easier for others to pitch in with time, money, and/or hardware if the requirements are known. We can post your estimates to both mailing lists and see what happens. Mainly is not a money/hardware problem, its a time/resource problem. I cannot leave my current activity out but I could try to fix the build environment and after manage the work of some other developer, but the availability of other human resource in the Windows project is essential. Regards Guido - Guido Serassio Acme Consulting S.r.l. - Microsoft Certified Partner Via Lucia Savarino, 1 10098 - Rivoli (TO) - ITALY Tel. : +39.011.9530135 Fax. : +39.011.9781115 Email: guido.seras...@acmeconsulting.it WWW: http://www.acmeconsulting.it/
Re: Status of Squid 3 on Windows
Hi Alex, Il 02.13 04/05/2009 Alex Rousskov ha scritto: Hi Guido, Thanks a lot for the update. It is sad to see the results of many years of work dissipating and I sympathize with your situation. I wonder whether the demand for Squid on Windows is just not there OR it is the current lack of Windows support that makes it difficult for Acme to quickly monetize the potential demand. I know very little about folks running Windows servers so it is difficult for me to judge what they are using and what they want to use. Squid on Windows suffers two big issues: the only available comm loop is select() and the maximum FDs number is hard coded to 2048 in the MSVCRT library. This limits the scalability and restrict its usage to low end implementation, and go over this is really not so simple, because very big changes in the comm code are needed. My feel is that Squid on Windows is mainly used on very low cost implementation, and only a few of user could be interested on very limited investment. I have compiled an experimental binary build of Squid 3.0 STABLE 13 asking on squid-users to test it and give some feedback to the list. After two months, the feedback number was NONE, really I cannot understand this ... Can you say what it would take for you and Acme to resurrect Windows support for Squid3? It might be easier for others to pitch in with time, money, and/or hardware if the requirements are known. We can post your estimates to both mailing lists and see what happens. Mainly is not a money/hardware problem, its a time/resource problem. I cannot leave my current activity out but I could try to fix the build environment and after manage the work of some other developer, but the availability of other human resource in the Windows project is essential. Regards Guido - Guido Serassio Acme Consulting S.r.l. - Microsoft Certified Partner Via Lucia Savarino, 1 10098 - Rivoli (TO) - ITALY Tel. : +39.011.9530135 Fax. : +39.011.9781115 Email: guido.seras...@acmeconsulting.it WWW: http://www.acmeconsulting.it/
Status of Squid 3 on Windows
Hi, Hoping that Bazaar 1.14 really fixes its problems on Windows (I still don't have tested if it really works), I have done a summary of the status of Squid on Windows, focusing on various aspects: build environment, resources for development, source code status and a TODO for both Squid 2 and Squid 3. Build environment Squid should compile on Windows with 3 build environment, but with some limits: - MinGW+MSYS: full automake/autoconf/libtool/gcc support based on GNU toolset, but not all functionality (Large file support, IPv6, mswin_check_ad_group helper, WSAPoll() support on Vista and later, 64 bit Windows support, AIO) can be compiled because a lack of many library definitions. The generated code quality is good and runs natively on Windows using the MSVCRT, good performances. - Cygwin: similar to MinGW, but with more problems. The Windows API libraries are common to MinGW, so, same compile problems, but moreover, the build environment itself is very inconstant due to continuous changes from Cygwin developers and many times the Squid build process fails after a Cygwin update. The generated code is not so good, because it relies on an emulation layer, with worse performances. - MS Visual Studio: it lacks any kind of configure script support, some of the magic is done using some special make projects included in the main Workspace. But all functionality can be compiled. The generated code quality is good and runs natively on Windows using the MSVCRT, good performances. So, Visual Studio environment is still a must for Squid 3 on Windows. Resources for development Currently I'm the only developer working on Windows, but Squid for Acme Consulting today is a very marginal business, because we are mainly focused on services for Windows System Administration/Deployment and because it seems that IT Italian Market is not so interested to Squid (not only on Windows). For this reason currently we cannot invest into Squid deployment, so I can dedicate only a little part of my limited free time to Squid Development. Source code status While Squid 2 source code is fine, things for Squid 3 are disastrous: Squid 3 (trunk and 3.1) was no more compiled on Visual Studio since March 2008, so the effects of all subsequent big changes are totally untested on Windows. At the time of the bazaar cutover, the IPv6 support on Windows was still missing. Today the Visual Studio workspace is totally outdated and needs a big work to be repaired. Just to more complicate things, the NT CVS branch on Sourgeforge is corrupted since two months by a very big cvsmerge failed for a time-out during script execution. TODO Squid 2 - Upgrade Visual Studio support from old 6.0 (even if it's very good, but obsolete) to 2008 (better if we can use the free version) - Add real run-time support for WSAPoll() on Windows Vista and later, the Enhancement from the bug #1739 is required TODO Squid 3 - Allow Squid to build again on Visual Studio - Check if Squid 3 still runs on Windows - Upgrade Visual Studio support from 2005 to 2008 (better if we can use the free version) - Add IPv6 support - Add real run-time support for WSAPoll() on Windows Vista and later, the Enhancement from the bug #1739 is required - . IMHO, the port of Squid 3 on Windows seems to be too much damaged for my capabilities and time resources, so, without any other big development contribution, we should drop it. Regards Guido - Guido Serassio Acme Consulting S.r.l. - Microsoft Certified Partner Via Lucia Savarino, 1 10098 - Rivoli (TO) - ITALY Tel. : +39.011.9530135 Fax. : +39.011.9781115 Email: guido.seras...@acmeconsulting.it WWW: http://www.acmeconsulting.it/
Re: Status of Squid 3 on Windows
On 05/03/2009 05:02 AM, Guido Serassio wrote: Hi, Hoping that Bazaar 1.14 really fixes its problems on Windows (I still don't have tested if it really works), I have done a summary of the status of Squid on Windows, focusing on various aspects: build environment, resources for development, source code status and a TODO for both Squid 2 and Squid 3. Build environment Squid should compile on Windows with 3 build environment, but with some limits: - MinGW+MSYS: full automake/autoconf/libtool/gcc support based on GNU toolset, but not all functionality (Large file support, IPv6, mswin_check_ad_group helper, WSAPoll() support on Vista and later, 64 bit Windows support, AIO) can be compiled because a lack of many library definitions. The generated code quality is good and runs natively on Windows using the MSVCRT, good performances. - Cygwin: similar to MinGW, but with more problems. The Windows API libraries are common to MinGW, so, same compile problems, but moreover, the build environment itself is very inconstant due to continuous changes from Cygwin developers and many times the Squid build process fails after a Cygwin update. The generated code is not so good, because it relies on an emulation layer, with worse performances. - MS Visual Studio: it lacks any kind of configure script support, some of the magic is done using some special make projects included in the main Workspace. But all functionality can be compiled. The generated code quality is good and runs natively on Windows using the MSVCRT, good performances. So, Visual Studio environment is still a must for Squid 3 on Windows. Resources for development Currently I'm the only developer working on Windows, but Squid for Acme Consulting today is a very marginal business, because we are mainly focused on services for Windows System Administration/Deployment and because it seems that IT Italian Market is not so interested to Squid (not only on Windows). For this reason currently we cannot invest into Squid deployment, so I can dedicate only a little part of my limited free time to Squid Development. Source code status While Squid 2 source code is fine, things for Squid 3 are disastrous: Squid 3 (trunk and 3.1) was no more compiled on Visual Studio since March 2008, so the effects of all subsequent big changes are totally untested on Windows. At the time of the bazaar cutover, the IPv6 support on Windows was still missing. Today the Visual Studio workspace is totally outdated and needs a big work to be repaired. Just to more complicate things, the NT CVS branch on Sourgeforge is corrupted since two months by a very big cvsmerge failed for a time-out during script execution. TODO Squid 2 - Upgrade Visual Studio support from old 6.0 (even if it's very good, but obsolete) to 2008 (better if we can use the free version) - Add real run-time support for WSAPoll() on Windows Vista and later, the Enhancement from the bug #1739 is required TODO Squid 3 - Allow Squid to build again on Visual Studio - Check if Squid 3 still runs on Windows - Upgrade Visual Studio support from 2005 to 2008 (better if we can use the free version) - Add IPv6 support - Add real run-time support for WSAPoll() on Windows Vista and later, the Enhancement from the bug #1739 is required - . IMHO, the port of Squid 3 on Windows seems to be too much damaged for my capabilities and time resources, so, without any other big development contribution, we should drop it. Hi Guido, Thanks a lot for the update. It is sad to see the results of many years of work dissipating and I sympathize with your situation. I wonder whether the demand for Squid on Windows is just not there OR it is the current lack of Windows support that makes it difficult for Acme to quickly monetize the potential demand. I know very little about folks running Windows servers so it is difficult for me to judge what they are using and what they want to use. Can you say what it would take for you and Acme to resurrect Windows support for Squid3? It might be easier for others to pitch in with time, money, and/or hardware if the requirements are known. We can post your estimates to both mailing lists and see what happens. Thank you, Alex.
Re: storeurl_rewrite_program in squid 3.x?
There are long term plans to include it along with the rest of the useful stuff in Squid-2.7. It's most likely to occur in 3.2 and later, but we don't presently have anyone with time and interest to port it right now. If you are able to supply the developer time or cash resources to get it done please do so. Though I know python well, I am not comfortable with c/c++. And as I am a developer myself, can't really supply cash :( I'll sit back and wait. In the meantime will try to revamp my c/c++ concepts so that I can jump in squid-devel in future. The roadmap for work on Squid-3 can be found at http://wiki.squid-cache.org/RoadMap/Squid3/ Amos -- --- Thank you, Kulbir Saini, Computer Science and Engineering, International Institute of Information Technology, Hyderbad, India - 500032. My Home-Page: http://saini.co.in/ My Open-Source Project : http://cachevideos.com/ My Web-Blog: http://fedora.co.in/ IRC nick : generalBordeaux Channels : #fedora, #fedora-devel, #yum on freenode -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean.
storeurl_rewrite_program in squid 3.x?
Hi list, Is there any plans to port storeurl directives from 2.7 to 3.x? I don't see it in 3.1 beta. One of my plugins (intelligentmirror [1]) depends on it. I am not able to release a version for squid-3.x and most the distros are shipping squid-3.x now. [1] http://fedora.co.in/intelligentmirror --- Thank you, Kulbir Saini, Computer Science and Engineering, International Institute of Information Technology, Hyderbad, India - 500032. My Home-Page: http://saini.co.in/ My Open-Source Project : http://cachevideos.com/ My Web-Blog: http://fedora.co.in/ IRC nick : generalBordeaux Channels : #fedora, #fedora-devel, #yum on freenode -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean.
Squid 3 HEAD - compiler options
Just for the sake of consistency and preference for better troubleshooting and testing, are there any CFLAGS that I should be using during configure and compile of Squid 3HEAD? I was using the same options I was using with 2.6, which were: CFLAGS=-DNUMTHREADS=65 -march=nocona -03 -pipe -fomit-frame-pointer -funroll-loops -ffast-math -fno-exceptions I pseudo blindly copied the CFLAGS from what was consistent between two different deployments of squid on two different institutions that I was not a part of. This platform is a Intel Core 2 Duo system running 64bit Fedora 9 or 64bit CentOS 5.1, 4 GB RAM, 250GB SATA storage. I want to make sure I present any testing/troubleshooting information in a manner that is best suited to helping the project members resolve problems. Nick
Re: TPROXY support in Squid 3
Take 2: includes initial modularisation (untested; I'll test it at home this weekend when I get my tproxy box back online) and configure magic (with placeholders for tproxy4/freebsd.) http://www.creative.net.au/diffs/20080408-tproxy-fix-2.diff TODO: * pull out the capabilities stuff, removing the last #ifdef LINUX_TPROXY from the source * look at modifying comm_open*() to take a setup flag, and pass this in where required * make sure upstream/peer-forwarded requests aren't thrown to the tproxy code. * Once thats committed and working, go through and rename everything to be slightly more sensible. Adrian -- - Xenion - http://www.xenion.com.au/ - VPS Hosting - Commercial Squid Support - - $25/pm entry-level VPSes w/ capped bandwidth charges available in WA -
Re: TPROXY support in Squid 3
tis 2008-04-08 klockan 17:59 +1200 skrev Amos Jeffries: Currently: fde::flags::transparent == 'intercept/non-intercept' fde::flags::tproxy == real-transparent/non-transparent (new) COMM_TRANSPARENT == real-transparent Their use is currently good for what they do. A small re-naming is all that is needed there. I'm thinking flags::intercept and flags::realtransparent. In the request flags I would use intercepted and spoofclientip, with intercepted replacing the current transparent flag. Probably the same in the fde, where tproxy sets both. fde: intercepted This connection was intercepted spoofclientip Please mark received requests for spoofing the client ip request intercepted This requequest was received on an intercepted conection, need to act as if we were the origin server.. spoofclientip Spoof the outgoing connection using the client ip It's possible one may want to use spoofing without interception, and the above opens up for the possibility of marking requests with spoofclientip even without having to be received on an intercepted connection. Regards Henrik
Re: TPROXY support in Squid 3
tis 2008-04-08 klockan 14:28 +0800 skrev Adrian Chadd: Take 2: includes initial modularisation (untested; I'll test it at home this weekend when I get my tproxy box back online) and configure magic (with placeholders for tproxy4/freebsd.) http://www.creative.net.au/diffs/20080408-tproxy-fix-2.diff Comments: commTransparentRemote needs to move into comm_openex. You can't fit tproxy4/freebsd in a reasnoable manner otherwise. By removing the need_linux_tproxy you just killed the capability to fall back no non-tproxy connections if we failed to gain the capabilities needed.. cache.log will now get filled with tons of errors in such setups., where you earlier only got a single warning about continuing without tproxy support.. * pull out the capabilities stuff, removing the last #ifdef LINUX_TPROXY from the source TPROXY both versions need the special capabilities, and we do not want to keep these capabilities if TPROXY is not used.. * make sure upstream/peer-forwarded requests aren't thrown to the tproxy code. Consider a local proxy connecting to a remote proxy. You may want to still show the actual client IPs to the remote proxy just as you do to websites.. A cache_peer flag controlling this would make sense, probably defaulting to not spoof the client ip on requests sent to peers.. Regards Henrik
Re: TPROXY support in Squid 3
On Tue, Apr 08, 2008, Henrik Nordstrom wrote: tis 2008-04-08 klockan 14:28 +0800 skrev Adrian Chadd: Take 2: includes initial modularisation (untested; I'll test it at home this weekend when I get my tproxy box back online) and configure magic (with placeholders for tproxy4/freebsd.) http://www.creative.net.au/diffs/20080408-tproxy-fix-2.diff Comments: commTransparentRemote needs to move into comm_openex. You can't fit tproxy4/freebsd in a reasnoable manner otherwise. Yup, that'd be part of the follow up work. This first cut is to break out what is there; the next commit would be further API tidyups to support TPROXY4 and FreeBSD. By removing the need_linux_tproxy you just killed the capability to fall back no non-tproxy connections if we failed to gain the capabilities needed.. cache.log will now get filled with tons of errors in such setups., where you earlier only got a single warning about continuing without tproxy support.. Yes, I have to give that a little more thought. I may keep the global but not call it linux_tproxy.. * pull out the capabilities stuff, removing the last #ifdef LINUX_TPROXY from the source TPROXY both versions need the special capabilities, and we do not want to keep these capabilities if TPROXY is not used.. Ok. * make sure upstream/peer-forwarded requests aren't thrown to the tproxy code. Consider a local proxy connecting to a remote proxy. You may want to still show the actual client IPs to the remote proxy just as you do to websites.. A cache_peer flag controlling this would make sense, probably defaulting to not spoof the client ip on requests sent to peers.. I read the current code as don't do that; its the behaviour I'd like to maintain. We can always add the functionality later. Adrian -- - Xenion - http://www.xenion.com.au/ - VPS Hosting - Commercial Squid Support - - $25/pm entry-level VPSes w/ capped bandwidth charges available in WA -
Re: TPROXY support in Squid 3
On Tue, 2008-04-08 at 15:15 +0200, Henrik Nordstrom wrote: tis 2008-04-08 klockan 17:59 +1200 skrev Amos Jeffries: Currently: fde::flags::transparent == 'intercept/non-intercept' fde::flags::tproxy == real-transparent/non-transparent (new) COMM_TRANSPARENT == real-transparent Their use is currently good for what they do. A small re-naming is all that is needed there. I'm thinking flags::intercept and flags::realtransparent. In the request flags I would use intercepted and spoofclientip, with intercepted replacing the current transparent flag. Probably the same in the fde, where tproxy sets both. fde: intercepted This connection was intercepted spoofclientip Please mark received requests for spoofing the client ip request intercepted This requequest was received on an intercepted conection, need to act as if we were the origin server.. spoofclientip Spoof the outgoing connection using the client ip Please use camelCase or underscore_separators to make new names more readable. Thank you, Alex.
Re: TPROXY support in Squid 3
tis 2008-04-08 klockan 22:17 +0800 skrev Adrian Chadd: * make sure upstream/peer-forwarded requests aren't thrown to the tproxy code. I read the current code as don't do that; its the behaviour I'd like to maintain. We can always add the functionality later. Yes, it is not something you want to have done by default as in many cases peerings is local.. Regards Henrik
Re: TPROXY support in Squid 3
On Tue, Apr 08, 2008, Adrian Chadd wrote: Take 3: http://www.creative.net.au/diffs/20080408-tproxy-fix-2.diff * restored the global flag which indicates tproxyness; renamed to enable_spoof_srv * moved the tproxy bind stuff into comm.c, with a flag to comm_openex() * changed forward.c to try a tproxy comm_openex() if requested, falling back to a non-tproxy comm_openex() if that fails. TODO: * actually call the ipt_*Capabilities() hooks; I'll go add that in now. This stuff has almost no testing! Its more for feedback; I'll try to fire up the tproxy+wccp testbench this weekend to do some actual tproxy2 and tproxy4 testing before I look at committing it. Adrian -- - Xenion - http://www.xenion.com.au/ - VPS Hosting - Commercial Squid Support - - $25/pm entry-level VPSes w/ capped bandwidth charges available in WA -
Re: TPROXY support in Squid 3
On Tue, 2008-04-08 at 15:15 +0200, Henrik Nordstrom wrote: tis 2008-04-08 klockan 17:59 +1200 skrev Amos Jeffries: Currently: fde::flags::transparent == 'intercept/non-intercept' fde::flags::tproxy == real-transparent/non-transparent (new) COMM_TRANSPARENT == real-transparent Their use is currently good for what they do. A small re-naming is all that is needed there. I'm thinking flags::intercept and flags::realtransparent. In the request flags I would use intercepted and spoofclientip, with intercepted replacing the current transparent flag. Probably the same in the fde, where tproxy sets both. fde: intercepted This connection was intercepted spoofclientip Please mark received requests for spoofing the client ip request intercepted This request was received on an intercepted connection, need to act as if we were the origin server.. spoofclientip Spoof the outgoing connection using the client ip Please use camelCase or underscore_separators to make new names more readable. Agreed. For the flags I'm keeping with the lower case and using _. I'm part way through rollout of the 'intercepted' flag change now. Next up will be replacing the tproxy flag. I've enabled both flags as always-present where they are used and dropped the LINUX_NETFILTER and LINUX_TPROXY4 macros back to IPInterception.cc internals. LINUX_TPROXY2 requires a few extra bits of code which are still unique so until its agreed for dropping there are bits where its better to leave it. Amos
Re: TPROXY support in Squid 3
Adrian Chadd wrote: On Mon, Mar 31, 2008, Alex Rousskov wrote: What about Adrian plans (if I understood them correctly) to add TPROXY-like support to FreeBSD but not for TPROXY4-like API? Is that a good enough reason to continue supporting unsupported TPROXY versions? The FreeBSD API will be almost like the TPROXY-4 API. I'd suggest supporting TPROXY-2 for a few reasons: * Those who are using it may not want to track the latest kernel + TPROXY patches for various reasons (if it just works; company policy; etc.) and I think its easy enough to maintain support for both without too much hassle. * Supporting both TPROXY-2 and TPROXY-4 will (hopefully!) force someone to integrate it cleanishly and avoid the Squid-2 ip interception mess! * Thus making it easier for me to drop in a FreeBSD version of tproxy without too much hassle (or #ifdef's for that matter.) It shouldn't be that difficult to isolate the bits of the code required for spoofing the client IP in the request versus the TPROXY-specific stuff. In fact, the only tproxy-specific stuff I can really see is: * the logic in forward.c to the local bind, which can be wrapped up as part of the socket creation process, and * The initialisation code, which in the tproxy-2 case does capabilities magic. Adrian We have come up with a 'final-beta' patch for squid-3 now. http://treenet.co.nz/projects/squid/patches/tproxy-squid-3_20080407.patch Just waiting on Laszlo final approval. It's pretty much: * adding a COMM_TRANSPARENT flag to comm_openex(...) * adds a comm_set_transparent for internal comm use to do setsockopt() * adding a transparent flag to all fde's - set to 1 on fde's which are opened with the transparent flag OR, accepted from a listening fde with transparent flag. * override getOutgoingAddr ACL checks to produce the transparent client-address as outgoing addr. * adapt the existing Netfilter NAT getsockopt() for tproxy option. Should be easy to drop the squid side of your emulated TPROXY-4 alterations in there Adrian. Amos -- Please use Squid 2.6.STABLE19 or 3.0.STABLE4
Re: TPROXY support in Squid 3
mån 2008-04-07 klockan 23:11 +1200 skrev Amos Jeffries:
We have come up with a 'final-beta' patch for squid-3 now.
http://treenet.co.nz/projects/squid/patches/tproxy-squid-3_20080407.patch
Just waiting on Laszlo final approval.
Some comments...
There should be a general TPROXY define, shared by all, or the common
os-independent code should always be enabled. I do not want to see #if
LINUX_TPROXY2 || LINUX_TPROXY4 defines in the general code. Such ifs
should only be seen in the lowlevel implementation. And from a code
maintenance perspective it's best is the common code is always built,
just as is done for the transparent interception code. The drawback is
that it costs one or two cpu cycles more per requests even if the
feature is not available on the platform but it's much much less likely
the feature gets broken by other changes.
+/* AYJ: do we actually need to set this again on every accept? */
+if(fd_table[fd].flags.transparent == 1) {
+comm_set_transparent(sock);
+F-flags.transparent = 1;
+}
You need to at least set the internal flag. But you most likely do no
need to make the comm call telling the kernel it's a transparent
connection (it already knows..)
Is this indended?
@@ -652,7 +673,6 @@
addr.GetAddrInfo(AI);
AI-ai_socktype = sock_type;
AI-ai_protocol = proto;
-AI-ai_flags = flags;
debugs(50, 3, comm_openex: Attempt open socket for: addr );
Regards
Henrik
Re: TPROXY support in Squid 3
In fact, there shouldn't be any LINUX_TPROXY* defines in the main codetree.
There should be a SERVER_SPOOF define which ties in all of the connection
tracking stuff, and a clean cut API for doing TPROXY2/TPROXY4/etc socket
manipulation.
Gah, this makes me want to just ignore more classwork and fix the tproxy
support in Squid-2 to show how it should be done..
Adrian
On Mon, Apr 07, 2008, Henrik Nordstrom wrote:
m??n 2008-04-07 klockan 23:11 +1200 skrev Amos Jeffries:
We have come up with a 'final-beta' patch for squid-3 now.
http://treenet.co.nz/projects/squid/patches/tproxy-squid-3_20080407.patch
Just waiting on Laszlo final approval.
Some comments...
There should be a general TPROXY define, shared by all, or the common
os-independent code should always be enabled. I do not want to see #if
LINUX_TPROXY2 || LINUX_TPROXY4 defines in the general code. Such ifs
should only be seen in the lowlevel implementation. And from a code
maintenance perspective it's best is the common code is always built,
just as is done for the transparent interception code. The drawback is
that it costs one or two cpu cycles more per requests even if the
feature is not available on the platform but it's much much less likely
the feature gets broken by other changes.
+/* AYJ: do we actually need to set this again on every accept? */
+if(fd_table[fd].flags.transparent == 1) {
+comm_set_transparent(sock);
+F-flags.transparent = 1;
+}
You need to at least set the internal flag. But you most likely do no
need to make the comm call telling the kernel it's a transparent
connection (it already knows..)
Is this indended?
@@ -652,7 +673,6 @@
addr.GetAddrInfo(AI);
AI-ai_socktype = sock_type;
AI-ai_protocol = proto;
-AI-ai_flags = flags;
debugs(50, 3, comm_openex: Attempt open socket for: addr );
Regards
Henrik
--
- Xenion - http://www.xenion.com.au/ - VPS Hosting - Commercial Squid Support -
- $25/pm entry-level VPSes w/ capped bandwidth charges available in WA -
Re: TPROXY support in Squid 3
tis 2008-04-08 klockan 01:16 +0800 skrev Adrian Chadd: In fact, there shouldn't be any LINUX_TPROXY* defines in the main codetree. There should be a SERVER_SPOOF define which ties in all of the connection tracking stuff, and a clean cut API for doing TPROXY2/TPROXY4/etc socket manipulation. I say it shouldn't even be a SERVER_SPOOF define. Make the generic parts always compiled, doing nothing if spoofing capabilities is not available. Regards Henrik
Re: TPROXY support in Squid 3
Henrik Nordstrom wrote: tis 2008-04-08 klockan 01:16 +0800 skrev Adrian Chadd: In fact, there shouldn't be any LINUX_TPROXY* defines in the main codetree. There should be a SERVER_SPOOF define which ties in all of the connection tracking stuff, and a clean cut API for doing TPROXY2/TPROXY4/etc socket manipulation. I say it shouldn't even be a SERVER_SPOOF define. Make the generic parts always compiled, doing nothing if spoofing capabilities is not available. I'm agreeing with Henrik on this one. Transparent mode basics should be always available. But, baby steps people: - Get it in - Get it tested. - Polish into a class. So far we are at #1 Amos -- Please use Squid 2.6.STABLE19 or 3.0.STABLE4
Re: TPROXY support in Squid 3
tis 2008-04-08 klockan 09:57 +1200 skrev Amos Jeffries: But, baby steps people: - Get it in - Get it tested. - Polish into a class. So far we are at #1 And I won't approve the change of sprinkling #if LINUX_TPROXY4 over the code, even if it's just adding to the existing #if.. Get the basic polished up as part of #1.. it's simple, makes things much cleaner, and #3 gets a much narrover focus. Regards Henrik
Re: TPROXY support in Squid 3
On Tue, Apr 08, 2008, Henrik Nordstrom wrote: tis 2008-04-08 klockan 09:57 +1200 skrev Amos Jeffries: But, baby steps people: - Get it in - Get it tested. - Polish into a class. So far we are at #1 And I won't approve the change of sprinkling #if LINUX_TPROXY4 over the code, even if it's just adding to the existing #if.. Get the basic polished up as part of #1.. it's simple, makes things much cleaner, and #3 gets a much narrover focus. My attempt for Squid-2, step 1: http://www.creative.net.au/diffs/20080408-tproxy-fix-1.diff This just removes LINUX_TPROXY for the global state stuff and the linux_needs_tproxy global variable. Looking at it, it probably makes more sense to do this: comm_open(with flags if tproxy) if (tproxy for this request) comm_bind(non-local bind) if (comm_bind failed) disable tproxy for this request if (not tproxy) comm_bind(local bind - from the outgoing addr ACL logic) How does that sound? Adrian -- - Xenion - http://www.xenion.com.au/ - VPS Hosting - Commercial Squid Support - - $25/pm entry-level VPSes w/ capped bandwidth charges available in WA -
Re: TPROXY support in Squid 3
On Mon, Apr 07, 2008, Amos Jeffries wrote: We have come up with a 'final-beta' patch for squid-3 now. http://treenet.co.nz/projects/squid/patches/tproxy-squid-3_20080407.patch Just waiting on Laszlo final approval. It's pretty much: * adding a COMM_TRANSPARENT flag to comm_openex(...) This is the only thing I'm a bit weird on. This means that the same flag to TPROXY is used for outgoing sockets and incoming sockets? I'm thinking about seperating them just to make it absolutely clear - it doesn't matter that the backend sends the same sockopt and calls the same bind(). I don't think it matters in the FreeBSD case either as I'm pretty sure the patch I have here uses the same flag for incoming and outgoing sockets. What do you (and others) think? Adrian -- - Xenion - http://www.xenion.com.au/ - VPS Hosting - Commercial Squid Support - - $25/pm entry-level VPSes w/ capped bandwidth charges available in WA -
Re: TPROXY support in Squid 3
Adrian Chadd wrote: On Mon, Apr 07, 2008, Amos Jeffries wrote: We have come up with a 'final-beta' patch for squid-3 now. http://treenet.co.nz/projects/squid/patches/tproxy-squid-3_20080407.patch Just waiting on Laszlo final approval. It's pretty much: * adding a COMM_TRANSPARENT flag to comm_openex(...) This is the only thing I'm a bit weird on. This means that the same flag to TPROXY is used for outgoing sockets and incoming sockets? The sockets open either as listening (needs IP_TRANSPARENT option set for the OS to track new inbound connections as TPROXY targets etc.) Or as outbound spoofed connects (needs IP_TRANSPARENT option set for the OS to track the connection for special TPROXY handling.) Both need the sockopt set in the same manner on connect. Sequence matters for one, but not the other. So its done the same way for both and in sequence where it handles both nicely. I'm thinking about seperating them just to make it absolutely clear - it doesn't matter that the backend sends the same sockopt and calls the same bind(). Make what clear? that is a socket involved with transparent handling? Thats all the information that is needed by squid to operate well. It's simpler to keep it in squid as a boolean flag. Tri-states can get messy. Especially in squid case where we already have the non-transparent intercept handling for non-tproxy methods intertwined at higher levels. Keep-it-simple and all that. I don't think it matters in the FreeBSD case either as I'm pretty sure the patch I have here uses the same flag for incoming and outgoing sockets. What do you (and others) think? I think we need to clean up the two flags floating around squid rather than adding more state. Currently: fde::flags::transparent == 'intercept/non-intercept' fde::flags::tproxy == real-transparent/non-transparent (new) COMM_TRANSPARENT == real-transparent Their use is currently good for what they do. A small re-naming is all that is needed there. I'm thinking flags::intercept and flags::realtransparent. Amos -- Please use Squid 2.6.STABLE19 or 3.0.STABLE4
Re: TPROXY support in Squid 3
On Mon, Mar 31, 2008, Alex Rousskov wrote: What about Adrian plans (if I understood them correctly) to add TPROXY-like support to FreeBSD but not for TPROXY4-like API? Is that a good enough reason to continue supporting unsupported TPROXY versions? The FreeBSD API will be almost like the TPROXY-4 API. I'd suggest supporting TPROXY-2 for a few reasons: * Those who are using it may not want to track the latest kernel + TPROXY patches for various reasons (if it just works; company policy; etc.) and I think its easy enough to maintain support for both without too much hassle. * Supporting both TPROXY-2 and TPROXY-4 will (hopefully!) force someone to integrate it cleanishly and avoid the Squid-2 ip interception mess! * Thus making it easier for me to drop in a FreeBSD version of tproxy without too much hassle (or #ifdef's for that matter.) It shouldn't be that difficult to isolate the bits of the code required for spoofing the client IP in the request versus the TPROXY-specific stuff. In fact, the only tproxy-specific stuff I can really see is: * the logic in forward.c to the local bind, which can be wrapped up as part of the socket creation process, and * The initialisation code, which in the tproxy-2 case does capabilities magic. Adrian -- - Xenion - http://www.xenion.com.au/ - VPS Hosting - Commercial Squid Support - - $25/pm entry-level VPSes w/ capped bandwidth charges available in WA -
TPROXY support in Squid 3
Between Balabit and TreeNet we have come up with a working TPROXY 4+ patch. It'll be sent to trunk soon. The merge It would be a whole lot cleaner and actually less change overall if we could drop the TPROXY version 2 support from Squid-3. I know we will likely need to keep the old support in Squid-2. But with 3.1 we will have the option of saying it requires kernel 2.6.2* or later to run transparent TPROXY. Comments? Amos -- Please use Squid 2.6STABLE19 or 3.0STABLE3
Re: TPROXY support in Squid 3
On Tue, Apr 01, 2008, Amos Jeffries wrote: Between Balabit and TreeNet we have come up with a working TPROXY 4+ patch. It'll be sent to trunk soon. The merge It would be a whole lot cleaner and actually less change overall if we could drop the TPROXY version 2 support from Squid-3. I know we will likely need to keep the old support in Squid-2. But with 3.1 we will have the option of saying it requires kernel 2.6.2* or later to run transparent TPROXY. Comments? Abstract out the stuff that you need, and support both. Why? Because FreeBSD will grow the support (I've got the test code here to get ready to commit to FreeBSD) and I don't really want to try and special case a third transparency method. Adrian -- - Xenion - http://www.xenion.com.au/ - VPS Hosting - Commercial Squid Support - - $25/pm entry-level VPSes w/ capped bandwidth charges available in WA -
Re: TPROXY support in Squid 3
On Tue, 2008-04-01 at 00:18 +1300, Amos Jeffries wrote: Between Balabit and TreeNet we have come up with a working TPROXY 4+ patch. It'll be sent to trunk soon. The merge It would be a whole lot cleaner and actually less change overall if we could drop the TPROXY version 2 support from Squid-3. I know we will likely need to keep the old support in Squid-2. But with 3.1 we will have the option of saying it requires kernel 2.6.2* or later to run transparent TPROXY. Comments? We may be able to provide better comments when we see the current code. It does not have to be polished and ready for commit. Thank you, Alex.
