Re: Squid Issues and Problems
hmm sounds easy enough in concept to implement, but why not just assume it, makes it easier than writing up some code for it plus it could be a bool configuration directive for them and be done with it, it's related to the icap server, right, not the origin server? _J Henrik Nordstrom [EMAIL PROTECTED] 7/7/2008 1:07 PM It's Trend Micro way of telling the ICAP server (IWSS) that the ICAP client (the proxy) is capable of forwarding the response from the ICAP server before the entire object has been sent to the ICAP Server. Most others assume this by default without requiring the private X-TE: trailers header. The ICAP standard do not cover explicit how ICAP clients should behave in this regard. This is used by IWSS both for showing a download progress bar, and also in trickle mode where the data is slowly sent to the requestor while scanned for viruses. I do not know who proposed the X-TE: trailers name. It's a very odd name for the feature as it a) Does not have anything to do with transfer encoding (TE) b) Does not have anything to do with trailers. But with it being an X-* header it's free to mean anyting implementation specific, as long as everyone involved privately agrees on what the meaning actually is... Regards Henrik On mån, 2008-07-07 at 11:01 -0400, Jeremy Hall wrote: What do X-TE headers do? _J [EMAIL PROTECTED] 7/7/2008 5:28 AM Hi there all, Firstly many thanks for all your work on Squid thus far :) I have been testing Squid 3.0 since PRE6 in various configurations, and one of the more notable issues I have found is that when Squid is running in ICAP mode, coupled with Trend Micro IWSx (InterScan Web Security) - IWSx reports that Squid does not support the X-TE trailers for data trickling. The error is usually logged when dealing with video from CNN (at first I thought all flash video, but YouTube is unaffected) and downloading certain MS Hotfixes. There might be other triggers as well - but these seem to be the main ones. When I configure IWSx to use a different ICAP server - say NetCache or other, there is no issue or error logged and things work as expected. A quick search of the squid source provided no answers, however a search of the archives show that there was a patch for Squid 2.5 ICAP dealing with X-TE trailers: http://www.squid-cache.org/mail-archive/squid-dev/200311/att-0018/squid-icap -2_5-x-auth-user.diff http://www.squid-cache.org/~hno/changesets/squid/patches/7972.patch http://www.squid-cache.org/%7Ehno/changesets/squid/patches/7972.patch Looking at ICAPModXact.cc I can see that there are some similar references to the area's above, however most certainly the code is above my level of expertise to have a play around with to cobble something together. I was wondering if there were any plans to include support for X-TE trailers in this version? If you could let me know that would be greatly appreciated. Best Regards, Jerome http://websecurity.trendmicro.com/ Jerome Law | Solutions Architect, Regional Marketing EMEA Pacific House, Third Avenue, Globe Business Park, Marlow Buckinghamshire, SL7 1YL, United Kingdom Office: +44 (0) 1628 400586 | Mobile: +44 (0) 7979 99 33 77 http://feeds.feedburner.com/~r/Anti-MalwareBlog/~6/2 === TREND MICRO EMAIL NOTICE Trend Micro (UK) Limited, a Limited Liability Company. Registered in England No. 3698292. Registered Office: Pacific House, Third Avenue, Globe Business Park, Marlow, Bucks, SL7 1YL Telephone: +44 1628 400500 Facsimile: +44 1628 400511. The information contained in this email and any attachments is confidential and may be subject to copyright or other intellectual property protection. If you are not the intended recipient, you are not authorized to use or disclose this information, and we request that you notify us by reply mail or telephone and delete the original message from your mail system.
Re: squid3 comments
Is squid3 faster or slower than squid2? _J Alex Rousskov [EMAIL PROTECTED] 02/27/07 5:04 PM On Tue, 2007-02-27 at 13:27 +0200, Tsantilas Christos wrote: In the other hand I need a proxy with an icap client because I spent time (and continue spending) to an icap related project. Squid3 has a good icap client. The first problem someones can see in squid3 is that there are some parts derived from c-code, which are not (fully) converted to real c++ code. The second is a feeling that some parts of code are half-completed. I am thinking that maybe it is good practice for someone to start fixing this things first I agree that many Squid3 parts should be fixed, polished, or thrown away. However, I think that we should focus on making Squid3 stable first, and the performance/polishing work you are discussing should be done for v3.1. There are plenty of users who can use Squid3 that is stable but not very fast. An alternate for me is to try fix the bugs and rewrite some of the icap-related parts of the squid26-icap branch. I don't know This would be a bad idea from my biased point of view. While the code migration to Squid3 was poorly done, we are already at the point where we can make Squid3 work for your purposes instead of going back. Please do not forget that Squid2 has its own problems; it is not like you will be migrating to a great code that just needs a yet another ICAP patch! Alex.
Re: squid3 comments
Tsantilas Christos [EMAIL PROTECTED] 02/27/07 6:27 AM Hi Adrian, Adrian Chadd wrote: agree . Yes, I'd like to do all of what you've suggested above but I'm going to do it by junking most of the client-side request/reply handling routines and replacing them with stuff written from the ground up to use a sane API. Buffers won't be copied; stuff will be parsed once and parsed quickly; the whole of the URL rewrite/XFF/ Acceleration/ACL lookups/etc will be implemented as a state engine with events rather than callback-driven like it is today. Its also doable in a -lot- less code than is traversed today. I believe the only way we're going to get long-term benefits out of any improvement work is to pick some medium term goals (mostly structured around knowing what we'd like the code to look and picking incremental API and code decisions which take us down that path) and working on them piece by piece. Ok Adrian, I am watching the mailing list and I know what you want to do. I believe too that some parts of squid needs redesign, if the project wants to survive. Squid is an old and huge project. And you must continue your work because you want a fast http proxy, with cleaner code and better api's . In the other hand I need a proxy with an icap client because I spent time (and continue spending) to an icap related project. Squid3 has a good icap client. The first problem someones can see in squid3 is that there are some parts derived from c-code, which are not (fully) converted to real c++ code. The second is a feeling that some parts of code are half-completed. I am thinking that maybe it is good practice for someone to start fixing this things first An alternate for me is to try fix the bugs and rewrite some of the icap-related parts of the squid26-icap branch. I don't know Let me second this. When you start asking questions about squid3 and its stability, you get anything from it's stable to not for prime time and when you ask questions about using it in a production environment, most shy away from that. but I need ICAP support and proposals like this, although valid, scare me a little. (rewriting the client side code from scratch, not putting icap in 2.6) I don't have the time to go digging on 2.6 to make icap work better, so if you have some spare cycles it would be greatly appreciated. _J Regards, Christos
Re: squid3 comments
Adrian Chadd [EMAIL PROTECTED] 02/27/07 9:25 AM On Tue, Feb 27, 2007, Jeremy Hall wrote: Let me second this. When you start asking questions about squid3 and its stability, you get anything from it's stable to not for prime time and when you ask questions about using it in a production environment, most shy away from that. * noone's really stepped up to drag Squid-3 up to production quality. The bugs are relatively well-known and the issues with the codebase show up in bugzilla. * People seem to think we can keep adding functionality without fixing the Squid core. Which is a mess, and in my opinion, needs to be fixed first. but I need ICAP support and proposals like this, although valid, scare me a little. (rewriting the client side code from scratch, not putting icap in 2.6) I don't have the time to go digging on 2.6 to make icap work better, so if you have some spare cycles it would be greatly appreciated. I won't make icap better in squid-2.6. I want to fix the squid core so it can better support stuff like ICAP without needing to be a patch against the current codebase. I thought Squid-3 would bring this but it just hasn't. I think we are on the same page for end-term goals, but what would you recommend I do for today? Squid-3 is still a moving target. Every time I try to fix or add something interesting to Squid I keep hitting the same brick wall - the code is overly complicated for what it does. yes I agree with that. We need to spend time fixing the Squid internals and getting all of that fast, flexible and rock stable so stuff like ICAP can be implemented better. So are you saying those of us that need icap need to just wait? _J Adrian
Re: squid3 comments
Adrian Chadd [EMAIL PROTECTED] 02/27/07 10:00 AM On Tue, Feb 27, 2007, Jeremy Hall wrote: I think we are on the same page for end-term goals, but what would you recommend I do for today? Squid-3 is still a moving target. So are you saying those of us that need icap need to just wait? There's two parts. One: Alex is working on improvements to the ICAP code in Squid-3 which I hope will act as a kind of reference implementation to use in the future. Help him out any way you can. but if squid-3 is ultimately dead, why would we want to migrate to it? _J
icap in squid3
Hello, I can't remember. What was the decided path for what was once the icap_auth_scheme? I recall there was some concern about my suggestion of having the ability to use ldap://hostname/cn=%u,dc=%d,dc=name,dc=int but I don't remember what the outcome was. _J
Re: Squid 3 HEAD - ICAP Modifications
I do have need of the %d construct that I formulated. I am also formulating something with X-Authenticated-Groups that I got some recommendations from a vendor on how they want to see it. Should we use \n or %n for a newline character? I'm hoping for \n since that seems more intuitive, but the logic I wrote will need to be modified such that multiple %d, %u, and %n constructs might exist in the same uri. It doesn't m atter to me greatly what the form looks like, I just want to be able to understand what to put in my configuration file to get the desired results. I also understand that this cannot support ou's at this time. If there is an ldap squid authenticator that I'm not aware of so we can get the ou infos that'd be awesome. _J Alex Rousskov [EMAIL PROTECTED] 12/14/06 10:48 AM On Thu, 2006-12-14 at 08:47 +0100, Axel Westerhold wrote: There is one more change I am currently testing. The problem with this patch: It does not follow any ICAP document but only enables squid to get rid of the DOMAIN part frpm an NTLM Auth so I can use the result string as a query on samaccountname. Would it be possible to add this too ? I assume we talking about modifying the authenticated client username, passed via the icap_client_username_header to the ICAP server. I believe the right thing to do here is similar to what Jeremy Hall did with the icap_auth_scheme in http://www.squid-cache.org/mail-archive/squid-dev/200611/0066.html I would suggest to add an icap_authenticated_user_header_value option that takes a string with the following supported substitutions: %U -- complete username, as is %N -- username without the domain part %% -- percent I think we do not need a default value here, but would not object to Local://%U if somebody insists on having a default. The current icap_client_username_header option should be renamed to icap_authenticated_user_header_name. The current icap_client_username_encode option should be renamed to icap_authenticated_user_header_encode. On should be the default, I guess. The three icap_authenticated_user_* options can be merged into one multivalued option: icap_authenticated_user_header [name:] [encoding(] value [)] For example, icap_authenticated_user_header X-Authenticated-User: base64(Local://%U) or icap_authenticated_user_header identity(%N) Thank you, Alex. Am 14.12.2006 6:25 Uhr schrieb Alex Rousskov unter [EMAIL PROTECTED]: On Mon, 2006-12-04 at 12:57 +0100, Axel Westerhold wrote: Hi everyone, Second try this time hopefully complete. This is again patched against Squid 3 HEAD and includes 4 changes I would like to have when working with webwasher/squid systems. A.) ICAPServiceRep::TheSessionFailureLimit set through squid.conf B.) ICAPServiceRep delay for a down service set through squid.conf C.) Instead of hardcoding the Header used to transfer the username being able to set the used one through squid.conf D.) When using X-Authenticated-User in C I need the username to be base64 encoded so I added another option to turn on encoding if needed. The above changes, with minor modifications are now committed to squid3-icap branch. The corresponding patch is attached. I took the liberty to rename some of your new squid.conf options as well as polish squid.conf comments and code. A negative value for the icap_service_failure_limit disables the limit feature. Please test and let me know whether any further changes are needed. Thank you, Alex.
Re: TCP_DENIED logging as part of HTTP authentication
oh I WHOLE_HEARTEDLY agree! the logs are filled with duplicate requests because of this _J Adrian Chadd [EMAIL PROTECTED] 12/12/06 11:22 PM Hiya, I've had a few customers ask me why there's TCP_DENIEDs in the logfiles and I've tried to explain that its part of the NT authentication process. What would be nice is if we could log a bunch of different TCP_DENIEDs, covering for example: * no supplied credentials * in the middle of the Digest negotiation * supplied credentials were actually bad * denied with no chance of authenticating Thoughts? Adrian
icap in either squid-3 and squid-2
Hello, I'm trying to track down why the Content-Length header might not be sent to the icap server in squid-2's implementation of icap. I'm not running squid-3 because people tell me squid-3 isn't ready for prime time and production use, but they also tell me the icap implementation in squid-3 is somehow better than that of squid-2. The only problem I have so far with the squid-2 implementation is that if te icap server sends back a 403 in the http response, that response isn't sent back to the client, the client just is closed, causing a page cannot be displayed error. I haven't tried this with squid-3 because I don't want to manually upgrade my config file if the problem isn't resolved. Can somebody give me some ideas on how best to proceed? Thanks. _J
Re: squid-3 pre
Hi, I'm currently running 2.6-branch and am considering upgrading to 3, but I need to know how mature the code is. I would be updating for icap--I know an icap patch exists for 2.6 and I have it working with two anomolies: 1: the entire file is pulled before the content filter responds to the user rather than a deferred scan taking place and 2: if the icap server returns a 403 squid just hangs up the connection and doesn't send anything back to the browser at all. This produces page cannot be displayed errors. I have looked at the icap code a bit but was wondering if you had some pointers on where I might look to determine why the traffic is not rendering properly. _J Adrian Chadd [EMAIL PROTECTED] 11/03/06 8:00 AM So, who wants to roll the next squid-3 PRE? The ICAP changes alone make it worth it. I'm still studying for exams and thus I'm out of (much) time for two more weeks. Adrian
Re: [squid-users] 2.6S1 WCCP2 problems
The 6509 code around that release was a bit strange. I couldn't find any newer code either, so the problem might be on the cisco side. THe debug commands suggested shouldn't be too bad as the GRE-encapsulated packets don't seem to show up i the debug, but I'd start at off-peek hours just to be safe. _J Adrian Chadd [EMAIL PROTECTED] 07/17/06 8:02 AM On Mon, Jul 17, 2006, Shoebottom, Bryan wrote: Hey, It's a 6500 with 12.1(26)E code on it. It works with 2.5 stable code with the WCCP2 patch applied. If you're not afraid of a little risk, try this: I think these will turn on wccp packet logging: debug ip wccp packet debug ip wccp event make sure no logging console is on. Then, on the squid side: debug_options 80,99 .. and put the resultant logs into a bug submitted via bugzilla. Adrian
Re: WCCP2 compile problem
Hi, I noticed that those zeroized lists would sometimes have a router id of 0.0.0.0 and then either wccp would try to send to 0.0.0.0:2048, it would abort wccp2 processing when it encountered it, or it would throw an error. Did you try it after zeroizing the list? _J Henrik Nordstrom [EMAIL PROTECTED] 05/16/06 11:58 AM tis 2006-05-16 klockan 23:02 +1200 skrev Reuben Farrelly: Also just noted this *during* the compilation. This was with only the --enable-wccpv2 option with ./configure. Right. I had forgot to add --enable-wccp2 to my build script so no WCCP2 related problems was seen here obviously.. Now done, and got this parse_sockaddr_in_list related merge conflict fixed. Also cleaned up the wccp2_router directive slightly (it's a list so it can be empty) and made a few local parsing functions static to keep gcc happy. Regards Henrik
Re: WCCPv2 support
Hello, when I try to shut squid down, the code just goes on sending here_i_am messages, receiving replies, and basically ignores the fact that squid is going away. Squid keeps receiving http requests and doesn't politely send them away like it used to. Do you have thoughts on this? _J Henrik Nordstrom [EMAIL PROTECTED] 03/15/06 5:49 AM ons 2006-03-15 klockan 00:23 -0500 skrev Jeremy Hall: To build, apply the patch this came from, basically the unfinished work from the original author of wccpv2 support in suqid. Edit the src/Makefile and put in wccpv2.whatever next to every place you see wccp.whatever. Jeremy, would it be possible for you to use the devel.squid-cache.org services for hosting your developments? This would make it easier for others to follow your progress and perhaps help out where needed. Regards Henrik
Re: WCCPv2 support
Hi, I think I understand...and this version now properly handles redirect assignments. I'm excited for people to test this because it doesn't seem to crash anything, well then there's the router that seems to have some wccp bug in it, but I can't figure that out. It still doesn't handle the standard wccp (standard service0) which puzzles me--the router says service mismatch and doesn't provide more useful information. It knows it is 0:0 but something about my definition it doesn't like, very odd. To test this you'll need to mod the cf.data.pre bits a little to include Config.Wccp2.router_id (yeah I know bad bad bad) we're really not a router id and so perhaps we should be a cache_id and you'll want to make Config.Wccp2.routers (note the s) be a wordlist. This doesn't do multicast yet, but I want to get unicast working first, then I'll think about multicast. It doesn't check to make sure everybody is seeing him, this not really a problem in unicast mode but multicast mode will definitely cause problems no question about that. For now it only handles one cache. As I finally understand the hash bits a little, there are some comments about how one might decide to roll across caches--me not being a squid developer really I don't have much thought on how one might accomplish round-robining but maybe somebody could take some of the same ideas and put them into the code. It does not understand alternate assignment and looks like the wccp module for hte kernel might need to be modified--it assumes standard/0 for service only (I think, if I understand the code) well hopefully somebody can at least get it to compile. It does seem to work in a multirouter setup though. sorry about top-posting the mail software sucks :( _J Henrik Nordstrom [EMAIL PROTECTED] 03/13/06 5:41 AM sön 2006-03-12 klockan 23:04 -0500 skrev Jeremy Hall: I'm not sure whether it is a good idea to wait until routers are included (until we're 2waying with them, somebody understanding te protocol better than I would have great comments I'm sure) There is multiple negotiations going on. a) router-cache negotiation. This is the HERE_I_AM - I_SEE_YOU handshake. It is completed by the cache getting included the list of cache servers in I_SEE_YOU messages from the router, which requires a HERE_I_AM - I_SEE_YOU - HERE_I_AM - I_SEE_YOU sequence (first three to verify connectivity, the last to inform the designated cache you are available..) b) Election of the designated cache. Ther is an election algorithm (outside the protocol) defining which cache is the designated cache. The recommended algorithm is that the cache with the lowest IP gets elected. The list of potential designated cache servers is found in the I_SEE_YOU packets. If you are not there, or not the lowest IP there then you are not the designated cache.. c) designated cache - router assignment using REDIRECT_ASSIGNMENT messages, based on the information it received in the I_SEE_YOU packet from the router. In service groups the designated cache election gets a little messy as each router maintains it's own WCCP state and things can get out of sync if not all equipment is configured proper or some device is restarted. Consider for example a potential designated cache only announcing itself to a subset of the routers in the service group.. but these fine details can be left to deal with later.. The 1.5 HERE_I_AM_T delay helps a lot in reducing this. Regards Henrik wccpv2.2.c Description: Binary data
Re: WCCPv2 support
Hi, Is this better? I've tried to clean up all the structs that are static and reworked the here_i_am messages going out. The redirect assign still hasn't been worked. As for ftp, that's a little project over here on the same box as squid and I don't know how to represent that service in the config, so I'm just putting it in manually in the code. If you have suggestions on how to modify this code to allow services to be configured, that'd be awesome!!! now the routers aren't complaining with this code, but it doesn't draw any traffic yet, it happily pings i_see_you and here_i_am messages back and forth and notices if it is in sync. I'm not sure whether it is a good idea to wait until routers are included (until we're 2waying with them, somebody understanding te protocol better than I would have great comments I'm sure) anyway see you _J Henrik Nordstrom [EMAIL PROTECTED] 03/11/06 11:50 AM fre 2006-03-10 klockan 11:33 -0500 skrev Jeremy Hall: Hello, I think I have implemented the cbdata stuff properly. This has the following problems: 1: does not send valid router or cache view components 2: redirect assignment is totally untested and showing bad service id Ok. 3: standard service for http rejected by the router for some reason You also seem to have some mentioning of a ftp service in there. Why? Squid can not transparently proxy ftp... so what does it do right? it can keep track of wccpv2 received ids :-) Your thoughts welcome. You'll want to find the hooks (part of the patch) as this is only the wccpv2.c file. You seem to attempt handling several WCCPv2 packets as static structs.. this will be a bit difficult. It's better if you have one struct per layer, using the unterminated array hack for the variable data fields. then pack/unpack data to more manageable forms internally where needed. Regards Henrik wccpv2.1.c Description: Binary data
