NTLM authentication popups, etc

[Adrian Chadd]

I'm working on a couple of paid squid + active directory deployments
and they're both seeing the occasional NTLM auth popup happening.

The workaround is pretty simple - just enable the IP auth cache. This
however doesn't solve the fundamental problem(s), whatever they are.

The symptom is logs like this:

[2009/06/15 16:20:17, 1] libsmb/ntlmssp.c:ntlmssp_update(334)
  got NTLMSSP command 1, expected 3

And vice versa (expected 3, got 1.) These correspond to states in
samba/source/include/ntlmssp.h - 1 is NTLMSSP_NEGOTIATE; 3 is
NTLMSSP_AUTH.

The conclusion here is that there's a disconnect between the
authentication state of the client -and- the authentication state of
ntlm_auth.

I'm trying to eliminate the possibilities here.

The stateful helper stuff seems correct enough, so requests aren't
being queued to already busy stateful helpers.

The other two possibilities I can immediately think of:

* 1 - authentication is aborted somewhere for whatever reason; an
authentication helper is stuck at the wrong point in the state engine;
the next request coming along starts at NTLMSSP_NEGOTIATE but the
ntlm_auth helper it is handed to is at NTLMSSP_AUTH (from the partial
authentication attempt earlier); error
* 2 - the web browser is stuffing different phases of the negotiation
down different connections to the proxy.

Now, debugging (1) shouldn't be difficult at all. I'm going to try and
determine the code paths that lead to and from an aborted auth
request, add in some debugging and see if the helper is closed.

Debugging (2) without full logs (impractical in this environment) and
full traffic dump (again, impractical in production) is going to be a
bit more difficult. I'm thinking about adding some hacky code to the
Squid ntlm auth class which keeps a log of the auth blobs
sent/received from/to the client and ntlm_auth. I can then dump the
entire conversation out to cache.log whenever authentication
fails/errors. This should at least give me a hint as to what is going
on.

(1) can explain the client state == NTLMSSP_NEGOTATE but ntlm_auth
state is NTLMSSP_AUTH problem but not vice versa. (2) explains both.
It is quite possible it is the combination of both however.

Now, the reason this is getting somewhat annoying and why I'd like to
try and understand/fix it is that -another- problem seen by one of
these clients is negotiate/ntlm authentication from IE (at least IE8)
through Squid. I've got packet dumps showing the browser sending
different phases of the negotiation down separate proxy connections
and then reusing the original one incorrectly. My medium term plan is
to take whatever evidence I have of this behaviour and throw it at the
IE group(s) at Microsoft but in the short term I'd like to make
certain the proxy authentication side of things is completely
blameless before I hand off stuff to third parties.

Ideas? Comments?



adrian

Re: NTLM authentication popups, etc

[Robert Collins]

On Tue, 2009-06-16 at 15:14 +0800, Adrian Chadd wrote:
> 
> 
> The stateful helper stuff seems correct enough, so requests aren't
> being queued to already busy stateful helpers.

IIRC you can multiplex requests on single stateful helpers now (see
deferred_requests - the count of how many concurrent transactions are
going on on that helper). I may be wrong. Regardless, you *must* queue
to the same helper though.

A trace of 84,9 29,9 may help.

> The other two possibilities I can immediately think of:
> 
> * 1 - authentication is aborted somewhere for whatever reason; an
> authentication helper is stuck at the wrong point in the state engine;
> the next request coming along starts at NTLMSSP_NEGOTIATE but the
> ntlm_auth helper it is handed to is at NTLMSSP_AUTH (from the partial
> authentication attempt earlier); error

You can detect this by ascertaining the ntlm user request state (which
the trace above should give us) at the time of the failure. We preprend
the data with things like 'KK' to indicate where we think the state is
at.

> * 2 - the web browser is stuffing different phases of the negotiation
> down different connections to the proxy.

This would be fundamentally broken. OTOH I've seen firefox do this with
digest authentication so its entirely possible.

> Now, debugging (1) shouldn't be difficult at all. I'm going to try and
> determine the code paths that lead to and from an aborted auth
> request, add in some debugging and see if the helper is closed.

authenticateNTLMReleaseServer should be called to release a in progress
transaction on the helper.

> Debugging (2) without full logs (impractical in this environment) and
> full traffic dump (again, impractical in production) is going to be a
> bit more difficult. I'm thinking about adding some hacky code to the
> Squid ntlm auth class which keeps a log of the auth blobs
> sent/received from/to the client and ntlm_auth. I can then dump the
> entire conversation out to cache.log whenever authentication
> fails/errors. This should at least give me a hint as to what is going
> on.

Just the two settings I'm asking for should be enough. Just get a couple
of failed occurrences then reconfigure back to normal.

> (1) can explain the client state == NTLMSSP_NEGOTATE but ntlm_auth
> state is NTLMSSP_AUTH problem but not vice versa. (2) explains both.
> It is quite possible it is the combination of both however.
> 
> Now, the reason this is getting somewhat annoying and why I'd like to
> try and understand/fix it is that -another- problem seen by one of
> these clients is negotiate/ntlm authentication from IE (at least IE8)
> through Squid. I've got packet dumps showing the browser sending
> different phases of the negotiation down separate proxy connections
> and then reusing the original one incorrectly. My medium term plan is
> to take whatever evidence I have of this behaviour and throw it at the
> IE group(s) at Microsoft but in the short term I'd like to make
> certain the proxy authentication side of things is completely
> blameless before I hand off stuff to third parties.
> 
> Ideas? Comments? 

Blame MS :).

-Rob


signature.asc
Description: This is a digitally signed message part

Re: Problem with cached entries w/ETag and request without If-None-Match header

[Jason Noble]

You're right, looks like I read that section too quickly.  After reading 
the RFC more carefully, it appears that the case I'm having issues with 
is undefined.  The closest thing I can find is 13.3.4:


An HTTP/1.1 caching proxy, upon receiving a conditional request that
  includes both a Last-Modified date and one or more entity tags as
  cache validators, MUST NOT return a locally cached response to the
  client unless that cached response is consistent with all of the
  conditional header fields in the request.


But I'm actually interested in the case where the caching proxy receives 
a non-conditional request.  The current behavior of Squid is to return a 
cached entity.  It seems to me that not returning cached entries that 
include conditional header fields would be more consistent with the 
behavior of "not returning a locally cached response to the client 
unless that cached response is consistent with all of the conditional 
header fields in the request"  (i.e. the absence of conditional header 
fields does not match the presence of one or more conditional header 
fields.)  If this case is indeed defined in the RFC, I'd be interested 
to know where.


Thanks,
Jason

Mark Nottingham wrote:
Selecting request headers are specified by Vary; If-None-Match is a 
conditional request header.


Cheers,


On 16/06/2009, at 12:44 AM, Jason Noble wrote:


From RFC 2616 13.6:
...
When the cache receives a subsequent request whose Request-URI 
specifies one or
more cache entries including a Vary header field, the cache MUST NOT 
use such a

cache entry to construct a response to the new request unless all of the
selecting request-headers present in the new request match the 
corresponding

stored request-headers in the original request. ...


For the case in question, all selecting request headers do not match 
the stored request headers.  Therefore, the cache must not use the 
stored entry to construct a response.


--Jason

Mark Nottingham wrote:

What requirement in RFC2616 does this violate?

On 13/06/2009, at 3:02 AM, Jason Noble wrote:

I recently ran into a bug on Squid 2.7 regarding cached content 
with ETags.  Currently, if all cached entries for a URL include 
ETags, and a request is received for said URL with no If-None-Match 
header, Squid will serve a cached entry.  This behavior does not 
follow RFC 2616.  I have attached a patch that prevents Squid from 
serving the cached entries in said case here:  
http://www.squid-cache.org/bugs/show_bug.cgi?id=2677


I would appreciate any feedback regarding this patch.

Thanks,
Jason


--
Mark Nottingham   m...@yahoo-inc.com




--
Mark Nottingham   m...@yahoo-inc.com

Re: how to configure poxy cache server ?

[Amos Jeffries]

On Mon, 15 Jun 2009 21:28:51 -0700 (PDT), aungmyotint 
wrote:
> i have these address ,gateway 172.24.0.3 and proxy ip address is
> ygncache.mptadsl.net.mm .my ISP gives DHCP ip address to us, so how can i
> confifure into the squid.conf . now i have configured like these
> 
> http_port 192.168.11.1:9090
> cache_mem 512 MB
> cache_dir ufs /usr/local/squid/var/cache 1 16 256
> acl mynetwork 192.168.1.0/24
> http_acces allow mynetwork
> cache_peer ygncache.mtpadsl.net.mm parent 8080 0 default-no query
> never_direct allow all
> 
> but i cant access internet from my clients . so what configuration do i
> configure ? plz help me

I see two proxies involved:
  192.168.11.1:9090
and
 ygncache.mtpadsl.net.mm:8080

Which one is the DHCP assigned one?

On 192.168.11.1:9090, the _outgoing_ IP Squid uses is DHCP configured it
does not matter. Only the IP clients see to connect for proxy access. Which
is 192.168.11.1:9090.

On ygncache.mtpadsl.net.mm, the squid at 192.168.11.1:9090 will need to be
reconfigured. It looks up DNS address for the peer during configure. There
is currently no way to connect persistent peer links to a constantly
changing source.


Amos

Re: A little help with cbdata and requestLink

[Mark Nottingham]

Thanks. Anybody else have a second to look?


On 11/06/2009, at 11:28 PM, Amos Jeffries wrote:


Mark Nottingham wrote:

Would someone mind taking a quick look at this patch:
 http://www.squid-cache.org/bugs/attachment.cgi?id=1989
and telling me if I've royally stuffed up with managing fwdState  
and request linking?

It's to make miss_access a slow lookup...


Looks okay to these uneducated eyes.  Probably best to wait for  
someone else to double-check before a HEAD commit, but IMO it looks  
good enough for a patching.


This one is long-awaited by many. Thanks.

Amos
--
Please be using
 Current Stable Squid 2.7.STABLE6 or 3.0.STABLE15
 Current Beta Squid 3.1.0.8 or 3.0.STABLE16-RC1


--
Mark Nottingham   m...@yahoo-inc.com

Re: [squid-users] NONE/411 Length Required

[Mark Nottingham]

[ moving to squid-dev ]

From what I can see, the site is using JavaScript to do autocomplete  
on a search field. The autocomplete requests use POST, but without a  
body.


With Firefox, this results in a POST request without a body; i.e., it  
doesn't have transfer-encoding *or* content-length.


Such a POST request is legal (although atypical; Safari and I think  
others will include a Content-Length: 0 to signal no body explicitly).  
See .


I think the right thing to do here is for Squid to only 411 when  
there's a transfer-encoding present; if there's no content-length,  
it's safe to assume 0 length.


Cheers,


On 17/06/2009, at 2:07 PM, Bijayant Kumar wrote:



Bijayant Kumar


--- On Mon, 15/6/09, Bijayant Kumar  wrote:


From: Bijayant Kumar 
Subject: Re: [squid-users] NONE/411 Length Required
To: "squid users" 
Date: Monday, 15 June, 2009, 6:48 PM


--- On Mon, 15/6/09, Amos Jeffries 
wrote:


From: Amos Jeffries 
Subject: Re: [squid-users] NONE/411 Length Required
To: "Bijayant Kumar" 
Cc: "squid users" 
Date: Monday, 15 June, 2009, 6:06 PM
Bijayant Kumar wrote:

Hello list,

I have Squid version 3.0.STABLE 10 installed on

Gentoo

linux box. All things are working fine, means caching
proxying etc. There is a problem with some sites. When

I am

accessing one of those sites, in access.log I am

getting


NONE/411 3692 POST http://.justdial.com/autosuggest_category_query_main.php?

- NONE/- text/html


And on the webpage I am getting whole error page

of

squid. Actually its a search related page. In the

search

criteria field as soon as I am typing after two words

I am
getting this error. The website in a question is "http://justdial.com 
". But it works without the Squid.



I tried to capture the http headers also which

are as

below


http://.justdial.com/autosuggest_category_query_main.php?city=Bangalore&search=Ka



POST



/autosuggest_category_query_main.php?city=Bangalore&search=Ka

HTTP/1.1


Host: .justdial.com

User-Agent: Mozilla/5.0 (X11; U; Linux i686;

en-US;

rv:1.8.1.16) Gecko/20080807 Firefox/2.0.0.16


Accept:


text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/ 
plain;q=0.8,image/png,*/*;q=0.5


Accept-Language: en-us,en;q=0.7,hi;q=0.3

Accept-Encoding: gzip,deflate

Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7

Keep-Alive: 300

Connection: keep-alive

Referer: http://.justdial.com/

Cookie:

PHPSESSID=d1d12004187d4bf1f084a1252ec46cef;



__utma=79653650.2087995718.1245064656.1245064656.1245064656.1;

__utmb=79653650; __utmc=79653650;

__utmz=79653650.1245064656.1.1.utmccn=(direct)|utmcsr=(direct)| 
utmcmd=(none);

CITY=Bangalore


Pragma: no-cache

Cache-Control: no-cache



HTTP/1.x 411 Length Required

Server: squid/3.0.STABLE10

Mime-Version: 1.0

Date: Mon, 15 Jun 2009 11:18:10 GMT

Content-Type: text/html

Content-Length: 3287

Expires: Mon, 15 Jun 2009 11:18:10 GMT

X-Squid-Error: ERR_INVALID_REQ 0

X-Cache: MISS from bijayant.kavach.blr

X-Cache-Lookup: NONE from

bijayant.kavach.blr:3128


Via: 1.0 bijayant.kavach.blr

(squid/3.0.STABLE10)


Proxy-Connection: close

Please suggest me what could be the reason and

how to

resolve this. Any help/pointer can be a very helpful

for me.




Bijayant Kumar


 Get your new

Email

address!

Grab the Email name you've always wanted before

someone else does!

http://mail.promotions.yahoo.com/newdomains/aa/



NONE - no upstream source.
411  - "Content-Length missing"

HTTP requires a Content-Length: header on POST

requests.




How to resolve this issue. Because the website is on internet and  
its working fine without the squid. When I am bypassing the proxy, I  
am not getting any type of error.


Can't this website be accessed through the Squid?



Amos
-- Please be using
 Current Stable Squid 2.7.STABLE6 or

3.0.STABLE15

 Current Beta Squid 3.1.0.8 or

3.0.STABLE16-RC1





 New Email addresses available on
Yahoo!
Get the Email name you've always wanted on the new
@ymail and @rocketmail.
Hurry before someone else does!
http://mail.promotions.yahoo.com/newdomains/aa/




 Get your new Email address!
Grab the Email name you've always wanted before someone else does!
http://mail.promotions.yahoo.com/newdomains/aa/


--
Mark Nottingham   m...@yahoo-inc.com