Re: My debug problem

[Henrik Nordstrom]

On Sat, 24 Jan 2004, Robert Collins wrote:

> It would be great if you could throw that into my mempools squid-3.0
> branch, or hold off until I've merged that. The changes in there are
> quite a lot, it would be easier to cookify it, than to merge a
> cookifying change into it...

This is [EMAIL PROTECTED]/squid--MemPools--3.0, right?

A cached rev would had been nice there.. it now starts from 
[EMAIL PROTECTED]/squid--HEAD--3.0--patch-79 which takes some 
time for tla to patch up.. but I guess it will be finished eventually.

Regards
Henrik

Re: [Fwd: squid3 bug still exists]

[Henrik Nordstrom]

On Thu, 5 Feb 2004, David Nicklay wrote:

> Hi,
> 
> In HttpStateData::cacheableReply() returns -1 if the reply from the 
> origin server is HTTP_NOT_FOUND.  If I go back and change this to return 
> 1 for HTTP_NOT_FOUND, squid caches the request just fine.  As near as I 
> can tell the stored object never gets made public, when it is negatively 
> cached.  Is there any way to override the 'uncacheable' replies from the 
> config?  It is generally bad for uncacheable replies to get through in a 
> reverse proxy setup.

What needs to be done is to figure out why the vary marker object is not 
created on negatively cached object.

This object is created in storeSetPublicKey

Regards
Henrik

Re: squid-3.0 CVS httpMakeVaryMark

[David Nicklay]

Hi,

Scratch that.  I realize what it is doing now. :-)

David Nicklay wrote:
Hi,

In http.cc in the function httpMakeVaryMark(), there is a call to 
Tolower which is called on the header name.  httpHeaderGetByName is then 
called to get the value of that header.  From doing some debugging, it 
appears that the value always returns NULL in my tests, because the 
header is stored in its mixed upper/lower case form.  Is this just 
broken, or am I just missing something?

--
David Nicklay
Location: CNN Center - SE0811A
Office: 404-827-2698Cell: 404-545-6218

squid-3.0 CVS httpMakeVaryMark

[David Nicklay]

Hi,

In http.cc in the function httpMakeVaryMark(), there is a call to 
Tolower which is called on the header name.  httpHeaderGetByName is then 
called to get the value of that header.  From doing some debugging, it 
appears that the value always returns NULL in my tests, because the 
header is stored in its mixed upper/lower case form.  Is this just 
broken, or am I just missing something?

--
David Nicklay
Location: CNN Center - SE0811A
Office: 404-827-2698Cell: 404-545-6218

[Fwd: squid3 bug still exists]

[David Nicklay]

Hi,

In HttpStateData::cacheableReply() returns -1 if the reply from the 
origin server is HTTP_NOT_FOUND.  If I go back and change this to return 
1 for HTTP_NOT_FOUND, squid caches the request just fine.  As near as I 
can tell the stored object never gets made public, when it is negatively 
cached.  Is there any way to override the 'uncacheable' replies from the 
config?  It is generally bad for uncacheable replies to get through in a 
reverse proxy setup.

 Original Message 
Subject: squid3 bug still exists
Date: Fri, 23 Jan 2004 15:06:02 -0500
From: David Nicklay <[EMAIL PROTECTED]>
To: [EMAIL PROTECTED]
Hi,

I did some searching and found that the bug we are encountering has
existed since at least squid 2.5
Bug #616

Negative cached 404 replies with VARY header never matches

--
David Nicklay
Location: CNN Center - SE0811A
Office: 404-827-2698Cell: 404-545-6218
--
David Nicklay
Location: CNN Center - SE0811A
Office: 404-827-2698Cell: 404-545-6218

Re: Squid logfile >2GB problem

[Henrik Nordstrom]

On Thu, 5 Feb 2004, Luigi Gangitano wrote:

> Is there any plan to implement the rotation policies that where
> discussed in bug report #319?

As this is trivial to do outside the Squid binary I don't really see why
spend time on it. The user can just use logrotate or similar log rotation
policy tool. There is no problem calling logrotate every 5 minutes if so 
is required as it is designed for such use.

But if there is a reasonable patch submitted I see no reason not to accept 
the patch.

Regards
Henrik

Re: Squid logfile >2GB problem

[Henrik Nordstrom]

On Thu, 5 Feb 2004, Duane Wessels wrote:

> It looks to me like the --enable-large-files trick simply adds
> -D_FILE_OFFSET_BITS=64 to the compiler flags, so it seems relatively
> harmless.

In addition to enabling support for large files this increases the size of
all the file size related data types from 32 bits to 64 bits, and also
increases the maximum allowed object size.

The code has not at all been audited if this is safe and is why the flag 
does not exists in 2.5. It is known there is quite many casts to int, and 
I see it quite likely there is at least some wrong usage.

In addition this modifies the on-disk storage for the same reason and the
cache is not compatible between 32 and 64 bit file offset Squid binaries.

Regards
Henrik

Re: Squid logfile >2GB problem

[Luigi Gangitano]

> It looks to me like the --enable-large-files trick simply adds
> -D_FILE_OFFSET_BITS=64 to the compiler flags, so it seems relatively
> harmless.

I've just packaged a 2.5.STABLE4 version with _FILE_OFFSET_BITS=64 in
cflags. It's actually under testing. If everything goes well I plan to
upload it to debian (for more/deeper testing).

> However, I still think that squid users will be happier in the long run if
> they install a simple cron job to rotate their log files instead of letting
> them get to 2GB in size.

This is true until something weird happens and you get 2GB o logs in a
couple of hours (this is what happened to the bug submitter...).

I'm sure that disk size will be the next limit, but until then... :-)

Is there any plan to implement the rotation policies that where
discussed in bug report #319?

Thanks,

-- 
 Luigi Gangitano -- <[EMAIL PROTECTED]> -- <[EMAIL PROTECTED]>
 GPG: 1024D/924C0C26: 12F8 9C03 89D3 DB4A 9972  C24A F19B A618 924C 0C26


signature.asc
Description: Questa parte del messaggio =?ISO-8859-1?Q?=E8?= firmata

Re: Squid logfile >2GB problem

[Duane Wessels]

On Wed, 4 Feb 2004, Luigi Gangitano wrote:

> Hi,
> I'm the Debian maintainer for squid and got a bugreport about the 3GB
> limit on logfiles. I saw bug #319 and noticed that no solution was
> implemented except the --enable-large-files option in HEAD. Is there any
> way to backport that patch for 2.5.STABLE4?

Hi Luigi,

It looks to me like the --enable-large-files trick simply adds
-D_FILE_OFFSET_BITS=64 to the compiler flags, so it seems relatively
harmless.

I don't mind adding it to the squid-2.5 branch.  Anyone else?

However, I still think that squid users will be happier in the long run if
they install a simple cron job to rotate their log files instead of letting
them get to 2GB in size.

Duane W.

Re: Squid-2.5.STABLE5 planned

[Henrik Nordstrom]

On Thu, 5 Feb 2004, Serassio Guido wrote:

> FYI: For the use of NTLM NEGOTIATE with Samba 3, the ntlm_auth from the 
> incoming Samba 3.0.2 release is needed.

Thanks

Regards
Henrik

Re: Squid-2.5.STABLE5 planned

[Serassio Guido]

Hi Robert, Henrik,

At 23.41 04/02/2004, Robert Collins wrote:

On Thu, 2004-02-05 at 01:45, Henrik Nordstrom wrote:

> The patch to fix the challenge and helper maintenance rips out the
> "deferred" support layer in the stateful helpers. It was too complex to
> look into what things went wrong there and with the use somewhat dubious
> in terms of stability I see no reason to keep this.
It was only ever needed for challenge reuse. I haven't looked at the
patch you've done yet - have you eliminated the challenge reuse option
too? (You need to). The reason for the deferring was to ensure that the
helper didn't change the challenge unexpectedly : the early helpers
didn't have enough protocol control to choose their own challenge. With
Samba3 thats fixed. However, the only reason for reuse was to avoid
saturating the old compatability mode interface to the PDC with
expensive calls, which invariable dropped the link to the helper. Thus -
get rid of reuse and it will be a lot simpler.
> Robert: If you want to keep the "deferrer" stateful helper support in
> Squid-3 then you may want to look into the interactions with NTLM helpers
> as there was very bad interactions between the two in Squid-2.5. If not
> lets rip it out in favor of implementing the overlapped requests
> management. Implementing overlapped requests for stateful helpers is
> significantly with the deferred layer ripped out.
^easier^ ?

Yes, rip it out. We can't quite eliminate stateful helpers completely,
until we can get a challenge from helper 1 and submit it to helper 2.
But we can certainly (with Samba3 & Guido's helper) have multiple
parallel challenges in use.
Currently I'm out of office. I will merge in the nt-2_5 branch the latest 
NTLM changes Saturday and after I will start to testing immediately to see 
if the Windows helper is now happy.

FYI: For the use of NTLM NEGOTIATE with Samba 3, the ntlm_auth from the 
incoming Samba 3.0.2 release is needed.

Regards

Guido



-

Guido Serassio
Acme Consulting S.r.l.
Via Gorizia, 69 10136 - Torino - ITALY
Tel. : +39.011.3249426  Fax. : +39.011.3293665
Email: [EMAIL PROTECTED]
WWW: http://www.acmeconsulting.it/

RE: Squid-3 Release Date\Questions

[Henrik Nordstrom]

On Thu, 5 Feb 2004, Arbelaez, Jim wrote:

> If squid was a webserver it would be as simple as a webpage that
> performed a response.redirect "https://service.mydomain.com";.  I don't
> want to load apache to accomplish this action.  There has to be a way to
> use squid to redirect the browser

There is no problem doing this in Squid.

The easiest method is to use a redirector.

You can also use deny_info.

Regards
Henrik

RE: Squid-3 Release Date\Questions

[Arbelaez, Jim]

Henrik,

We now have the ability to use (https) --->
(Squidbox)---http--->(Exchange 2000 OWA) BUT we need the ability for our
users to type (http)>(Squidbox) and get redirected to (https) at the
client automatically so the client can have a secure session.

We have looked at the redirect program code below but it does not work:

#!/usr/local/bin/perl
$|=1;
while (<>) {
@X = split;
$url = $X[0];
if ($url =~ /^http:\/\/internal\.foo\.com/) {
$url =~ s/^http/https/;
$url =~ s/internal/secure/;
print "302:$url\n";
} else {
print "$url\n";
}
}

Any help or suggestions would be useful


Jimmy A.



-Original Message-
From: Henrik Nordstrom [mailto:[EMAIL PROTECTED] 
Sent: Thursday, January 29, 2004 6:15 AM
To: Arbelaez, Jim
Cc: [EMAIL PROTECTED]
Subject: Re: Squid-3 Release Date\Questions

On Tue, 27 Jan 2004, Arbelaez, Jim wrote:

> What is the ETA for stable release of "Squid-3"?

Good question. I don't have an answer at the moment. Robert?

> We are wanting to use this version in a proxy configuration for
> Microsoft Outlook Web Access servers over SSL.  Our goal is to place
> only unix based systems in our DMZ and not Windows based.
>
> Note:  If you have the information to provide concerning the SSL fix
> and Squid 2.5 stable proxying exchange 2000 Outlook Web Access it
would
> be much appreciated.

Squid-2.5 + SSL update works for https->https proxying of OWA with the
help of a small redirector to clean up the URLs in the reverse proxy,
but
lacks some features required for https->http proxying of OWA.

Regards
Henrik

RE: Squid-3 Release Date\Questions

[Henrik Nordstrom]

On Wed, 4 Feb 2004, Arbelaez, Jim wrote:

> We have looked at the redirect program code below but it does not work:

The same approach works for many others.

Note: "httpd_accel_port virtual" makes it easier to see difference between 
http_port and https_port in Squid-2.5.

Regards
Henrik

Re: Squid-2.5.STABLE5 planned

[Henrik Nordstrom]

On Thu, 5 Feb 2004, Robert Collins wrote:

> It was only ever needed for challenge reuse. I haven't looked at the
> patch you've done yet - have you eliminated the challenge reuse option
> too?

No, but it has been restricted to not have more than one concurrent client 
per challenge so it is kind of worthless at the moment.

> (You need to). The reason for the deferring was to ensure that the
> helper didn't change the challenge unexpectedly

And I can tell you this failed totally. Helpers were rechallenged randomly 
with no apparent pattern.

> the early helpers didn't have enough protocol control to choose their
> own challenge. With Samba3 thats fixed. However, the only reason for
> reuse was to avoid saturating the old compatability mode interface to
> the PDC with expensive calls, which invariable dropped the link to the
> helper. Thus - get rid of reuse and it will be a lot simpler.

Correct. With the winbind helpers the only reason to use challenge reuse 
is to not have to run as many helpers, but the complexity involved for 
doing this is not worth the effort.

> > management. Implementing overlapped requests for stateful helpers is
> > significantly with the deferred layer ripped out.
> 
> ^easier^ ?

Yes ;-)

> Yes, rip it out. We can't quite eliminate stateful helpers completely,
> until we can get a challenge from helper 1 and submit it to helper 2.
> But we can certainly (with Samba3 & Guido's helper) have multiple
> parallel challenges in use.

The stateful helper design should not be eleminated in my opinion. That
part is good for this purpose. Making the helper challenge neutral puts
too much requirements on the helper design.

The overlapping requests design solves all of these problems quite nicely
by allowing a single helper to maintain multiple states. The helper
protocol only needs very minimal changes to support this, and with a
threaded helper or similar the exact same net effect as if the stateful
helpers was eleminated completely.

Regards
Henrik

Re: Robert & busyness

[Vladimir V. Ivanov]

Hi! 
On Thu, 05 Feb 2004 10:00:06 +1100
Robert Collins <[EMAIL PROTECTED]> wrote:
> 
> I haven't reviewed bugzilla recently, but when I last did, the only
> real killer bug in squid-3, that prevents usage equivalent to
> squid-2.5(modulo the not forward ported fixes) is the range bug.
> 
> The range bug: given a request to an offset in a range, under some
> conditions squid will request from the store an unavailable memory
> range. This triggers an assert that dumps the available ranges and
> terminates.

Please also take attention at the bug#900+patch, which is related to
range requests. 

-- 
Regards,
Vladimir Ivanov

Re: Squid-2.5.STABLE5 planned

[Robert Collins]

On Thu, 2004-02-05 at 01:45, Henrik Nordstrom wrote:

> The patch to fix the challenge and helper maintenance rips out the
> "deferred" support layer in the stateful helpers. It was too complex to
> look into what things went wrong there and with the use somewhat dubious
> in terms of stability I see no reason to keep this.

It was only ever needed for challenge reuse. I haven't looked at the
patch you've done yet - have you eliminated the challenge reuse option
too? (You need to). The reason for the deferring was to ensure that the
helper didn't change the challenge unexpectedly : the early helpers
didn't have enough protocol control to choose their own challenge. With
Samba3 thats fixed. However, the only reason for reuse was to avoid
saturating the old compatability mode interface to the PDC with
expensive calls, which invariable dropped the link to the helper. Thus -
get rid of reuse and it will be a lot simpler.

> Robert: If you want to keep the "deferrer" stateful helper support in
> Squid-3 then you may want to look into the interactions with NTLM helpers 
> as there was very bad interactions between the two in Squid-2.5. If not 
> lets rip it out in favor of implementing the overlapped requests
> management. Implementing overlapped requests for stateful helpers is
> significantly with the deferred layer ripped out.

^easier^ ?

Yes, rip it out. We can't quite eliminate stateful helpers completely,
until we can get a challenge from helper 1 and submit it to helper 2.
But we can certainly (with Samba3 & Guido's helper) have multiple
parallel challenges in use.

IMO the helper protocol should be revamped to reflect this - it can be
made simpler.

Rob
-- 
GPG key available at: .


signature.asc
Description: This is a digitally signed message part

Arch mirror of CVS

[Robert Collins]

I've been trialling this for a while now. I've got cronnable (but still
run by hand as I'm not always connected to the net on my laptop) scripts
that mirror CVS commits across to Arch.

The mirrored branches are squid3 HEAD and squid2 SQUID_2_5.

I wrote a mini-howto for Henrik, but it applies to any Squid development
that folk may want to do in Arch:
http://www.robertcollins.net/squid-arch.htm

If anyone here plans to use arch, please tag from my branches:
[EMAIL PROTECTED]/squid--HEAD--2.5
or
[EMAIL PROTECTED]/squid--HEAD--3.0
in order to preserve the global file-id's.

Committing back into CVS may be done any which way, but if you want me
to synthesis the logs so that the Arch record contains the merge history
(which CVS cannot), drop me a merge request (your branch, CVS branch,
your repository location), and I'll do that (usually within 48 hours.)

Cheers,
Rob


-- 
GPG key available at: .


signature.asc
Description: This is a digitally signed message part

Robert & busyness

[Robert Collins]

I've been somewhat inactive here recently. Given that I'm meant to be
doing the squid-3 release management, thats, well, bad.

So this is just a quick apology to folk : I got overcommitted with
customers for a while. For the next month, that won't change. After
that, I hope to be doing a bit less frenetic role, with much more
predictable hours, allowing me some time to focus on squid.

I haven't reviewed bugzilla recently, but when I last did, the only real
killer bug in squid-3, that prevents usage equivalent to squid-2.5
(modulo the not forward ported fixes) is the range bug.

The range bug: given a request to an offset in a range, under some
conditions squid will request from the store an unavailable memory
range. This triggers an assert that dumps the available ranges and
terminates.

This is caused because we have no sane notification for new data - data
can only arrive at a predicted offset - yet http allows for arbitrary
offsets to arrive.

Henrik and I disagree on the right fix here. See the previous thread for
details. However, after reviewing the HTTP 100 series support for a
prospect a few months back, I'm convinced that Henriks 'easy' solution
won't cut it in the long run. Arbitrary 1xx support requires
notification to the client of both the nature of the message, and it's
content. That same notification can allow use to deliver the real byte
ranges that have been made available to us, before triggering the read.

There is a related design issue in passing that info down the client
streams path, but that part is IMO not hard.

Anyway, thats my current thoughts on the matter. I'm not going anywhere,
but while I wrap up current commitments I will be even more frenetic
than usual, which is roughly for the next 3 weeks.

Rob


-- 
GPG key available at: .


signature.asc
Description: This is a digitally signed message part

henrik: memory pool stuff

[Robert Collins]

Henrik,
How did you go with the use of arch to get the memory pool code?

Was it easy/hard? Do you have any feedback..

Rob

-- 
GPG key available at: .


signature.asc
Description: This is a digitally signed message part