Re: so what is involved in calling squid-3.0 'stable'?

[Doug Dixon]

On 23 Apr 2006, at 01:11, Robert Collins wrote:


On Sun, 2006-04-23 at 00:58 +1200, Reuben Farrelly wrote:


On 23/04/2006 12:41 a.m., Robert Collins wrote:

Asking adrian on irc - '
22:34 < adrian__> Enough people using it as a traditional forward  
cache

22:34 < adrian__> and saying there aren't any strange problems
22:34 < adrian__> Because its got a bad name
'

So, what is required. How can we engage the community in making  
squid-3
stable ? There seems to be non-trivial interest in making it  
happen, but

whats the actual benchmark ?


I'll start using it again and pushing forward with bug reports if  
there's
someone there to work on them...last time I tried squid-3 I was  
seeing some odd
stuff with client side connections being closed randomly and  
requiring frequent
refreshing with my end browser, but at the time I didn't gather  
anything useful.


There are definately people doing things around the source - I think
harnessing the energy is the issue. I only have a small amount of  
time,
and I'll probably be using it on toolchain support to make it  
easier for

others to fix bugs - because thats something effective I can do in the
timeframes I have available.

I've added some missing files - I could swear I had added them. Is  
that

better?

Rob

--
GPG key available at: .


As one such person who's joined the list recently with a view to  
pushing for 3.0-STABLE, I've got a few observations.


When someone new joins the squid-dev list, it's unclear what the  
drive of the group is. New members are invited to say what they're  
interested in (http://www.squid-cache.org/mailing-lists.html#squid- 
dev) but it's not easy to see (a) who else is there or (b) where  
we're going.


I think the website is too static and so gives the impression of a  
stale project... there needs to be a frequently updated and visible  
section (the homepage?) that tells everyone what has happened  
recently, and where we're headed next. E.g. Drive for 3.0-PRE4. This  
would rally people immediately.


Next thing to do is to use Bugzilla properly to drive and control  
that effort. Define a policy which says "If there are no Confirmed  
bugs of Severity 'normal' or worse for Target Milestone 3.0 (http:// 
tinyurl.com/fnkv9), then we've got a candidate for PRE4". We need to  
make sure that list contains ONLY the bugs that are stopping a PRE4.  
All others for 3.0 should either be reassigned to a future release,  
or have their severity adjusted downwards.


Getting a 3.0-STABLE out of the door will inevitably mean deferring  
some features to a subsequent target release (3.1?), and people  
having confidence that they won't be forgotten forever.


However, once there is a reliable task list for each release, people  
can volunteer in the knowledge they are helping towards a well  
defined goal, rather than one they're not sure will ever be finished  
- a fear that now seems well-founded given the gap since PRE3.


If this makes sense, I think the current hard-core who know the code  
best (and so are qualified to make these judgements) should bite the  
bullet and spend half a day making Bugzilla reflect the both the  
state and goals of the project. From then on, all bugs should start  
off Unconfirmed, and only core dudes should be able to confirm them.  
This will keep the buglist reliable and its meaning re: release  
readiness intact.


Of course what I'm suggesting requires consensus, and if it's agreed,  
some slightly boring admin work. But I think that's what we need to  
move forward with a release and also to attract people to the cause.



Cheers
Doug

Re: so what is involved in calling squid-3.0 'stable'?

[Nick Lewycky]

Reuben Farrelly wrote:
> 
> Changing the subject a little, there have been many new people introduce
> themselves on this list maybe with good intentions of working on squid,
> who seem to vanish as fast as they arrive.  I wonder if they've simply
> (a) never intended to contribute in the first place, (b) done some work
> privately but never released it or (c) taken a good look at the code,
> and run away fast deciding it was all too hard ;-)

Option c.

Let me tell you what happened in my research group. It started with one
undergraduate thesis at my University. The intent was to do what I've
done in my branch; add prefetching by interpreting HTML. The thesis
student is a competent programmer, but after four months of work
(reading code) and consulting other non-Squid programmers, he still
didn't know how to attack the problem. It'd gotten so bad that he was
just trying to read the log files for URLs fetched and then figure out
which entry in the cache directory corresponded to the newly fetched
object. Four months into an eight month thesis, the advising professor
started looking for another student to help and found me. I only managed
through judicious use of doxygen, profiling and debugging tools, trial
and error (my first version hooked all the code into http.cc) then
finally a help session with Robert Collins.

The Squid source code is miserable. I'm not saying it's the worst code
I've seen, because it isn't, but it's really really bad. Functions don't
always do what they say they'll do. Take urlParse for example. You'd
think it parses a URL, but it actually takes a URL and returns a new
HttpRequest. Sure HttpRequests contain URLs, but they also contain a
whole ton of other stuff too that goes well beyond the idea of breaking
up a URL. "urlCanonicalClean" is the function that converts an
HttpRequest back to a URL. That's not what the name sounds like.

There's some confusion about the architecture. A lot of mailing list
posts begin with "which part is the cache?" to which the answer is
invariably "the whole thing is a cache!" An HTTP parser is not a cache.
A client stream is not a cache. The config system is not a cache. The
"store" is. Now, some parts of Squid are clearly delineated. Some parts
are fuzzier. Can you tell me where the bloom filter is? And if you do
know which functions are responsible, why haven't you grouped them into
a bloom_filter.cc?

The good news is that it looks like newer code that gets checked in is
usually clearer and better commented. Perhaps it's because Squid3 is
undergoing various transitions (C->C++, new ClientStream system, etc)
which are unfinished and so it looks more complicated than it is. Either
that or I'm starting to understand the internals better.

I would recommend putting a doxygen output up on the website, then learn
the doxygen comment format and start using it. That'd be a start. Beyond
that, finish the transitions I guess. Try to make Squid a more
straight-forward pipeline.

Sorry if I sound too negative. I'm trying to be helpful with this email,
but I find it very difficult to point at one thing and say "there's your
problem". As for me, I want squid-prefetching off my hands, whether by
landing it into HEAD or just walking away from it.

Nick Lewycky

Re: so what is involved in calling squid-3.0 'stable'?

[Reuben Farrelly]

On 23/04/2006 1:11 a.m., Robert Collins wrote:

On Sun, 2006-04-23 at 00:58 +1200, Reuben Farrelly wrote:

On 23/04/2006 12:41 a.m., Robert Collins wrote:

Asking adrian on irc - '
22:34 < adrian__> Enough people using it as a traditional forward cache
22:34 < adrian__> and saying there aren't any strange problems
22:34 < adrian__> Because its got a bad name
'

So, what is required. How can we engage the community in making squid-3
stable ? There seems to be non-trivial interest in making it happen, but
whats the actual benchmark ?
I'll start using it again and pushing forward with bug reports if there's 
someone there to work on them...last time I tried squid-3 I was seeing some odd 
stuff with client side connections being closed randomly and requiring frequent 
refreshing with my end browser, but at the time I didn't gather anything useful.


There are definately people doing things around the source - I think
harnessing the energy is the issue. I only have a small amount of time,
and I'll probably be using it on toolchain support to make it easier for
others to fix bugs - because thats something effective I can do in the
timeframes I have available.


That's cool.

Changing the subject a little, there have been many new people introduce 
themselves on this list maybe with good intentions of working on squid, who seem 
to vanish as fast as they arrive.  I wonder if they've simply (a) never intended 
to contribute in the first place, (b) done some work privately but never 
released it or (c) taken a good look at the code, and run away fast deciding it 
was all too hard ;-)


From a development perspective, I think it'd be of value to know why are there 
not more people developing squid.  It seems to be just a "hardcore" few.




I've added some missing files - I could swear I had added them. Is that
better?


Yes, gets much further now, but runs into grief again here:

g++ -DHAVE_CONFIG_H -DDEFAULT_CONFIG_FILE=\"/etc/squid/squid.conf\" -I. -I. 
-I../include -I. -I. -I../include -I../include -I../lib/libTrie/include 
-I../lib/cppunit-1.10.0/include -I ../lib/cppunit-1.10.0/include 
-I/usr/kerberos/include  -Werror -Wall -Wpointer-arith -Wwrite-strings 
-Wcomments  -D_REENTRANT -g -O2 -c -o http.o http.cc

http.cc:50:23: error: HttpHdrSc.h: No such file or directory
http.cc:51:29: error: HttpHdrScTarget.h: No such file or directory
make[3]: *** [http.o] Error 1
make[3]: Leaving directory `/usr/src/squid/squid-3.0/src'
make[2]: *** [all-recursive] Error 1
make[2]: Leaving directory `/usr/src/squid/squid-3.0/src'
make[1]: *** [all] Error 2
make[1]: Leaving directory `/usr/src/squid/squid-3.0/src'
make: *** [all-recursive] Error 1
[EMAIL PROTECTED] squid-3.0]#

I guess a few more missing files.

Reuben

Re: so what is involved in calling squid-3.0 'stable'?

[Reuben Farrelly]

On 23/04/2006 1:44 a.m., Robert Collins wrote:

Meh, and this:

Forgetting to remove the enum cascades slightly.




--- src/acl.cc  23 Oct 2005 11:55:32 -  1.316
+++ src/acl.cc  22 Apr 2006 13:44:31 -
@@ -428,8 +428,7 @@
 link = link->next;
 }

-auth_match = NULL;
-auth_match = (acl_proxy_auth_match_cache
*)memAllocate(MEM_ACL_PROXY_AUTH_MATCH);
+auth_match = new acl_proxy_auth_match_cache();
 auth_match->matchrv = matchForCache (checklist);
 auth_match->acl_data = this;
 dlinkAddTail(auth_match, &auth_match->link, cache);
@@ -451,7 +450,7 @@
 tmplink = link;
 link = link->next;
 dlinkDelete(tmplink, cache);
-memFree(auth_match, MEM_ACL_PROXY_AUTH_MATCH);
+   delete auth_match;
 }
 }


Ok it's a bit better now, but still crashing out:

[EMAIL PROTECTED] squid-3.0]# gdb /usr/sbin/squid-3.0
GNU gdb Red Hat Linux (6.3.0.0-1.128.FC6rh)
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu"...Using host libthread_db 
library "/lib64/libthread_db.so.1".


(gdb) run -NCd1 -f /etc/squid/squid.conf-3.0
Starting program: /usr/sbin/squid-3.0 -NCd1 -f /etc/squid/squid.conf-3.0
[Thread debugging using libthread_db enabled]
[New Thread 47650264002256 (LWP 486)]
2006/04/23 01:57:17| Notice: Short Strings is 36 bytes instead of requested 36 
bytes
2006/04/23 01:57:17| diskd started

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 47650264002256 (LWP 486)]
0x0047ec1b in memAllocate (type=Variable "type" is not available.
) at mem.cc:160
160 return MemPools[type]->alloc();
(gdb) bt full
#0  0x0047ec1b in memAllocate (type=Variable "type" is not available.
) at mem.cc:160
No locals.
#1  0x0041e7f9 in wordlistAdd (list=0x701600, key=0x745ae3 "cgi-bin") at 
cache_cf.cc:175

No locals.
#2  0x0041e85c in parse_wordlist (list=0x701600) at cache_cf.cc:2516
token = Variable "token" is not available.
(gdb)

Re: so what is involved in calling squid-3.0 'stable'?

[Reuben Farrelly]

On 23/04/2006 1:26 a.m., Robert Collins wrote:

On Sun, 2006-04-23 at 01:20 +1200, Reuben Farrelly wrote:

On 23/04/2006 1:11 a.m., Robert Collins wrote:

On Sun, 2006-04-23 at 00:58 +1200, Reuben Farrelly wrote:

On 23/04/2006 12:41 a.m., Robert Collins wrote:

Asking adrian on irc - '
22:34 < adrian__> Enough people using it as a traditional forward cache
22:34 < adrian__> and saying there aren't any strange problems
22:34 < adrian__> Because its got a bad name
'

So, what is required. How can we engage the community in making squid-3
stable ? There seems to be non-trivial interest in making it happen, but
whats the actual benchmark ?
I'll start using it again and pushing forward with bug reports if there's 
someone there to work on them...last time I tried squid-3 I was seeing some odd 
stuff with client side connections being closed randomly and requiring frequent 
refreshing with my end browser, but at the time I didn't gather anything useful.

There are definately people doing things around the source - I think
harnessing the energy is the issue. I only have a small amount of time,
and I'll probably be using it on toolchain support to make it easier for
others to fix bugs - because thats something effective I can do in the
timeframes I have available.

That's cool.

Changing the subject a little, there have been many new people introduce 
themselves on this list maybe with good intentions of working on squid, who seem 
to vanish as fast as they arrive.  I wonder if they've simply (a) never intended 
to contribute in the first place, (b) done some work privately but never 
released it or (c) taken a good look at the code, and run away fast deciding it 
was all too hard ;-)





 From a development perspective, I think it'd be of value to know why are there 
not more people developing squid.  It seems to be just a "hardcore" few.


Yep. Agree with this.


I've added some missing files - I could swear I had added them. Is that
better?

Yes, gets much further now, but runs into grief again here:

..

I guess a few more missing files.



Added, hopefully got them all now.


Yes.  Compiles, but fails to run...here it is under gdb.

I'm on x86_64.

[EMAIL PROTECTED] squid]# gdb /usr/sbin/squid-3.0
GNU gdb Red Hat Linux (6.3.0.0-1.128.FC6rh)
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu"...Using host libthread_db 
library "/lib64/libthread_db.so.1".


(gdb) run -NCd1
Starting program: /usr/sbin/squid-3.0 -NCd1
[Thread debugging using libthread_db enabled]
[New Thread 47347710906064 (LWP 28014)]
2006/04/23 01:33:58| Notice: Short Strings is 36 bytes instead of requested 36 
bytes
2006/04/23 01:33:58| diskd started
2006/04/23 01:33:58| WARNING: use of 'reload-into-ims' in 'refresh_pattern' 
violates HTTP

2006/04/23 01:33:58| Initializing https proxy context
2006/04/23 01:33:58| Starting Squid Cache version 3.0-PRE3-CVS for 
x86_64-unknown-linux-gnu...

2006/04/23 01:33:58| Process ID 28014
2006/04/23 01:33:58| With 256 file descriptors available
2006/04/23 01:33:58| Performing DNS Tests...
2006/04/23 01:33:58| Successful DNS name lookup tests...
2006/04/23 01:33:58| DNS Socket created at 0.0.0.0, port 35919, FD 8
2006/04/23 01:33:58| Adding domain reub.net from /etc/resolv.conf
2006/04/23 01:33:58| Adding nameserver 192.168.0.5 from /etc/resolv.conf
2006/04/23 01:33:58| Adding nameserver 202.89.128.17 from /etc/resolv.conf
2006/04/23 01:33:58| helperOpenServers: Starting 5 'wrapzap.sh' processes
Detaching after fork from child process 28017.
Detaching after fork from child process 28018.
Detaching after fork from child process 28019.
Detaching after fork from child process 28020.
Detaching after fork from child process 28021.
2006/04/23 01:33:58| User-Agent logging is disabled.
2006/04/23 01:33:58| Referer logging is disabled.
Detaching after fork from child process 28022.
2006/04/23 01:33:58| Unlinkd pipe opened on FD 18
2006/04/23 01:33:58| Store logging disabled
2006/04/23 01:33:58| Swap maxSize 1740800 KB, estimated 158254 objects

2006/04/23 01:33:58| Target number of buckets: 7912
2006/04/23 01:33:58| Using 8192 Store buckets
2006/04/23 01:33:58| Max Mem  size: 8192 KB
2006/04/23 01:33:58| Max Swap size: 1740800 KB
2006/04/23 01:33:58| Rebuilding storage in /var/spool/squid-1 (DIRTY)
2006/04/23 01:33:58| Rebuilding storage in /var/spool/squid-2 (DIRTY)
2006/04/23 01:33:58| Using Least Load store dir selection
2006/04/23 01:33:58| Set Current Directory to /var/spool/squid
2006/04/23 01:33:58| Loaded Icons.
2006/04/23 01:33:58| Accepting  HTTP connections at 192.168.0.5, port 3128, FD 
21.
2006/04/23 01:33:58| commBind: Cannot bind socket FD 22 to 192.168.0.5:8080: 
(98) Add

Re: so what is involved in calling squid-3.0 'stable'?

[Robert Collins]

Meh, and this:

Forgetting to remove the enum cascades slightly.




--- src/acl.cc  23 Oct 2005 11:55:32 -  1.316
+++ src/acl.cc  22 Apr 2006 13:44:31 -
@@ -428,8 +428,7 @@
 link = link->next;
 }

-auth_match = NULL;
-auth_match = (acl_proxy_auth_match_cache
*)memAllocate(MEM_ACL_PROXY_AUTH_MATCH);
+auth_match = new acl_proxy_auth_match_cache();
 auth_match->matchrv = matchForCache (checklist);
 auth_match->acl_data = this;
 dlinkAddTail(auth_match, &auth_match->link, cache);
@@ -451,7 +450,7 @@
 tmplink = link;
 link = link->next;
 dlinkDelete(tmplink, cache);
-memFree(auth_match, MEM_ACL_PROXY_AUTH_MATCH);
+   delete auth_match;
 }
 }


Cheers,
Rob


signature.asc
Description: This is a digitally signed message part

Re: so what is involved in calling squid-3.0 'stable'?

[Robert Collins]

On Sun, 2006-04-23 at 01:37 +1200, Reuben Farrelly wrote:



> 2006/04/23 01:33:58| assertion failed: mem.cc:433: "MemPools[t]"
> 


> Config is at http://www.reub.net/files/squid/squid.conf


Thanks, try this:

:!cvs diff -u src/enums.h
Index: src/enums.h
===
RCS file: /cvsroot/squid/squid3/src/enums.h,v
retrieving revision 1.247
diff -u -r1.247 enums.h
--- src/enums.h 22 Apr 2006 05:29:19 -  1.247
+++ src/enums.h 22 Apr 2006 13:41:52 -
@@ -455,7 +455,6 @@
 MEM_64K_BUF,
 MEM_ACL_DENY_INFO_LIST,
 MEM_ACL_NAME_LIST,
-MEM_ACL_PROXY_AUTH_MATCH,
 #if USE_CACHE_DIGESTS
 MEM_CACHE_DIGEST,
 #endif


Cheers,
Rob
-- 
GPG key available at: .


signature.asc
Description: This is a digitally signed message part

Re: so what is involved in calling squid-3.0 'stable'?

[Robert Collins]

On Sun, 2006-04-23 at 01:20 +1200, Reuben Farrelly wrote:
> 
> On 23/04/2006 1:11 a.m., Robert Collins wrote:
> > On Sun, 2006-04-23 at 00:58 +1200, Reuben Farrelly wrote:
> >> On 23/04/2006 12:41 a.m., Robert Collins wrote:
> >>> Asking adrian on irc - '
> >>> 22:34 < adrian__> Enough people using it as a traditional forward cache
> >>> 22:34 < adrian__> and saying there aren't any strange problems
> >>> 22:34 < adrian__> Because its got a bad name
> >>> '
> >>>
> >>> So, what is required. How can we engage the community in making squid-3
> >>> stable ? There seems to be non-trivial interest in making it happen, but
> >>> whats the actual benchmark ?
> >> I'll start using it again and pushing forward with bug reports if there's 
> >> someone there to work on them...last time I tried squid-3 I was seeing 
> >> some odd 
> >> stuff with client side connections being closed randomly and requiring 
> >> frequent 
> >> refreshing with my end browser, but at the time I didn't gather anything 
> >> useful.
> > 
> > There are definately people doing things around the source - I think
> > harnessing the energy is the issue. I only have a small amount of time,
> > and I'll probably be using it on toolchain support to make it easier for
> > others to fix bugs - because thats something effective I can do in the
> > timeframes I have available.
> 
> That's cool.
> 
> Changing the subject a little, there have been many new people introduce 
> themselves on this list maybe with good intentions of working on squid, who 
> seem 
> to vanish as fast as they arrive.  I wonder if they've simply (a) never 
> intended 
> to contribute in the first place, (b) done some work privately but never 
> released it or (c) taken a good look at the code, and run away fast deciding 
> it 
> was all too hard ;-)

>
>
>
>  From a development perspective, I think it'd be of value to know why are 
> there 
> not more people developing squid.  It seems to be just a "hardcore" 
> few.

Yep. Agree with this.

> > I've added some missing files - I could swear I had added them. Is that
> > better?
> 
> Yes, gets much further now, but runs into grief again here:
..
> 
> I guess a few more missing files.


Added, hopefully got them all now.

Rob
-- 
GPG key available at: .


signature.asc
Description: This is a digitally signed message part

Re: so what is involved in calling squid-3.0 'stable'?

[Robert Collins]

On Sun, 2006-04-23 at 00:58 +1200, Reuben Farrelly wrote:
> 
> On 23/04/2006 12:41 a.m., Robert Collins wrote:
> > Asking adrian on irc - '
> > 22:34 < adrian__> Enough people using it as a traditional forward cache
> > 22:34 < adrian__> and saying there aren't any strange problems
> > 22:34 < adrian__> Because its got a bad name
> > '
> > 
> > So, what is required. How can we engage the community in making squid-3
> > stable ? There seems to be non-trivial interest in making it happen, but
> > whats the actual benchmark ?
> 
> I'll start using it again and pushing forward with bug reports if there's 
> someone there to work on them...last time I tried squid-3 I was seeing some 
> odd 
> stuff with client side connections being closed randomly and requiring 
> frequent 
> refreshing with my end browser, but at the time I didn't gather anything 
> useful.

There are definately people doing things around the source - I think
harnessing the energy is the issue. I only have a small amount of time,
and I'll probably be using it on toolchain support to make it easier for
others to fix bugs - because thats something effective I can do in the
timeframes I have available.

I've added some missing files - I could swear I had added them. Is that
better?

Rob

-- 
GPG key available at: .


signature.asc
Description: This is a digitally signed message part

Re: squid-3 - cleanup of select_* stuff.

[Adrian Chadd]

On Sat, Apr 22, 2006, Robert Collins wrote:
> Meta comment on this after chatting with adrian - I dont think this will
> itself make the IO stuff beautiful. But it should allow writing some
> test helper routines so that anything I find time to fix has a test for
> it that ensures it stays fixed.

You're all welcome, btw.

#squiddev on Freenode.



adrian

Re: squid-3 - cleanup of select_* stuff.

[Robert Collins]

Meta comment on this after chatting with adrian - I dont think this will
itself make the IO stuff beautiful. But it should allow writing some
test helper routines so that anything I find time to fix has a test for
it that ensures it stays fixed.

-Rob


On Sat, 2006-04-22 at 22:23 +1000, Robert Collins wrote:
> I'd like to (slowly, I'm expecting about 4 hours a week on this) cleanup
> the select loop in squid to allow things like using
> poller/libevent/completion ports. I've been using twisted a bit over the
> last couple of years, and their event loop - their reactor - has a lot
> in parallel with the ACE reactor/proactor pair of patterns.
> http://www.cs.wustl.edu/~schmidt/PDF/reactor-rules.pdf
> http://www.cs.wustl.edu/~schmidt/PDF/proactor.pdf
> 
> 
> Our current structure is partway between a reactor and proactor - we
> have multiple reactor style objects - the async io thread stuff for disk
> io, the single instance select loop.
> 
> Using the language of the proactor pattern..
>  Right now we have one thread for each reactor-like thing - we have one
> thread for the async disk engine when its in use, and one thread (the
> main thread) for the comms queue. This is a little ugly because its
> asymmetrical: theres nothing intrinsically special about sockets to make
> them be the reactor in the main thread. I think its ugly because it
> presupposes that our efficiency on sockets will be better than that on
> disk. So I'd like to propose that we tweak our code so that we no longer
> assume comms requests happen in the main thread.
> 
> One way of doing this is to have a dispatcher instance for each type of
> event we can dispatch, and loop on them in the main loop. (this is a
> trivial tweak to what we have today). If we also have objects to
> represent each async-activity that occurs (i.e. a select loop, a poll l
> loop, a completion-port loop), I'd like to give all async-operating code
> paths an object to represent them. 
> 
> Now, as we would like to not busy wait, we need to pass in a non-zero
> timeout to any select/poll style calls, but this will cause latency if
> other async activity does occur concurrently. So - each async engine
> will have a method on it which can be called to 'cheaply' notify it of
> activity that is occuring (i.e. the threaded-async disk engine can
> inform the current primary engine that a disk io has completed).
> 
> And because we don't know a-priori whether an async engine is os-backed
> (i.e. completion ports with overlapped I/O) or polled, there needs to be
> a poll() or checkEvents() or similar method called on each engine once
> per loops.
> 
> our main loop can then become something like:
> 
> 
> while (!finished) {
>   for (dispatchers::iterator i = dispatchers.first();
>i != dispatchers.end(); ++i) {
> i->dispatch();
>   }
>   for (engines::iterator i = engines.first();
>i != engines.end(); ++i) {
> i->checkEvents();
>   }
> }
> 
> This will have the following benefits:
>  - We can properly support Overlapped IO on windows
>  - We can change the engines in use at runtime - just keep
>an engine in the engines list until all the pending events on it have
>completed and then remove it.
>  - management of the main loop and reconfiguration becomes conceptually 
>clearer - we move the special casing out of the main loop and into 
>specific event handlers. (processing signals like 'please 
>reconfigure' becomes just another event that can be dispatched).
> 
> 
>  I think we want a single completion dispatcher class for each group of
> events that occur asynchronously. I.e. a dispatcher that knows how to
> dispatch socket events, one that knows how to dispatch disk events, one
> that knows how to dispatch timer events etc, one that knows how to 
> dispatch informational signals.
> 
> 
> How does this sound in principal? If it sounds ok, I'll start doing a
> series of small (a few hours each) patches heading in this direction.
> One of the reasons I want to do this is to make it possible to write a
> test harness that can exercise callback requiring code by having a
> trivial controllable event loop that can be invoked in a test.
> 
> Rob
> 
> 
-- 
GPG key available at: .


signature.asc
Description: This is a digitally signed message part

so what is involved in calling squid-3.0 'stable'?

[Robert Collins]

Asking adrian on irc - '
22:34 < adrian__> Enough people using it as a traditional forward cache
22:34 < adrian__> and saying there aren't any strange problems
22:34 < adrian__> Because its got a bad name
'

So, what is required. How can we engage the community in making squid-3
stable ? There seems to be non-trivial interest in making it happen, but
whats the actual benchmark ?

Rob

-- 
GPG key available at: .


signature.asc
Description: This is a digitally signed message part

squid-3 - cleanup of select_* stuff.

[Robert Collins]

I'd like to (slowly, I'm expecting about 4 hours a week on this) cleanup
the select loop in squid to allow things like using
poller/libevent/completion ports. I've been using twisted a bit over the
last couple of years, and their event loop - their reactor - has a lot
in parallel with the ACE reactor/proactor pair of patterns.
http://www.cs.wustl.edu/~schmidt/PDF/reactor-rules.pdf
http://www.cs.wustl.edu/~schmidt/PDF/proactor.pdf


Our current structure is partway between a reactor and proactor - we
have multiple reactor style objects - the async io thread stuff for disk
io, the single instance select loop.

Using the language of the proactor pattern..
 Right now we have one thread for each reactor-like thing - we have one
thread for the async disk engine when its in use, and one thread (the
main thread) for the comms queue. This is a little ugly because its
asymmetrical: theres nothing intrinsically special about sockets to make
them be the reactor in the main thread. I think its ugly because it
presupposes that our efficiency on sockets will be better than that on
disk. So I'd like to propose that we tweak our code so that we no longer
assume comms requests happen in the main thread.

One way of doing this is to have a dispatcher instance for each type of
event we can dispatch, and loop on them in the main loop. (this is a
trivial tweak to what we have today). If we also have objects to
represent each async-activity that occurs (i.e. a select loop, a poll l
loop, a completion-port loop), I'd like to give all async-operating code
paths an object to represent them. 

Now, as we would like to not busy wait, we need to pass in a non-zero
timeout to any select/poll style calls, but this will cause latency if
other async activity does occur concurrently. So - each async engine
will have a method on it which can be called to 'cheaply' notify it of
activity that is occuring (i.e. the threaded-async disk engine can
inform the current primary engine that a disk io has completed).

And because we don't know a-priori whether an async engine is os-backed
(i.e. completion ports with overlapped I/O) or polled, there needs to be
a poll() or checkEvents() or similar method called on each engine once
per loops.

our main loop can then become something like:


while (!finished) {
  for (dispatchers::iterator i = dispatchers.first();
   i != dispatchers.end(); ++i) {
i->dispatch();
  }
  for (engines::iterator i = engines.first();
   i != engines.end(); ++i) {
i->checkEvents();
  }
}

This will have the following benefits:
 - We can properly support Overlapped IO on windows
 - We can change the engines in use at runtime - just keep
   an engine in the engines list until all the pending events on it have
   completed and then remove it.
 - management of the main loop and reconfiguration becomes conceptually 
   clearer - we move the special casing out of the main loop and into 
   specific event handlers. (processing signals like 'please 
   reconfigure' becomes just another event that can be dispatched).


 I think we want a single completion dispatcher class for each group of
events that occur asynchronously. I.e. a dispatcher that knows how to
dispatch socket events, one that knows how to dispatch disk events, one
that knows how to dispatch timer events etc, one that knows how to 
dispatch informational signals.


How does this sound in principal? If it sounds ok, I'll start doing a
series of small (a few hours each) patches heading in this direction.
One of the reasons I want to do this is to make it possible to write a
test harness that can exercise callback requiring code by having a
trivial controllable event loop that can be invoked in a test.

Rob


-- 
GPG key available at: .


signature.asc
Description: This is a digitally signed message part

anyone played with http://kegel.com/dkftpbench/doc/Poller.html?

[Robert Collins]

I'm not planning to put it in squid or anything, I'm more wondering what
you thought/think of the API.

Rob
-- 
GPG key available at: .


signature.asc
Description: This is a digitally signed message part

Re: wikis...

[Henrik Nordstrom]

lör 2006-04-22 klockan 09:45 +0200 skrev Kinkie:

> There has a been a chat over IRC between Duane and myself, where we
> agreed that squidwiki.kinkie.it would be renamed to wiki.squid-cache.org
> once it had absorbed all content in the current wiki.

Excellent.

Regards
Henrik


signature.asc
Description: Detta är en digitalt signerad	meddelandedel

Re: wikis...

[Kinkie]

On Sat, 2006-04-22 at 00:25 +0200, Henrik Nordstrom wrote:
> fre 2006-04-21 klockan 22:05 +1000 skrev Robert Collins:
> > I heard that the wiki kinkie had up is ready to go with all content
> > migrated, is it ok if we update the link on squid-cache.org to point to
> > it now ?
> 
> Has there been an agreement on which wiki is "the" Squid-Cache wiki?

There has a been a chat over IRC between Duane and myself, where we
agreed that squidwiki.kinkie.it would be renamed to wiki.squid-cache.org
once it had absorbed all content in the current wiki.

I still haven't imported the annotated list of configuration directives,
but the FAQ has been copied over and I've improved it in the process.

As far as administration goes, squidwiki.kinkie.it is backed-up daily
on-site, more-or-less monthly off-site, and is monitored daily for
changes.

Recent versions of MoinMoin (included the one I'm running) also offer a
WYSIWYG Javascript-based editor for those who don't want to meddle with
wiki-syntax.

Kinkie