Re: Squid-3 release cycle
Hi Henrik, At 17.18 05/05/2006, Henrik Nordstrom wrote: For those who have not been on the #squid-dev IRC channel lately I can tell that the last weeks has been quite interesting. The most significant news is that Doug Dixon (aka ganso on the IRC) has volunteered for the role as Squid-3.0.PRE4 release manager. Expect a message from him shortly presenting his ideas on how we can get there. This is a very good news :-) To follow up on a few questions from him regarding the current state: The Squid-3 tree is currently best described as in DEVEL state, even if it is carrying a PRE tag. The reason to this is that the original Squid-3.0 release cycle could not be met due to various events and the tree had to be unlocked again to allow for new developments. The goal now should be to be able to enter PRE state again, with ultimately a PRE4 release from where we can work towards the STABLE release. I do not think there is any major changes waiting in queue for getting into Squid-3, and imho minor/isolated features like WCCPv2 or a improved COSS may well get into the tree during a PRE cycle. But there is a quite long list of bugs, both verified and to be analyzed ones. Some critical, many not so critical ones.. Developers having new features in queue which they would like to get into Squid-3.0 please speak up now, allowing for Doug to do his job proper. As for all of us his time is somewhat limited and the timeframe currently considered for a PRE4 release is not very distant. In my queue there some minor changes related to MinGW Windows support: - configure.in Makefile.am - some additional source library for missing functions on Windows (getopt, rusage, strtok_r) The majority of this work is already present in the nt branch, but I like to refactor something in a better way before the addition to HEAD. I like too that the changes based on the old Roberts's IPC refactoring work could go in Squid 3.0: http://www.squid-cache.org/~robertc/ipc.refactoring.patch http://www.squid-cache.org/~robertc/ipc.h Currently these changes are included in the nt branch (they are mandatory for Windows support) and I have opened the ipc branch with all the changes: http://devel.squid-cache.org/projects.html#ipc But I have a big problem: my C++ knowledge is still not enough for this job ... :-( Some help on this could be very useful. Regards Guido - Guido Serassio Acme Consulting S.r.l. - Microsoft Certified Partner Via Lucia Savarino, 1 10098 - Rivoli (TO) - ITALY Tel. : +39.011.9530135 Fax. : +39.011.9781115 Email: [EMAIL PROTECTED] WWW: http://www.acmeconsulting.it/
Re: Proposal: Drop the Squid-3.0 patches page
Hi Henrik, At 02.53 06/05/2006, Henrik Nordstrom wrote: Proposal: Drop the Squid-3 patches page. http://www.squid-cache.org/Version/v3/3.0/bugs/ Fine for me. Regards Guido It's not actively maintained and does not give much benefit to the release process until we are ready to go to STABLE state I think. If needed some of it's functionality can be resurrected from the commit logs. Regards Henrik - Guido Serassio Acme Consulting S.r.l. - Microsoft Certified Partner Via Lucia Savarino, 1 10098 - Rivoli (TO) - ITALY Tel. : +39.011.9530135 Fax. : +39.011.9781115 Email: [EMAIL PROTECTED] WWW: http://www.acmeconsulting.it/
reply body modification
Hi all, I'm running an experimental version of squid-2.5.STABLE12 with patches that allow for modifiying the client reply body. Patches have been applied mostly to client_side.c and especially clientSendMoreData(). Since I need a chunk of body data which meets certain size criteria, I check the size parameter on entry to clientSendMoreData() and return without sending anything to the client if not enough data is available. For practical reasons, I chose to wait until CLIENT_SOCK_SZ (default 4k) bytes are passed via buf. Returning from clientSendMoreData() is accomplished by calling clientWriteComplete(fd, NULL, 0, 0, http); memFree(buf, MEM_CLIENT_SOCK_BUF); httpReplyDestroy(rep); return; This has been tested and works ok up to CLIENT_SOCK_SZ=16384. A further increase of CLIENT_SOCK_SZ to 32k leads to a strange phenomenon, in that reading the reply of the origin server often seems to starve out at sizes around 17k. Although the origin server keeps on sending data and seems to fill the receiving socket buffer (netstat claims that sufficient data is available for reading), the associated file descriptor never comes back via comm_poll() signalling that new data has arrived. AFAIK this strange behaviour only occurs, when data is read from a distant server and httpReadReply() is called several times with small chunks of data (e.g. 12x1448bytes). Serving the same data from a local httpd, seems to work flawlessly since the receiving buffer is mostly filled by only one call to httpReadReply(). I'd very much appreciate, if somebody could give me any pointers where to look further and probably find a solution to this problem. TIA + best regards ... Michael Ulbrich
Re: [VOTE] Squid-3.0.PRE4 release criteria
Hi Henrik Doug, At 02.21 06/05/2006, Henrik Nordstrom wrote: lör 2006-05-06 klockan 12:08 +1200 skrev Doug Dixon: Option 1 - severity based We release PRE4 when all the criticals and blockers have been fixed. Over optimistic. This is the quality level required for a RC/STABLE release. Option 2 - hand-picked We hand pick a fixed list of bugs which we think we can fix within a reasonably short time. We release when they are all fixed. No other bugs can enter the list once it is picked. Option 3 - time based We release PRE4 on June 1, and fix whatever bugs we can until then (blockers first). Combination of 2 3 is my vote. Start with a hand picked list and a target date. Adjust the list over time, removing items found to not be realistic for the target date and add items which is found sufficiently important, and if necessary to preserve reasonable faith in the release adjust the time (limited +1 week or so, no limit on early release when goals met). This is a scaled down version of the process currently used for the STABLE releases. Regards Henrik I fully agree with the Henrik's vote. Regards Guido - Guido Serassio Acme Consulting S.r.l. - Microsoft Certified Partner Via Lucia Savarino, 1 10098 - Rivoli (TO) - ITALY Tel. : +39.011.9530135 Fax. : +39.011.9781115 Email: [EMAIL PROTECTED] WWW: http://www.acmeconsulting.it/
Re: reply body modification
Could you share the patch? adrian On Sat, May 06, 2006, Michael Ulbrich wrote: Hi all, I'm running an experimental version of squid-2.5.STABLE12 with patches that allow for modifiying the client reply body. Patches have been applied mostly to client_side.c and especially clientSendMoreData(). Since I need a chunk of body data which meets certain size criteria, I check the size parameter on entry to clientSendMoreData() and return without sending anything to the client if not enough data is available. For practical reasons, I chose to wait until CLIENT_SOCK_SZ (default 4k) bytes are passed via buf. Returning from clientSendMoreData() is accomplished by calling clientWriteComplete(fd, NULL, 0, 0, http); memFree(buf, MEM_CLIENT_SOCK_BUF); httpReplyDestroy(rep); return; This has been tested and works ok up to CLIENT_SOCK_SZ=16384. A further increase of CLIENT_SOCK_SZ to 32k leads to a strange phenomenon, in that reading the reply of the origin server often seems to starve out at sizes around 17k. Although the origin server keeps on sending data and seems to fill the receiving socket buffer (netstat claims that sufficient data is available for reading), the associated file descriptor never comes back via comm_poll() signalling that new data has arrived. AFAIK this strange behaviour only occurs, when data is read from a distant server and httpReadReply() is called several times with small chunks of data (e.g. 12x1448bytes). Serving the same data from a local httpd, seems to work flawlessly since the receiving buffer is mostly filled by only one call to httpReadReply(). I'd very much appreciate, if somebody could give me any pointers where to look further and probably find a solution to this problem. TIA + best regards ... Michael Ulbrich
Re: [VOTE] Squid-3.0.PRE4 release criteria
I agree too. I think part of the reason behind squid-3 falling 'behind' has been all the cute stuff being done to it rather than concentrating on getting something stable. Get something stable, ship it out the door, then concentrate on more code base tidying. Adrian On Sat, May 06, 2006, Guido Serassio wrote: Hi Henrik Doug, At 02.21 06/05/2006, Henrik Nordstrom wrote: l??r 2006-05-06 klockan 12:08 +1200 skrev Doug Dixon: Option 1 - severity based We release PRE4 when all the criticals and blockers have been fixed. Over optimistic. This is the quality level required for a RC/STABLE release. Option 2 - hand-picked We hand pick a fixed list of bugs which we think we can fix within a reasonably short time. We release when they are all fixed. No other bugs can enter the list once it is picked. Option 3 - time based We release PRE4 on June 1, and fix whatever bugs we can until then (blockers first). Combination of 2 3 is my vote. Start with a hand picked list and a target date. Adjust the list over time, removing items found to not be realistic for the target date and add items which is found sufficiently important, and if necessary to preserve reasonable faith in the release adjust the time (limited +1 week or so, no limit on early release when goals met). This is a scaled down version of the process currently used for the STABLE releases. Regards Henrik I fully agree with the Henrik's vote. Regards Guido - Guido Serassio Acme Consulting S.r.l. - Microsoft Certified Partner Via Lucia Savarino, 1 10098 - Rivoli (TO) - ITALY Tel. : +39.011.9530135 Fax. : +39.011.9781115 Email: [EMAIL PROTECTED] WWW: http://www.acmeconsulting.it/
Re: [VOTE] Squid-3.0.PRE4 release criteria
Hi, On 6/05/2006 10:05 p.m., Guido Serassio wrote: Combination of 2 3 is my vote. Start with a hand picked list and a target date. Adjust the list over time, removing items found to not be realistic for the target date and add items which is found sufficiently important, and if necessary to preserve reasonable faith in the release adjust the time (limited +1 week or so, no limit on early release when goals met). This is a scaled down version of the process currently used for the STABLE releases. Regards Henrik I fully agree with the Henrik's vote. Regards Guido Me too. Having a reasonably definite date known in advance with only a few days room to shift it will hopefully encourage more of a sense of working towards a target. Some of the bugs in bugzilla would likely hold the release up for months if we were to try and fix them all before the next PRE. Additionally, why I'm less inclined towards option 1 is that at this early stage I think it's well worth just having list of *known* bugs with good bug reports, even if we haven't actually fixed them. Given the very limited testing of the code now, there are likely to be many currently unknown bugs show up anyway as soon as people start banging on the code again, so aiming for no criticals or high priority bugs after so many changes is likely to be somewhat futile ;-) In terms of dates, we'll also need some resource to be able to respond to the new bugs that people find immediately after the release, whatever date that turns out to be. Btw, thanks for taking on this task, Doug. reuben
Current -cvs crashing when invalid hostname in client requested URL
Seems to be new breakage in the last few days, but if I try to surf to a URL
which is invalid eg www.firfox.org, squid-3/CVS dies an ugly death:
Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 47741279648528 (LWP 29234)]
0x0043543c in clientReplyContext::processMiss (this=0x2b6ba27dd028)
at ../include/RefCount.h:110
110 p.p_-RefCountReference();
(gdb) bt full
#0 0x0043543c in clientReplyContext::processMiss (this=0x2b6ba27dd028)
at ../include/RefCount.h:110
url = Variable url is not available.
(gdb) frame 0
#0 0x0043543c in clientReplyContext::processMiss (this=0x2b6ba27dd028)
at ../include/RefCount.h:110
110 p.p_-RefCountReference();
(gdb) frame 1
#1 0x004358e8 in clientReplyContext::doGetMoreData (
this=0x2b6ba27dd028) at client_side_reply.cc:1718
1718processMiss();
(gdb) frame 2
#2 0x004389b6 in ClientHttpRequest::httpStart (this=0xbddeb8)
at client_side_request.cc:953
953 clientStreamRead(node, this, node-readBuffer);
(gdb) frame 3
#3 0x0043970f in ClientHttpRequest::doCallouts (this=0xbddeb8)
at client_side_request.cc:1097
1097processRequest();
(gdb) frame 4
#4 0x0040c305 in ACLChecklist::checkCallback (this=0xbefa98,
answer=ACCESS_ALLOWED) at ACLChecklist.cc:248
248 callback_(answer, cbdata_);
(gdb) frame 5
#5 0x0047ec90 in ipcacheCallback (i=0x83c620) at ipcache.cc:265
265 callback(i-flags.negcached ? NULL : i-addrs, cbdata);
(gdb) frame 6
#6 0x004520ae in idnsCallback (q=0x2b6ba2800010, answers=0x0, n=-3,
error=0x51e158 Name Error: The domain name does not exist.)
at dns_internal.cc:877
877 callback(cbdata, answers, n, error);
(gdb) frame 7
#7 0x0045384a in idnsGrokReply (buf=Variable buf is not available.
) at dns_internal.cc:984
984 idnsCallback(q, message-answer, n, q-error);
(gdb) frame 8
#8 0x00453d9a in idnsRead (fd=8, data=Variable data is not available.
) at dns_internal.cc:1052
1052idnsGrokReply(rbuf, len);
(gdb) frame 9
#9 0x00444ad6 in comm_select (msec=Variable msec is not available.
) at comm_epoll.cc:269
269 hdl(fd, F-read_data);
(gdb) frame 10
#10 0x004813ba in main (argc=Variable argc is not available.
) at main.cc:1177
1177switch (comm_select(loop_delay)) {
(gdb) info locals
wait = 4294967295
loop_delay = Variable loop_delay is not available.
(gdb)
100% reproduceable.
Reuben
Re: reply body modification
Adrian Chadd wrote:
Could you share the patch?
adrian
Hi Adrian,
thanks for your quick reply!
Yes, of course, you will find the patch attached to this message. I
deleted the lines which actually modify the JPEG body.
Basically data from buf is copied to mod_buf and modified there. Sizes
and the Content-Length header field are updated in accordance to the
amount of inserted data. Towards the end buf and size take over the
modified values and normal clientSendMoreData() processing continues.
If modifications have been done, care is taken that http-out.offset is
incremented only by old_body_size. Otherwise the followup data on the
next call to clientSendMoreData() will be out of sync.
I had to modify one line in clientWriteComplete(), otherwise the call to
clientWriteComplete(fd, NULL, 0, 0, http); with size = 0 was interpreted
as transfer DONE.
With CLIENT_SOCK_SZ set to 16k this code works ok. Problems arise when
increasing this value to 32k, as I wrote in my earlier message.
Please apologize the somewhat crude hack, but it was meant as a first
test, whether something like that could be done at all.
Thanks again and best regards ... Michael
--- src/client_side.c.orig 2006-05-03 16:19:57.491424980 +0200
+++ src/client_side.c 2006-05-03 16:48:03.920598576 +0200
@@ -34,6 +34,7 @@
*/
#include squid.h
+#include libiptc/iptc.h
#if IPF_TRANSPARENT
#if HAVE_SYS_IOCTL_H
@@ -101,7 +102,10 @@
#endif
static const char *const crlf = \r\n;
+extern int errno;
+#define JPEG_MOD 1
+#define BUFSIZE 16384
#define FAILURE_MODE_TIME 300
/* Local functions */
@@ -1968,6 +1972,32 @@
squid_off_t body_size = size;
MemBuf mb;
squid_off_t check_size = 0;
+
+int i = 0, j = 0, ret = 0, done = 0;
+int APP13_len = 0, BIM_num = 0;
+FILE * bin_in_fd = NULL;
+FILE * txt_in_fd = NULL;
+FILE * txt_out_fd = NULL;
+FILE * bin_out_fd = NULL;
+FILE * jpg_out_fd = NULL;
+unsigned int marker;
+unsigned char iptc_line[BUFSIZE];
+unsigned char tag_text[BUFSIZE];
+int iptc = 0, recno = 0, spec = 0;
+long iptc_bin_len = 0;
+long iptc_bin_elen = 0;
+long content_length = 0;
+long diff_hdr_size = 0;
+long diff_body_size = 0;
+char *mod_buf = NULL;
+char *mod_body_buf = NULL;
+char new_clength[10];
+char tag_time[13];
+char *client_ip;
+squid_off_t mod_size = 0, mod_body_size = 0;
+squid_off_t old_body_size = 0, old_hdr_size = 0;
+int mod_flag = 0;
+
debug(33, 5) (clientSendMoreData: %s, %d bytes\n, http-uri, (int) size);
assert(size = CLIENT_SOCK_SZ);
assert(http-request != NULL);
@@ -2000,6 +2030,289 @@
memFree(buf, MEM_CLIENT_SOCK_BUF);
return;
}
+
+if (JPEG_MOD http-out.offset == 0) {
+rep = clientBuildReply(http, buf, size);
+if(rep) {
+if(rep-content_type.buf != NULL) {
+debug(33, 2) (clientSendMoreData: Reply content-type is :
%s\n, rep-content_type.buf);
+if(strncmp(rep-content_type.buf, image/jpeg, 10) == 0) {
+debug(33, 2) (clientSendMoreData: Reply content-type is
JPEG\n);
+debug(33, 2) (clientSendMoreData: Reply content-length
is: %ld\n, (long)rep-content_length);
+if(rep-content_length == -1 || rep-content_length
25000) {
+debug(33, 2) (clientSendMoreData: Reply
content-length unknown or 25000\n);
+if(size CLIENT_SOCK_SZ) {
+if(rep-content_length == -1) {
+debug(33, 2) (clientSendMoreData: JPEG with
unknown Content-Length: Checking buf for EOI\n);
+debug(33, 2) (clientSendMoreData: buf[size -
2]: %02x\n, (unsigned char)buf[(int)size - 2]);
+debug(33, 2) (clientSendMoreData: buf[size -
1]: %02x\n, (unsigned char)buf[(int)size - 1]);
+if((unsigned char)buf[(int)size - 2] != 0xff
|| (unsigned char)buf[(int)size - 1] != 0xd9) { /* No EOI Marker. Wait for more
... */
+clientWriteComplete(fd, NULL, 0, 0, http);
+memFree(buf, MEM_CLIENT_SOCK_BUF);
+httpReplyDestroy(rep);
+return;
+} else {
+/* image is too small (completely fits in
buf of size CLIENT_SOCK_SZ)
+ break out of JPEG_MOD block and
continue with normal reply processing */
+debug(33, 2) (clientSendMoreData: small
JPEG image found: %d\n, size - rep-hdr_sz);
+}
+} else {
+clientWriteComplete(fd, NULL, 0, 0, http);
+memFree(buf,
Squid-3.0.PRE4 release plan
Okay - let's go with this: 1. Our aim is to release Squid-3.0.PRE4 snapshot over the weekend of 10/11 June. 2. We will work to a fixed list of bugs: those that have PRE4 in the Status Whiteboard. http://tinyurl.com/kaoqq This list contains the nine criticals/blockers as of today, so we are still aiming at the most serious bugs. However, the list won't grow just because we find more. PRE4 will roll as soon as we fix them all. If we hit the target date and we haven't fixed all of these bugs, the plan is still to make a release, as there is still value in this. If we have _nearly_ met our bug target, and a few more days are needed to tie things up, then we may extend the deadline once by up to one week. I will monitor the rate of progress, and adjust the goal if it starts to look either too easy or too difficult. Let's see how it goes. While the focus is on fixing the bugs which are blocking PRE4, it would obviously be great to fix as many of the others as possible. So if you've got the time and inclination to fix bugs, but the ones on the critical path are either too much work or are already taken, please feel free to squash the smaller ones. If we have a manpower problem, but otherwise a good rate of progress, it may be worth trying to attract others we know may be interested in helping. Let me know if this sounds like a plan Cheers Doug On 6 May 2006, at 12:08, Doug Dixon wrote: All I aim to get Squid-3.0.PRE4 released in four to six weeks' time. The list of bugs to be fixed in 3.0 is here: http://tinyurl.com/f56qm This includes bugs that were discovered in 2.5 but which still need fixing in 3.0. I am considering three options for setting the Squid-3.0.PRE4 release criteria: - Option 1 - severity based We release PRE4 when all the criticals and blockers have been fixed. Option 2 - hand-picked We hand pick a fixed list of bugs which we think we can fix within a reasonably short time. We release when they are all fixed. No other bugs can enter the list once it is picked. Option 3 - time based We release PRE4 on June 1, and fix whatever bugs we can until then (blockers first). - Option 1 could leave us vulnerable to a never-ending flow of new criticals/blockers. I understand something like this happened in the past. Are we still vulnerable to this or has the real level of serious bugs dropped to a quantifiable level yet? The benefit of this option is that PRE4 means something positive about the level of known defects. But if the cost is too high, e.g. we never get PRE4 out, then I'm not prepared to do this. Options 2 and 3 have the benefit of having fixed criteria, but at the cost of PRE4 not meaning much about its quality. E.g. why don't we release PRE4 today? In principle I prefer Option 1, but if it's too risky then I'd go for Option 3. Please could you reply to this email and vote for Option 1, 2 or 3 with brief reason. And as Henrik says, if you've got new stuff in the pipeline, please shout now. We just need to agree which PRE it goes into - whether 4 or later. Hope this all sounds okay. Cheers Doug On 6 May 2006, at 03:18, Henrik Nordstrom wrote: For those who have not been on the #squid-dev IRC channel lately I can tell that the last weeks has been quite interesting. The most significant news is that Doug Dixon (aka ganso on the IRC) has volunteered for the role as Squid-3.0.PRE4 release manager. Expect a message from him shortly presenting his ideas on how we can get there. To follow up on a few questions from him regarding the current state: The Squid-3 tree is currently best described as in DEVEL state, even if it is carrying a PRE tag. The reason to this is that the original Squid-3.0 release cycle could not be met due to various events and the tree had to be unlocked again to allow for new developments. The goal now should be to be able to enter PRE state again, with ultimately a PRE4 release from where we can work towards the STABLE release. I do not think there is any major changes waiting in queue for getting into Squid-3, and imho minor/isolated features like WCCPv2 or a improved COSS may well get into the tree during a PRE cycle. But there is a quite long list of bugs, both verified and to be analyzed ones. Some critical, many not so critical ones.. Developers having new features in queue which they would like to get into Squid-3.0 please speak up now, allowing for Doug to do his job proper. As for all of us his time is somewhat limited and the timeframe currently considered for a PRE4 release is not very distant. Regards Henrik
