Re: Major pending change in 2.5 for multi-gigabyte file support
Hi Henrik, At 04.38 26/03/2005, Henrik Nordstrom wrote: On Sat, 12 Mar 2005, Henrik Nordstrom wrote: As some of you may have know the support in Squid for handling multi-gigabyte files has long been non-existing at the best, both not at all being able to handle files above 1.9 GB, bad interactions with the cache (both on-disk and memory) in how largeish objects is processed and the inability to write large log files on 32-bit platforms. Unless someone objects I plan to incorporate these changes into Squid-2.5 shortly (14 days or so). Now committed to 2.5 as there was no objections, and I consider this a major flaw of Squid. Most devel branches should be relatively safe from these changes even if this touched quite many files, and where cvsmerge does not figure it out automatically for you it shouldn't be too hard to resolve manually as most changes are relatively trivial in nature. I have merged the Windows branch: only two minor little conflicts !!! :-) Now I'm starting with the testing work. I have found a typo error in configure.in: Index: configure.in === RCS file: /squid/squid/configure.in,v retrieving revision 1.251.2.85 diff -u -p -r1.251.2.85 configure.in --- configure.in26 Mar 2005 02:50:50 - 1.251.2.85 +++ configure.in26 Mar 2005 19:04:11 - @@ -759,7 +759,7 @@ AC_ARG_WITH(large-files, ]) AC_ARG_ENABLE(large-cache-files, -[ --enable-cache-large-files +[ --enable-large-cache-files Enable support for large cache files (2GB). WARNING: on-disk cache format is changed by this option], [ if test $enableval = yes ; then You may want to audit your own developments for the same class of problems as adressed by this patch. Working on awin32 source, the native Windows version of aufs, I have found some little things that could be refined: - Use FD_WRITE_METHOD/FD_READ_METHOD instead of write/read in all ufs, diskd, aufs code, now there is a confusing mix. - There is an old Windows and OS/2 file rename problem already fixed in 3.0 affecting diskd, coss, ufs and aufs: http://www.squid-cache.org/mail-archive/squid-dev/200308/0028.html If you think that this changes are fine, I will commit this. Regards Guido - Guido Serassio Acme Consulting S.r.l. - Microsoft Certified Partner Via Lucia Savarino, 1 10098 - Rivoli (TO) - ITALY Tel. : +39.011.9530135 Fax. : +39.011.9781115 Email: [EMAIL PROTECTED] WWW: http://www.acmeconsulting.it/
Re: Major pending change in 2.5 for multi-gigabyte file support
On Sat, 26 Mar 2005, Reuben Farrelly wrote: Oops...won't compile now :( disk.c:300: error: conflicting types for 'file_write' protos.h:226: error: previous declaration of 'file_write' was here disk.c:335: error: conflicting types for 'file_write_mbuf' protos.h:227: error: previous declaration of 'file_write_mbuf' was here disk.c:392: error: conflicting types for 'file_read' protos.h:228: error: previous declaration of 'file_read' was here Thanks. Should be fixed now. Only worked if --with-large-files were used. Regards Henrik
Re: Major pending change in 2.5 for multi-gigabyte file support
On Sat, 26 Mar 2005, Serassio Guido wrote: AC_ARG_ENABLE(large-cache-files, -[ --enable-cache-large-files +[ --enable-large-cache-files Thanks. Fixed. Working on awin32 source, the native Windows version of aufs, I have found some little things that could be refined: - Use FD_WRITE_METHOD/FD_READ_METHOD instead of write/read in all ufs, diskd, aufs code, now there is a confusing mix. I see your point. The writing of the swap.state header is not optimal. To be proper it should be done using file_write(). - There is an old Windows and OS/2 file rename problem already fixed in 3.0 affecting diskd, coss, ufs and aufs: http://www.squid-cache.org/mail-archive/squid-dev/200308/0028.html If you think that this changes are fine, I will commit this. I am fine with this rename related cleanup. Regards Henrik
Re: Major pending change in 2.5 for multi-gigabyte file support
Hi Henrik, At 09:57 a.m. 27/03/2005, Henrik Nordstrom wrote: On Sat, 26 Mar 2005, Reuben Farrelly wrote: Oops...won't compile now :( disk.c:300: error: conflicting types for 'file_write' protos.h:226: error: previous declaration of 'file_write' was here disk.c:335: error: conflicting types for 'file_write_mbuf' protos.h:227: error: previous declaration of 'file_write_mbuf' was here disk.c:392: error: conflicting types for 'file_read' protos.h:228: error: previous declaration of 'file_read' was here Thanks. Should be fixed now. Only worked if --with-large-files were used. Yep it's fixed - thanks. Now I am seeing requests for icons break (logged in cache.log): 2005/03/27 12:36:05| internalStart: unknown request: GET /squid-internal-static/icons/anthony-dir.gif HTTP/1.0 Host: tornado.reub.net:3128 User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8b2) Gecko/20050325 Firefox/1.0+ Accept: image/png,*/*;q=0.5 Accept-Language: en-us,en;q=0.7,en-nz;q=0.3 Accept-Encoding: gzip,deflate Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7 Keep-Alive: 300 Connection: keep-alive Referer: ftp://ftp.cisco.com/ If-Modified-Since: Wed, 16 Mar 2005 15:12:01 GMT Probably unrelated to this, I also noticed this message in cache.log: 2005/03/27 12:28:22| storeAufsDirRebuildFromSwapLog: Old version detected. Upgrading How 'backward compatible' is this change, ie what are the consequences of using an earlier version or squid-3 where this code wasn't committed? Will it destroy the swap.state? Reuben
Re: Major pending change in 2.5 for multi-gigabyte file support
On Sun, 27 Mar 2005, Reuben Farrelly wrote: Now I am seeing requests for icons break (logged in cache.log): 2005/03/27 12:36:05| internalStart: unknown request: GET /squid-internal-static/icons/anthony-dir.gif HTTP/1.0 Host: tornado.reub.net:3128 Hmm... works here... Probably unrelated to this, I also noticed this message in cache.log: 2005/03/27 12:28:22| storeAufsDirRebuildFromSwapLog: Old version detected. Upgrading This message is due to the safeguards around the --enable-large-cache-files option. How 'backward compatible' is this change, ie what are the consequences of using an earlier version or squid-3 where this code wasn't committed? Will it destroy the swap.state? Provided you are not using the --enable-large-cache-files option the cache should be backwards and forward compatible. Only forward compatibility is provided for the --enable-large-cache-files option. Regards Henrik
Re: Major pending change in 2.5 for multi-gigabyte file support
At 12:56 p.m. 27/03/2005, Henrik Nordstrom wrote: On Sun, 27 Mar 2005, Reuben Farrelly wrote: Now I am seeing requests for icons break (logged in cache.log): 2005/03/27 12:36:05| internalStart: unknown request: GET /squid-internal-static/icons/anthony-dir.gif HTTP/1.0 Host: tornado.reub.net:3128 Hmm... works here... That last commit to stat.c fixed it, thanks :) reuben
Re: Major pending change in 2.5 for multi-gigabyte file support
On Sun, 27 Mar 2005, Reuben Farrelly wrote: At 12:56 p.m. 27/03/2005, Henrik Nordstrom wrote: On Sun, 27 Mar 2005, Reuben Farrelly wrote: Now I am seeing requests for icons break (logged in cache.log): 2005/03/27 12:36:05| internalStart: unknown request: GET /squid-internal-static/icons/anthony-dir.gif HTTP/1.0 Host: tornado.reub.net:3128 Hmm... works here... That last commit to stat.c fixed it, thanks :) Completely unrelated to icons... just a cosmetic change to get the cachemgr objects vm_objects listings correct. Regards Henrik
