Tim Connors <[EMAIL PROTECTED]> writes: > > > > wwwoffle has the setting: > > > > # lock-files = yes | no > > > > # Enable the use of lock files to stop more than one WWWOFFLE > > > > process > > > > # from downloading the same URL at the same time (default=no). > > > > > > > > Either way this is set, it is broken. > > > > > > > > If set to yes, it seems that a lockfile only manages to tell the > > > > second process to give up loading the page at all, giving back a HTTP > > > > 500 WWWOFFLE Server Error: > > > > > > > > for i in `seq 1 10 ` ;do lynx -dump http://www.google.com.au & done > > > > > > _________________________________________________________________ > > > > > > > > WWWOFFLE Server Error > > > > > > > > The WWWOFFLE server encountered a fatal error: > > > > > > > > Cannot open the spooled web page to read. > > > > The program cannot continue to service this request. > > > > > > _________________________________________________________________ > > > > You should not be getting this error message. You should get this > > error instead: > > > > _________________________________________________________________ > > > > WWWOFFLE File Locked > > > > Your request for URL > > > > <<<URL>>> > > is already being modified by another WWWOFFLE server. > > I've seen that one too, but in the case above, it was the error message > given.
As I said in the previous message I don't understand why you get this behaviour, you should be getting the message that is dedicated to lock-file timeouts. The only way to debug this one is to capture a log file with full debugging enabled (FAQ explains how). I have been using WWWOFFLE with lock-files for 10 years and I very rarely see the "File Locked" message page. I don't remember ever seeing the "Server error" page in this case. > > The purpose of this page is to explain exactly what has happened. The > > system is not broken, but busy. If it happens all the time then it > > might be broken and there is even an explanation of how to solve it. > > > > This error page should not appear until a timeout of 1/6th of the > > socket timeout option in the configuration file (so 10 seconds for a > > 60 second socket timeout). > > So the condition will be met always for reasonably large downloads (one > that takes longer than 10 seconds), or when there's a bit of latency > around? It will likely be met if the same URL was requested twice quite close to each other. If the timeout was made longer (up to infinite time to avoid giving an error page) then one more of the limited number of WWWOFFLE servers would be tied up waiting for the file. If a page is requested twice then why not three times or four times until all of the WWWOFFLE servers are waiting for the same download to finish. > > The purpose of the lockfile is to stop the same file being downloaded > > many times. One of the key features of WWWOFFLE is the ability to > > reduce the number of bytes downloaded. To do this you need a method > > to ensure that mutliple downloads of the same file do not occur. > > I have outlined a couple of situations in a prior message to this bug, > (http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=400329.html) > where I think this behaviour is just plain unreasonable. > > Sure, squid is a lot more complicated and not aimed at the purpose of > caching for a small installation behind a dialup, but it gets this > behaviour dead right -- the first process to access a page will get the > page come down at line speed, and simulataneously, the page gets spooled > to disk. A second subsequent request will get the page so far on disk, at > disk read speed, then when that has been exhausted, it gets exactly the > same data simultaneously being sent to the first client. It is very difficult to get this to work correctly. You can end up with infinite waiting if the original server crashes and doesn't delete the lock file. You can need very complicated inter-process communication so that each server knows who is waiting for which file so that if one dies the others can take over. You can get race conditions where the original server finishes, deletes the lock file and then another process gets the file again creating a new lock file and blocking for the same duration. The solution that I have implemented is one that works most of the time, does not require any inter-process communication (just the lock file) and does not lock-up forever in failure cases. Which solution do you want? An infinite wait on the lock-file to disapear is easy to implement. A configurable wait can also be done. Inter-process communication to get it perfect gets a "wontfix" in Debian bug report language. You obviously don't want the current behaviour without lock-files since you have given that a "dataloss" tag. Without lock-files as a simple method of inter-process communication (and any other method is a "wontfix") I don't see that this option leads to a solution. -- Andrew. ---------------------------------------------------------------------- Andrew M. Bishop [EMAIL PROTECTED] http://www.gedanken.demon.co.uk/ WWWOFFLE users page: http://www.gedanken.demon.co.uk/wwwoffle/version-2.9/user.html -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
