The server does not send the whole file because we dont read it all. The amount of wasted work is just how aggressively your server reads ahead.
-Galt On May 7, 2010, at 1:10 PM, "Charnecki, Timothy A" <[email protected]> wrote: > Galt, > > Thanks for getting back to me. > > Are there any plans to return to just requesting the small chunks > for bigWig and bigBed? This would definitely save some work that's > done on my server, and I imagine might speed up things on your end > as my server would be free to serve up the next chunk instead > reading and sending the rest of the file on every request. > > Thanks again, > > Regards, > Tim > > > ________________________________________ > From: Galt Barber [[email protected]] > Sent: Friday, May 07, 2010 2:56 PM > To: Charnecki, Timothy A > Cc: [email protected] > Subject: Re: [Genome] Open-ended Range in GET header requests for > bigWig files > > By the way, it is more accurate to say that we added this change > to support BAM files (not for FTP). > > BigBed and BigWig files were designed > from the beginning for random access in blocks, and so the > byteRange start and end are known and only a minimal > number of blocks are read. Also, the index itself > is very compact. > > The BAM structure however is apparently such that > you don't know ahead of time how much stuff you'll > be reading. You read a little, get some more info, > then read a little more. Luckily many of the items > are grouped sequentially so that keeping the connection > open and reading a little more next time is a good > bet for BAM. > > -Galt > > Galt Barber wrote: >> Hi, Tim! >> >> This was a change added to UDC cache layer >> back in November. >> >> As you have noted, in the current >> implementation the end of the byterange is >> not being used. >> >> I believe that this had something to do with trying to >> improve FTP performance, >> trying to keep a connection open to read successive >> sequential blocks in one go. >> >> You are right that it is actually asking the server for more data >> than >> it will actually read, simply closing the connection when it has >> received enough data. >> But it doesn't appear to be causing any harm. >> >> So, this is working as intended. >> I will double-check with the developer about this question. >> Thanks for reporting it. >> >> -Galt >> >>> -------- Original Message -------- >>> Subject: Open-ended Range in GET header requests for bigWig files >>> Date: Tue, 4 May 2010 14:38:08 -0500 >>> From: Charnecki, Timothy A <[email protected]> >>> To: [email protected] <[email protected]> >>> >>> Hello, >>> >>> I noticed something that appears to have changed When viewing bigWig >>> files in the browser I am seeing GET requests to my server that have >>> headers containing byte ranges that don't have the end byte >>> defined. I >>> understand that this is allowed by HTTP, but understand that the >>> server >>> will serve the start byte through the end of the file. I can see >>> requests with header "Range => bytes=0-". This causes the server to >>> attempt to serve the entire file. I know I used to see requests with >>> much smaller request ranges such as "Range => bytes=0-8191". My >>> data is >>> displaying without any problems in the browser, I'm just wondering >>> if >>> these partial range requests are intentional on your end. >>> >>> Thanks, >>> Tim Charnecki >>> >> >> _______________________________________________ >> Genome maillist - [email protected] >> https://lists.soe.ucsc.edu/mailman/listinfo/genome _______________________________________________ Genome maillist - [email protected] https://lists.soe.ucsc.edu/mailman/listinfo/genome
