Davi Arnaut wrote: > Graham Leggett wrote: >> Davi Arnaut wrote: >> >>> Yes, first because size_t is 32 bits :). When you do a read like this on >>> a file bucket, the whole bucket is not read into memory. The read >>> function splits the bucket and changes the current bucket to refer to >>> data that was read. >> 32 bits is 4GB. A large number of webservers don't have memory that >> size, thus the problem. >> >>> The problem lies that those new buckets keep accumulating in the >>> brigade! See my patch again. >> Where? > > I was referring to "it will attempt to read all 4.7GB". Such a thing > does not exist! > >> We start with one 4.7GB bucket in a brigade. > > No, we start with a file bucket on the brigade, I will try to explain > what happens to see if we are talking about the same thing. > > Suppose we have a brigade containing a a file bucket, and the file size > is 4.7GB. We want to read it fully. > > When we call apr_bucket_read() on this bucket, we end-up calling the > bucket read function (file_bucket_read). What does the bucket file read do ? > > If mmap is supported, it mmaps APR_MMAP_LIMIT bytes (4MB) by creating a > new mmap bucket and splits the bucket. So, after calling read on a file > bucket you have two buckets on the brigade. The first one is the mmap > bucket and last is file bucket with a updated offset.
s/last/last one is the old/ > The same thing happens if mmap is not supported, but the bucket type > will be a heap bucket. If we don't delete or flush those implicitly > created buckets they will keep the whole file in memory, but one single > read will not put the entire file on memory. s/keep the whole file in memory/will accumulate on the brigade, implying that the whole file is in memory/ > What Joe's patch does is remove this first implicitly created bucket > from the brigade, placing it on the brigade on a temporary brigade for > sending it to the client. s/on the brigade// > That's why splitting the brigade with magical values (16MB) is not such > a good idea, because the bucket type knows betters and will split the > bucket anyway. -- Davi Arnaut
