Re: [PATCH 5.1] zread: read files in 4k chunks
The 'stat(2)' system call returns an optimal i/o size for files since some files in addition to being on disks with a 4k sector size (making 128bytes a slow choice for reads, and a real slow choice for writes), also can be on a RAID with it's own optimal i/o size. I think the 'stat(1)' prog shows the same value with "%o": "stat -c%o" . 65536 On Mon, Jun 22, 2020 at 1:19 PM Jason A. Donenfeld wrote: > On Mon, Jun 22, 2020 at 2:16 PM Ilkka Virta wrote: > > > > On 22.6. 19.35, Chet Ramey wrote: > > > On 6/22/20 1:53 AM, Jason A. Donenfeld wrote: > > >> Currently a static sized buffer is used for reading files. At the > moment > > >> it is extremely small, making parsing of large files extremely slow. > > >> Increase this to 4k for improved performance. > > > > > > I bumped it up to 1024 initially for testing. > > > > It always struck me as odd that Bash used such a small read of 128 > > bytes. Most of the GNU utils I've looked at on Debian use 8192, and a > > simple test program seems to indicate glibc's stdio reads 4096 bytes at > > one read() call. > > Plus most other shells people use... > >
Re: [PATCH 5.1] zread: read files in 4k chunks
On 6/22/20 4:16 PM, Ilkka Virta wrote: > On 22.6. 19.35, Chet Ramey wrote: >> On 6/22/20 1:53 AM, Jason A. Donenfeld wrote: >>> Currently a static sized buffer is used for reading files. At the moment >>> it is extremely small, making parsing of large files extremely slow. >>> Increase this to 4k for improved performance. >> >> I bumped it up to 1024 initially for testing. > > It always struck me as odd that Bash used such a small read of 128 bytes. > Most of the GNU utils I've looked at on Debian use 8192, and a simple test > program seems to indicate glibc's stdio reads 4096 bytes at one read() call. Yes, 128 is too small for modern systems. It made more sense when the code was written. -- ``The lyf so short, the craft so long to lerne.'' - Chaucer ``Ars longa, vita brevis'' - Hippocrates Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/
Re: [PATCH 5.1] zread: read files in 4k chunks
On Mon, Jun 22, 2020 at 2:16 PM Ilkka Virta wrote: > > On 22.6. 19.35, Chet Ramey wrote: > > On 6/22/20 1:53 AM, Jason A. Donenfeld wrote: > >> Currently a static sized buffer is used for reading files. At the moment > >> it is extremely small, making parsing of large files extremely slow. > >> Increase this to 4k for improved performance. > > > > I bumped it up to 1024 initially for testing. > > It always struck me as odd that Bash used such a small read of 128 > bytes. Most of the GNU utils I've looked at on Debian use 8192, and a > simple test program seems to indicate glibc's stdio reads 4096 bytes at > one read() call. Plus most other shells people use...
Re: [PATCH 5.1] zread: read files in 4k chunks
On 22.6. 19.35, Chet Ramey wrote: On 6/22/20 1:53 AM, Jason A. Donenfeld wrote: Currently a static sized buffer is used for reading files. At the moment it is extremely small, making parsing of large files extremely slow. Increase this to 4k for improved performance. I bumped it up to 1024 initially for testing. It always struck me as odd that Bash used such a small read of 128 bytes. Most of the GNU utils I've looked at on Debian use 8192, and a simple test program seems to indicate glibc's stdio reads 4096 bytes at one read() call. -- Ilkka Virta / itvi...@iki.fi
Re: [PATCH 5.1] zread: read files in 4k chunks
Hi Chet, A little bit larger than 1024, I believe, so 2048 would be reasonable. However, consider taking this patch as-is with the 4096 page size. This will yield much better performance and have basically no drawbacks on any system. Another place lbuf should be increased is in zcatfd, by the way. Jason
Re: [PATCH 5.1] zread: read files in 4k chunks
On 6/22/20 1:53 AM, Jason A. Donenfeld wrote: > Currently a static sized buffer is used for reading files. At the moment > it is extremely small, making parsing of large files extremely slow. > Increase this to 4k for improved performance. I bumped it up to 1024 initially for testing. > This also allows reading > files like /dev/kmsg on Linux, which will error if the read buffer is > too small. What is the minimum size for those reads? Chet -- ``The lyf so short, the craft so long to lerne.'' - Chaucer ``Ars longa, vita brevis'' - Hippocrates Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/
