On 06/04/2024 03:52, Takashi Kusumi wrote:
Hi,I have found a performance issue with the sort command when used on pseudo files with zero size. For instance, sorting `/proc/kallsyms`, as demonstrated below, takes significantly longer than executing with `cat`, generating numerous temporary files. I confirmed this issue on v8.32 as well as on commit 8f3989d in the master branch. $ time cat /proc/kallsyms | sort > /dev/null real 0m0.954s user 0m0.873s sys 0m0.096s $ time sort /proc/kallsyms > /dev/null real 0m8.555s user 0m3.367s sys 0m5.064s $ strace -e trace=openat sort /proc/kallsyms 2>&1 > /dev/null \ | grep /tmp/sort | head -100 ... openat(AT_FDCWD, "/tmp/sortM6Y6Y1", ... openat(AT_FDCWD, "/tmp/sortPrHKMG", ... $ strace -e trace=openat -c sort /proc/kallsyms > /dev/null % time seconds usecs/call calls errors syscall ------ ----------- ----------- --------- --------- ---------------- 100.00 6.419777 19 333258 8 openat ------ ----------- ----------- --------- --------- ---------------- 100.00 6.419777 19 333258 8 total It appears that the buffer size allocated for pseudo files with zero size is insufficient, likely because it is based on their file size, which is zero. As seen in the attached patch, I think using `INPUT_FILE_SIZE_GUESS` to calculate the buffer size when the file size is zero would resolve this issue.
I'll apply this. BTW we should improve sort buffer handling in general. From my TODO... 0. Have sort --debug output memory buffer sizes and space avail at $TMPDIR(s) 1. auto increase buffer when reading from pipe or zero sized files. This will be more efficient and more importantly enable parallel operation. See http://superuser.com/questions/938558/sort-parallel-isnt-parallelizing/ At least your more appropriate default buffer sizes in this case. I.e. bigger mins and probably smaller maxs as half avail mem is too aggressive. 2. check() should not need full buffer size? only merge buffer size or something small at least. 3. Look at minimizing the amount of mem used by default. Hmm, sort auto adjusts down to avail mem in initbuf() (Test with ulimit -v) 4. Careful with too small buffers as that may initiate an extra merge step (see section above). If anyone wants to look at the above give me a heads up, or I'll get to it sometime in the next release cycle. thanks! Pádraig.
