Hi dennis berger!
On Wed, 15 Jun 2011 13:14:32 +0200; dennis berger wrote about 'Re: strange
differencies: find -exec \+ vs find | xargs':
> Am 15.06.2011 um 12:33 schrieb Vadim Goncharov:
>> Hi,
>>
>> There was a flamewar in a local maillist about the fastest way to do chmod
>> 644
>> on files and chmod 755 on directories in a sbubtree. It suddenly revealed the
>> strange thing: on the same subtree, find -type d -exec \+ on 76000
>> directories
>> has the same as find | xargs, but on 246000 files it was 4 times slower -
>> while
>> there must be almost identical number of forks.
>>
>> I've repeated the tests on deliberately old machine, a Pentium I 233 MHz with
>> 80 Mb RAM, 7.4R UFS2. Subtree for tests was copy of /usr/src with several
>> categories from /usr/ports. The output is from ZSH's builtin 'time' (also
>> set up to automatically output if cmd was run more than 30 secs).
>>
>> Below is the ooutput, first with 80K files, it doesn't differ, than added
>> several more /usr/ports categories and got slow result on 100K. Why is
>> the difference at all? This machine has maxvnodes about 4600 and too small
>> RAM for caches, if that's buffers, I'd expect the difference to begin on
>> mush slow number of files. But it runs the same 80K normally...
>>
>> kernblitz:/home/vadim# find /warehouse/garbage -type f | wc -l; find
>> /warehouse/garbage -type d | wc -l
>> 80208
>> find /warehouse/garbage -type f 5,01s user 53,12s system 53% cpu 1:48,52
>> total
>> wc -l 0,54s user 0,31s system 0% cpu 1:48,49 total
>> 13989
>> find /warehouse/garbage -type d 4,52s user 52,00s system 53% cpu 1:45,63
>> total
>> wc -l 0,11s user 0,06s system 0% cpu 1:45,61 total
>>
>> kernblitz:/home/vadim# time chmod -R u+rwX,go+rX,go-w /warehouse/garbage
>> chmod -R u+rwX,go+rX,go-w /warehouse/garbage 4,39s user 54,94s system 53%
>> cpu 1:50,73 total
>>
>> kernblitz:/home/vadim# time find /warehouse/garbage -type f -exec chmod 644
>> {} \+
>> find /warehouse/garbage -type f -exec chmod 644 {} \+ 7,29s user 100,30s
>> system 65% cpu 2:43,29 total
>>
>> kernblitz:/home/vadim# time find /warehouse/garbage -type d -exec chmod 755
>> {} \+
>> find /warehouse/garbage -type d -exec chmod 755 {} \+ 5,04s user 58,91s
>> system 54% cpu 1:58,05 total
>>
>> kernblitz:/home/vadim# time find /warehouse/garbage -type f -print0 | xargs
>> -0 chmod 644
>> find /warehouse/garbage -type f -print0 4,78s user 55,14s system 37% cpu
>> 2:40,27 total
>> xargs -0 chmod 644 3,26s user 46,70s system 30% cpu 2:41,26 total
>>
>> kernblitz:/home/vadim# cp -r /usr/ports/[e-k]* /warehouse/garbage
>> cp -r /usr/ports/[e-k]* /warehouse/garbage 4,57s user 115,70s system 29%
>> cpu 6:54,47 total
>>
>> kernblitz:/home/vadim# find /warehouse/garbage -type f | wc -l; find
>> /warehouse/garbage -type d | wc -l
>> 102014
>> find /warehouse/garbage -type f 6,46s user 69,75s system 48% cpu 2:38,31
>> total
>> wc -l 0,78s user 0,27s system 0% cpu 2:38,29 total
>> 19714
>> find /warehouse/garbage -type d 6,11s user 72,32s system 48% cpu 2:43,26
>> total
>> wc -l 0,13s user 0,09s system 0% cpu 2:43,24 total
>>
>> kernblitz:/home/vadim# time chmod -R u+rwX,go+rX,go-w /warehouse/garbage
>> chmod -R u+rwX,go+rX,go-w /warehouse/garbage 5,59s user 69,97s system 48%
>> cpu 2:36,53 total
>>
>> kernblitz:/home/vadim# time find /warehouse/garbage -type d -exec chmod 755
>> {} \+
>> find /warehouse/garbage -type d -exec chmod 755 {} \+ 6,72s user 83,05s
>> system 23% cpu 6:15,22 total
>>
>> kernblitz:/home/vadim# time find /warehouse/garbage -type f -exec chmod 644
>> {} \+
>> find /warehouse/garbage -type f -exec chmod 644 {} \+ 10,49s user 138,30s
>> system 20% cpu 12:14,49 total
>>
>> kernblitz:/home/vadim# time find /warehouse/garbage -type f -print0 | xargs
>> -0 chmod 644
>> find /warehouse/garbage -type f -print0 6,58s user 70,84s system 35% cpu
>> 3:36,67 total
>> xargs -0 chmod 644 3,93s user 59,30s system 28% cpu 3:39,50 total
>>
> I suspect that you're hitting a cache limit for that machine.
> I had similar behaviour when rsyncing huge directories. I usually raise
> vfs.ufs.dirhash_maxmem sysctl variable, which solves the problem.
So I've run those tests again on that old machine, doing in another shell:
while sleep 5; do sysctl vfs.ufs.dirhash_mem vfs.ufs.dirhash_lowmemcount; done
and dound that memory usage for dirhash hasn't go above 110 Kb.
Even more strange, now last tests were not 12 minutes but less than 4 minutes,
while chmod still was less than 3 minutes.
So, this was far from dirhash_maxmem. I don't know how to explain this. I can
only suspect that this old machine was hitting another memory issue (there were
39 Mb of swap used).
Any ideas for more correct tests / test conditions ?..
--
WBR, Vadim Goncharov. ICQ#166852181 mailto:vadim_nucli...@mail.ru
[Moderator of RU.ANTI-ECOLOGY][FreeBSD][http://antigreen.org][LJ:/nuclight]
_______________________________________________
freebsd-performance@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-performance
To unsubscribe, send any mail to "freebsd-performance-unsubscr...@freebsd.org"
