On Friday 13 November 2009 17:18:17 Edmund R. MacKenty wrote: > On Friday 13 November 2009 17:14, McKown, John wrote: > >I think you're right. He was worried that instead of his program just > > reading the file(s), the I/O would be: (1) "cat" reading the files from > > disk; (2)writing the contents to the pipe and (3) his program reading the > > pipe. Or about 3x the I/O. But pipes are not hardened to disk (ignoring > > paging?), so it is more like a VIO dataset in z/OS (VIO datasets are > > written to memory only). > > I was curious to see what the overhad is, so I made five 1GB files: > > # for f in one two three four five; do \ > dd if=/dev/urandom of=$f bs=1M count=1024; done > > and then tried running a simple tool that would just read through them all > to get a base time: > > # time od one two three four five > /dev/null > real 21m13.009s > user 19m52.603s > sys 0m11.557s > > Then I tried the named pipe approach: > > # mknod pipe p > # time (cat one two three four five > pipe & od -c pipe > /dev/null) > real 58m19.154s > user 56m56.490s > sys 0m20.361s > > Now this is just on a laptop, and a very crude measurement, but it sure > looks like there's a bit of overhead in them thar named pipes and cat! > - MacK.
If I remember correctly, the originator of this thread was attempting to process LOTS of LITTLE files, so it might be interesting to try it that way... ========================== 19:03:14 turr...@pinto$ concatTest Making 200 test files... 10+0 records in 10+0 records out 10240 bytes (10 kB) copied, 0.00760478 s, 1.3 MB/s : 10+0 records in 10+0 records out 10240 bytes (10 kB) copied, 0.00420962 s, 2.4 MB/s Processing by simple catenation... real 0m0.498s user 0m0.388s sys 0m0.028s Processing by named pipe... real 0m1.571s user 0m1.436s sys 0m0.028s Removing test files... Done. ============================== Seems like there is quite a bit of overhead with a pipe... ============================== 19:03:22 turr...@pinto$ cat ~/bin/concatTest #!/bin/bash echo echo "Making 200 test files..." for f in f{1..200} do dd if=/dev/urandom of=${f} bs=1K count=10 done echo echo "Processing by simple catenation..." time od f{1..200} >/dev/null echo echo "Processing by named pipe..." mknod pipe p time (cat f{1..200} >pipe & od -c pipe >/dev/null) echo echo "Removing test files..." rm f{1..200} echo echo "Done." echo ================================ Leslie ---------------------------------------------------------------------- For LINUX-390 subscribe / signoff / archive access instructions, send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390