On Tue, May 28, 2013 at 07:51:37PM +0200, Michael Gmelin wrote: > On Tue, 28 May 2013 15:16:00 +0100 > RW <rwmailli...@googlemail.com> wrote: > > > On Tue, 28 May 2013 15:14:52 +0200 > > John Marino wrote: > > > > > All > > > patches only take 74 seconds to download[2] so there is no sympathy > > > for your obviously single data point anecdote, > > > > Well at the point you provided one data-point there was only one data > > point. And it was like pulling teeth to get you to eliminate the > > alternative explanations. Was it really too much to ask that you > > provided some actual evidence. > > > > > you're clearly doing > > > something wrong. You need to stop complaining and start think about > > > folks with slow connections[3] who also rebuild Vim frequently. > > > > Don't make things up. I never said anything about frequent rebuilds, > > the patches all get redownloaded on the next rebuild. > > The real issue is not the number of patches, but the fact that every > single patch is downloaded by invoking the fetch(1) command, creating > lots of overhead not limited to the fetch command itself. The ports > system wasn't designed for such an amount of distfiles in a single > port I guess. > > I just timed fetching the patches through ports vs. fetching over > HTTP/1.1 using ftp/curl vs calling fetch directly. The VIM tarball was > already downloaded, so this is really just the patches (plus > downloading 6mb is barely noticeable on a fast line). It's a slow > machine on a fast line. > > Fetch: > [user@server /usr/ports/editors/vim]$ time sudo make fetch > .... > real 4m57.327s > user 0m17.010s > sys 0m39.588s > > Curl: > [user@server /tmp]$ longcurlcommandline > .... > real 0m15.291s > user 0m0.026s > sys 0m0.272s > > Fetch on the command line (after initial make fetch, so this is only > measuring transmission of the files): > cd /usr/ports/editors/distfiles > time for name in 7.3.*; do > fetch http://artfiles.org/vim.org/patches/7.3/$name > done > .... > real 1m25.329s > user 0m0.660s > sys 0m3.174s > > So just the fact we're invoking fetch for every file costs us about one > minute - I assume the time lost is much bigger on a slow line with > long latency. The remaining 3.5 minutes are spent somewhere in the > ports infrastructure and clearly depend on the performance of the > machine used. For comparison I timed "make fetch" on a reasonably fast > server (good IO, fast datacenter connection), make fetch still took > about 120 seconds(!). > > So the bottomline is: > - Using HTTP/1.1 and keepalive could safe a lot of time > - The ports infrastructure creates a lot of overhead per patch file > > Maybe there's something we can do to improve the situation. > > Cheers, > Michael > > PS: I don't use vim myself and have no stake in this discussion > whatsoever.
Someone in this thread proposed to change the port to use phttpget, so I
gave it a try using a German mirror nearby with 6 Mbit/s downlink:
$ time /usr/libexec/phttpget ftp.vim.ossmirror.de $(eval echo
/pub/vim/patches/7.3/{$(make -C /usr/ports/editors/vim -VPATCHFILES | sed 's/\
/,/g')})
http://ftp.vim.ossmirror.de//pub/vim/patches/7.3/7.3.001: 200 OK
http://ftp.vim.ossmirror.de//pub/vim/patches/7.3/7.3.002: 200 OK
http://ftp.vim.ossmirror.de//pub/vim/patches/7.3/7.3.003: 200 OK
http://ftp.vim.ossmirror.de//pub/vim/patches/7.3/7.3.004: 200 OK
http://ftp.vim.ossmirror.de//pub/vim/patches/7.3/7.3.005: 200 OK
[...]
http://ftp.vim.ossmirror.de//pub/vim/patches/7.3/7.3.974: 200 OK
http://ftp.vim.ossmirror.de//pub/vim/patches/7.3/7.3.984: 200 OK
http://ftp.vim.ossmirror.de//pub/vim/patches/7.3/7.3.985: 200 OK
http://ftp.vim.ossmirror.de//pub/vim/patches/7.3/7.3.1000: 200 OK
real 0m12.509s
user 0m0.154s
sys 0m0.089s
That's really nice!
Compare this to the current version using fetch(1):
time make PATCH_SITES=http://ftp.vim.ossmirror.de/pub/vim/patches/7.3/ fetch
===> Found saved configuration for vim-7.3.669_1
===> vim-7.3.1014 depends on file: /usr/local/sbin/pkg - found
=> 7.3.002 doesn't seem to exist in /usr/ports/distfiles/vim.
=> Attempting to fetch http://ftp.vim.ossmirror.de/pub/vim/patches/7.3/7.3.002
7.3.002 100% of 1610 B 16 MBps 00m00s
=> 7.3.003 doesn't seem to exist in /usr/ports/distfiles/vim.
=> Attempting to fetch http://ftp.vim.ossmirror.de/pub/vim/patches/7.3/7.3.003
7.3.003 100% of 1299 B 1281 kBps 00m00s
=> 7.3.004 doesn't seem to exist in /usr/ports/distfiles/vim.
[...]
=> 7.3.984 doesn't seem to exist in /usr/ports/distfiles/vim.
=> Attempting to fetch http://ftp.vim.ossmirror.de/pub/vim/patches/7.3/7.3.984
7.3.984 100% of 1706 B 2852 kBps 00m00s
=> 7.3.985 doesn't seem to exist in /usr/ports/distfiles/vim.
=> Attempting to fetch http://ftp.vim.ossmirror.de/pub/vim/patches/7.3/7.3.985
7.3.985 100% of 1691 B 14 MBps 00m00s
=> 7.3.1000 doesn't seem to exist in /usr/ports/distfiles/vim.
=> Attempting to fetch http://ftp.vim.ossmirror.de/pub/vim/patches/7.3/7.3.1000
7.3.1000 100% of 1637 B 1715 kBps 00m00s
===> Fetching all distfiles required by vim-7.3.1014 for building
Total time : 3:48.55s
CPU utilisation (percentage) : 54.5%
That's much slower using the same mirror.
pgpJAsBTpip0v.pgp
Description: PGP signature
