editors/vim -- long fetch times

[Jeremy Chadwick]

This is a subject I've wanted to bring up for a few years now.  Why I
decided to bring it up now, I don't know; I guess I just felt it was
time.

Simply put: the time it takes for make fetch to complete in
editors/vim gets worse and worse as more patches are released:

icarus# cd /usr/ports/editors/vim-lite  make distclean
icarus# time make fetch
...
10.801u 3.297s 6:16.33 3.7% 568+1347k 0+52io 0pf+0w

That's over 6 minutes to fetch 116 patches and the base vim tarball,
totalling about 7MBytes of data:

icarus# du -sk /usr/ports/distfiles/vim
7248/usr/ports/distfiles/vim

True, it's only excessive when you have none of the patches in
DISTFILES/vim (that is to say, if you already have 115 patches and
there's 116 of them, the wait time isn't very long).

The wasted time is caused by a few things.  This opens somewhat of a
can of worms (in regards to how to enhance fetch(1)), but the main two:

* FTP protocol is used for each and every patch.  FTP server may be
  doing things like forward and reverse DNS look-ups on client, as well
  as an RFC931/1413 ident check.
* fetch(1) does not support the ability to fetch multiple files during
  the same FTP session.  The idea would be to modify fetch(1) to support
  doing multiple RETRs during a single FTP session.

Solutions as I see them:

* Change the first site in MASTER_SITE_VIM to the HTTP version of
  the main distribution site: http://ftp.vim.org/pub/vim/unix/.  This
  is already in the list, but the HTTP mirrors are *last*, not first.
* Provide snapshots of the vim patches as a tgz-ball somewhere.
  Have make fetch download that and untar it into DISTFILES/vim/.
  Snapshots would need to be updated every time a new patch was added.
* Modify fetch(1) as described above.
* Get Bram to stop releasing hundreds of individual patches between
  minor releases.  :-) 

Using the HTTP fetch method, the time drastically decreases to 1.5
minutes for me:

icarus# cd /usr/ports/editors/vim-lite  make distclean
icarus# time make MASTER_SITE_VIM=http://ftp.vim.org/pub/vim/unix/ fetch
9.282u 2.747s 1:28.83 13.5% 628+1444k 1+52io 0pf+0w

Thoughts?

-- 
| Jeremy Chadwickjdc at parodius.com |
| Parodius Networking   http://www.parodius.com/ |
| UNIX Systems Administrator  Mountain View, CA, USA |
| Making life hard for others since 1977.  PGP: 4BD6C0CB |

___
freebsd-ports@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to [EMAIL PROTECTED]

Re: editors/vim -- long fetch times

[RW]

On Mon, 8 Oct 2007 01:47:57 -0700
Jeremy Chadwick [EMAIL PROTECTED] wrote:

 This is a subject I've wanted to bring up for a few years now.  Why I
 decided to bring it up now, I don't know; I guess I just felt it was
 time.
 
 Simply put: the time it takes for make fetch to complete in
 editors/vim gets worse and worse as more patches are released:
 
 ...
 Solutions as I see them:
 
 * Change the first site in MASTER_SITE_VIM to the HTTP version of
   the main distribution site: http://ftp.vim.org/pub/vim/unix/.  This
   is already in the list, but the HTTP mirrors are *last*, not first.

You can already change this by setting 

MASTER_SORT_REGEX= ^http

on such ports

 * Provide snapshots of the vim patches as a tgz-ball somewhere.
   Have make fetch download that and untar it into DISTFILES/vim/.
   Snapshots would need to be updated every time a new patch was added.

The trouble is that this penalises people who do keep their distfiles
cached, in favour of those that don't. And as you pointed-out this
whole thing is a non-issue if you do cache them.

 * Modify fetch(1) as described above.

It's quite easy to fetch distfiles in parallel with building, which
in most cases eliminates waiting for downloads. 

___
freebsd-ports@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to [EMAIL PROTECTED]

Re: editors/vim -- long fetch times

[Pav Lucistnik]

Jeremy Chadwick píše v po 08. 10. 2007 v 01:47 -0700:

 Simply put: the time it takes for make fetch to complete in
 editors/vim gets worse and worse as more patches are released:
 
 icarus# cd /usr/ports/editors/vim-lite  make distclean
 icarus# time make fetch
 ...
 10.801u 3.297s 6:16.33 3.7% 568+1347k 0+52io 0pf+0w
 
 That's over 6 minutes to fetch 116 patches and the base vim tarball,
 totalling about 7MBytes of data:

 Solutions as I see them:

 * Provide snapshots of the vim patches as a tgz-ball somewhere.
   Have make fetch download that and untar it into DISTFILES/vim/.
   Snapshots would need to be updated every time a new patch was added.

vim people already do that - they do a rollup tarball every 100 patches.
See at http://ftp.vim.org/pub/vim/patches/7.1/

 * Modify fetch(1) as described above.

Wouldn't help - the ports framework is getting one distfile at a time.

-- 
Pav Lucistnik [EMAIL PROTECTED]
  [EMAIL PROTECTED]


signature.asc
Description: Toto je digitálně	 podepsaná část	 zprávy