Re: setup RFC: Ditch homegrown http/ftp code and use a library?

[Robert Collins]

On Sun, 2004-11-14 at 03:08 +0100, Lapo Luchini wrote:
> Robert Collins wrote:
> 
> >Thats my point : rsync doesn't require the same file name to use the
> >content for optimisation. You tell rsync what file to use as the basis,
> >and what file to write to.
> >
> This, if you rsync one file at a time, but if you create the symlinks in
> every directory, then you can rsync the whole directory tree with a
> single command, and is much more efficient. (only one connection for all
> the files)

Huh? Doing what I suggest has nothing to do with the # of connections,
the protocol supports it completely. It might, if you are spawning rsync
locally each time, with the file list option, you could (easily) add a
new file destination list, and that would provide the single connection,
custom basis approach.

> Anyway... this could lead to a more-optimized script to "download most
> of it", not really to be included in "setup" so...

Sure.

Rob 



signature.asc
Description: This is a digitally signed message part

Re: setup RFC: Ditch homegrown http/ftp code and use a library?

[Lapo Luchini]

Robert Collins wrote:

>Thats my point : rsync doesn't require the same file name to use the
>content for optimisation. You tell rsync what file to use as the basis,
>and what file to write to.
>
This, if you rsync one file at a time, but if you create the symlinks in
every directory, then you can rsync the whole directory tree with a
single command, and is much more efficient. (only one connection for all
the files)

Anyway... this could lead to a more-optimized script to "download most
of it", not really to be included in "setup" so...

-- 
Lapo Luchini
[EMAIL PROTECTED] (PGP & X.509 keys available)
http://www.lapo.it (ICQ UIN: 529796)

Re: (TRY 2) Re: many complaints about setup and X

[Bobby McNulty]

Christopher Faylor wrote:
On Thu, Nov 11, 2004 at 10:47:20PM -0500, Christopher Faylor wrote:
 

Are any of the setup.exe developers attempting to get to the bottom of
the problems that have been reported lately when installing X?
I see that Igor is attempting to help but it seems like we're seeing
an increasing number of people complaining about this and it would be
nice if someone could try to get to the bottom of the problem.
   

Any possibility that someone could take a look at what seems to be a
nasty setup.exe problem?  There is lots of discussion in cygwin-xfree.
It's mainly under the subject "xorg installation fails, 99% complete only".
cgf
 

Chris, I had to install Cygwin and then xorg.
This solved my problem.
But still, its a problem with setup.

(TRY 2) Re: many complaints about setup and X

[Christopher Faylor]

On Thu, Nov 11, 2004 at 10:47:20PM -0500, Christopher Faylor wrote:
>Are any of the setup.exe developers attempting to get to the bottom of
>the problems that have been reported lately when installing X?
>
>I see that Igor is attempting to help but it seems like we're seeing
>an increasing number of people complaining about this and it would be
>nice if someone could try to get to the bottom of the problem.

Any possibility that someone could take a look at what seems to be a
nasty setup.exe problem?  There is lots of discussion in cygwin-xfree.
It's mainly under the subject "xorg installation fails, 99% complete only".

cgf

Re: setup RFC: Ditch homegrown http/ftp code and use a library?

[Christopher Faylor]

On Sat, Nov 13, 2004 at 02:08:58AM -0800, Brian Dessent wrote:
>Lapo Luchini wrote:
>>>It would be nice if setup.exe supported the rsync protocol.  That would
>>>also give you the ability to suspend and resume.  I don't know how easy
>>>it would be to get librsync working in a mingw environment, though.
>>
>>This may not be really "up-to-date" but rsync use is still "your life
>>is so much better if you spawn an actual rsync program to do the work",
>>e.g.  rsync authors themself "suggest" not to use librsync.
>
>Plus, with ftp resume and http byte-range support from libcurl (or
>equiv) you can achieve almost the same thing.

I am aware that you can do the same thing with other protocols.

Some of the mirrors provide rsync so it would be nice to be able to
use rsync as another option alongside http and ftp.

I wasn't aware that librsync used a different protocol so that
suggestion is obviously no good.

cgf

Re: setup RFC: Ditch homegrown http/ftp code and use a library?

[John Morrison]

> John Morrison wrote:
>> If you are worried about the size of setup, would it help to move
>> /cygwin.bat and /cygwin.ico into base-files?  I've an update going out
>> for
>> this soon and it would be little/no trouble...  (I could even rename
>> them
>> to have a captial 'C' so they don't clash with /cygdrive ;)
>
> It would probably only save a few KB.
> However, I think base-files would be a far more appropriate place for
> these
> to live. They are used in setup.exe's shortcut creation, though, so I
> think
> we should delay the move until we can transfer the shortcut creation to
> base-files too.

I can add that too, mkshortcut from the cygutils package in a postinstall.
 The only issue would be how to get setup to communicate which (if any)
should be created and wether for all users or just current... environment
variables?

J.

Re: setup RFC: Ditch homegrown http/ftp code and use a library?

[Robert Collins]

On Sat, 2004-11-13 at 12:11 +0100, Lapo Luchini wrote:
> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA1
> 
> Robert Collins wrote:
> 
> > Calculating which file to use as a basis for rsync is easy without
> > symlinks: Its just the 'closest to the version being downloaded'.
> > Define closest as you would intuitively: two package files with the
> > same x.y.z components are closer than two with the same x.y and
> > different z.
> 
> In my idea symlinks were not used to "select" the version, just to let
> rsync think the file is already present locally, and with old file"
> content, so that its content is used for optimization.

Thats my point : rsync doesn't require the same file name to use the
content for optimisation. You tell rsync what file to use as the basis,
and what file to write to.

Cheers,
Rob



signature.asc
Description: This is a digitally signed message part

Re: setup RFC: Ditch homegrown http/ftp code and use a library?

[Lapo Luchini]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Robert Collins wrote:

> Calculating which file to use as a basis for rsync is easy without
> symlinks: Its just the 'closest to the version being downloaded'.
> Define closest as you would intuitively: two package files with the
> same x.y.z components are closer than two with the same x.y and
> different z.

In my idea symlinks were not used to "select" the version, just to let
rsync think the file is already present locally, and with old file"
content, so that its content is used for optimization.

> As for using rsync on package files, there is strong evidence from
> the debian project that it is marginal, unless you have the zlib
> with the patched hunk-reset which is much more likey to keep
> consistent data in successive minor releases of a package. For bz2,
> I don't know for sure, but I suspect a similar issue will apply.

You're perfectly right. Moreover with package "method 2" this won't
EVEN suffice as 90% of the packaghe is the "internal gz/bz2", which is
made upstream and we have no control upon it at all.

All in all, rsync is maybe really NOT useful, in its present state, to
setup.exe.
Except as a tool to "help" sync the WHOLE packages system, but not for
the rsync-compression, more for the rsync management-of-file&date...

Lapo

- --
Lapo Luchini
[EMAIL PROTECTED] (PGP & X.509 keys available)
http://www.lapo.it (ICQ UIN: 529796)
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.2.4 (Cygwin)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iEYEARECAAYFAkGV6+wACgkQaJiCLMjyUvveBwCfevmf93EOdRr74ljAn+h/1NEM
Qz8An2NrHVBN0A4BNbUrqRV1zQQfBw2T
=Mrja
-END PGP SIGNATURE-

Re: setup RFC: Ditch homegrown http/ftp code and use a library?

[Robert Collins]

On Sat, 2004-11-13 at 11:57 +0100, Lapo Luchini wrote:
> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA1
> 
> Max Bowsher wrote:
> 
> > * Package file scarcely ever change whilst keeping the same name.
> > Also, they are compressed, so a small change is likely to cause the
> > entire contents to change anyway. Therefore, rsync would not help.
> 
> It could eb somewhat used anyways just doing a "ln -s" of the old file
> on the new name, THEN rsyncing it... I shoudl try but in theory it
> should read it for calculating the diff, then if writes a NEW file
> with the transfer and delete the old, so it would both have the "pro"
> of rsync and the "pro" of not copy the ACTUAL file (creating all the
> symlinks for every "newer" package in setup.ini is not so much work).
> 
> In fact, I think I will try and do such a bash script to do this, to
> "test" the idea.
> Maybe only for packages ALREADY locally present.

Calculating which file to use as a basis for rsync is easy without
symlinks: Its just the 'closest to the version being downloaded'. Define
closest as you would intuitively: two package files with the same x.y.z
components are closer than two with the same x.y and different z. 

As for using rsync on package files, there is strong evidence from the
debian project that it is marginal, unless you have the zlib with the
patched hunk-reset which is much more likey to keep consistent data in
successive minor releases of a package. For bz2, I don't know for sure,
but I suspect a similar issue will apply.

Rob 



signature.asc
Description: This is a digitally signed message part

Re: setup RFC: Ditch homegrown http/ftp code and use a library?

[Lapo Luchini]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Max Bowsher wrote:

> * Package file scarcely ever change whilst keeping the same name.
> Also, they are compressed, so a small change is likely to cause the
> entire contents to change anyway. Therefore, rsync would not help.

It could eb somewhat used anyways just doing a "ln -s" of the old file
on the new name, THEN rsyncing it... I shoudl try but in theory it
should read it for calculating the diff, then if writes a NEW file
with the transfer and delete the old, so it would both have the "pro"
of rsync and the "pro" of not copy the ACTUAL file (creating all the
symlinks for every "newer" package in setup.ini is not so much work).

In fact, I think I will try and do such a bash script to do this, to
"test" the idea.
Maybe only for packages ALREADY locally present.

Lapo

- --
Lapo Luchini
[EMAIL PROTECTED] (PGP & X.509 keys available)
http://www.lapo.it (ICQ UIN: 529796)
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.2.4 (Cygwin)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iEYEARECAAYFAkGV6IkACgkQaJiCLMjyUvu3PgCeI51AcZKvkfBhuRTC7WMv/is1
tcsAoKl4VYyMmPgAjcpJLG3FEx75aW0H
=qWrz
-END PGP SIGNATURE-

Re: setup RFC: Ditch homegrown http/ftp code and use a library?

[Max Bowsher]

John Morrison wrote:
If you are worried about the size of setup, would it help to move
/cygwin.bat and /cygwin.ico into base-files?  I've an update going out for
this soon and it would be little/no trouble...  (I could even rename them
to have a captial 'C' so they don't clash with /cygdrive ;)
It would probably only save a few KB.
However, I think base-files would be a far more appropriate place for these 
to live. They are used in setup.exe's shortcut creation, though, so I think 
we should delay the move until we can transfer the shortcut creation to 
base-files too.

Max.

Re: setup RFC: Ditch homegrown http/ftp code and use a library?

[Max Bowsher]

Gerrit P. Haase wrote:
Is it already compressed (ie. upx executable compressor)?
Yes, it is. Works wonders, too - the compression ratio is 36%.
Max.

Re: setup RFC: Ditch homegrown http/ftp code and use a library?

[Max Bowsher]

Christopher Faylor wrote:
It would be nice if setup.exe supported the rsync protocol.
Lapo and Brian have already covered this, so I'll just summarise:
* No library available implementing the rsync protocol
 (librsync implements a *different* protocol !!!)
* Package file scarcely ever change whilst keeping the same name.
 Also, they are compressed, so a small change is likely to cause
 the entire contents to change anyway. Therefore, rsync would not help.
Would it help if setup.exe was broken down into two different programs?
One for accessing the network and another for manipulating packages?
Why would it? I think it would be unnecessary complexity. All we need to
do is pick an appropriately compact ftp/http lib (and it's beginning to
look like libcurl is appropriately compact) and use it.
Max.

Re: setup RFC: Ditch homegrown http/ftp code and use a library?

[Max Bowsher]

Brian Dessent wrote:
Max Bowsher wrote:
Indeed, libcurl was one which came to mind.
I've yet to play with it, but one thing I'm a little worried about is it
being *too* extensive - I don't want setup.exe to grow very much.
I'm certainly going to look into libcurl, but I'm open to other
possibilities too.
Sure, that's a concern.  However current setup.exe is only ~270kb when
stripped, and even if that, say, doubled I don't see what the big deal
would be.  It seems to me that being anorexic about the .exe size at the
cost of robust net/url code is a premature optimization.
As a test I grabbed the stock 7.12.2 sources and compiled a static
libcurl.a with everything but http and ftp support disabled.  Before
stripping the file is 2.0M, after stripping it's 129k.  The
corresponding curl.exe is 163k after strip.  The only external lib that
this .exe depends on is zlib, which is in setup already, no?
Thankyou, those numbers make me feel rather more optimistic about using 
libcurl in setup.

Max.

Re: setup RFC: Ditch homegrown http/ftp code and use a library?

[Brian Dessent]

Lapo Luchini wrote:

> >It would be nice if setup.exe supported the rsync protocol.  That would
> >also give you the ability to suspend and resume.  I don't know how easy
> >it would be to get librsync working in a mingw environment, though.
>
> This may not be really "up-to-date" but rsync use is still "your life is
> so much better if you spawn an actual rsync program to do the work",
> e.g. rsync authors themself "suggest" not to use librsync.

Plus, with ftp resume and http byte-range support from libcurl (or
equiv) you can achieve almost the same thing.  If you have a local
.tar.bz2 file of the same name but smaller size from a prior aborted
download, just resume the download a few kbytes prior to the tail of the
file that you have, and if that rollback amount matches up then just
continue the download to the end.  And in any case you've always got the
md5sum so worst case you just redownload the package.

The case that rsync would be nice is where the package files are updated
on the mirror but the filename/version not bumped.  However that seems
to happen so infrequently (if ever) that it's probably not a big deal.

(And of course, you can always use rsync out-of-band.  For example, you
can keep a local package dir rsync'd from mirrors.kernel.org or another
mirror site that runs rsync and then run setup on the local dir.)

Brian

Re: setup RFC: Ditch homegrown http/ftp code and use a library?

[John Morrison]

Max Bowsher wrote:
> Brian Dessent wrote:
>> Max Bowsher wrote:
>>
>>> So, I'm looking for comments, and suggestions for candidate http/ftp
>>> client libraries to investigate.
>>
>> What's wrong with libcurl?  It has extensive support for all kinds of
>> extended http/ftp features, see for example
>> , and is MIT-licensed so
>> should be no problem statically linking it into setup.exe.
>
> Indeed, libcurl was one which came to mind.
> I've yet to play with it, but one thing I'm a little worried about is it
> being *too* extensive - I don't want setup.exe to grow very much.

If you are worried about the size of setup, would it help to move
/cygwin.bat and /cygwin.ico into base-files?  I've an update going out for
this soon and it would be little/no trouble...  (I could even rename them
to have a captial 'C' so they don't clash with /cygdrive ;)

J.

Re: setup RFC: Ditch homegrown http/ftp code and use a library?

[Lapo Luchini]

Christopher Faylor wrote:

>It would be nice if setup.exe supported the rsync protocol.  That would
>also give you the ability to suspend and resume.  I don't know how easy
>it would be to get librsync working in a mingw environment, though.
>  
>
This may not be really "up-to-date" but rsync use is still "your life is
so much better if you spawn an actual rsync program to do the work",
e.g. rsync authors themself "suggest" not to use librsync.
I asked this very question some time ago because I was ionterested ni
adding rsync support to Mozilla (it would be oh so great...), then I
tries using protozailla (to actualyl spawn a rsync exe) but protozilla
kinda died, it seems.

Extract from http://librsync.sourceforge.net/
"librsync is not wire-compatible with rsync 2.x, and is not likely to be
in the future. This is a new codebase, designed to allow a fresh start
on the problem and a cleaner design. It may be used in a future version
of rsync or a successor project."

>Would it help if setup.exe was broken down into two different programs?
>One for accessing the network and another for manipulating packages?
>
Updating local "mirror" with rsync would be certainly possible, but only
after the base system is already set up, as there is no "rsync-on-mingw"
visible even on the fartest horizon, right now..

-- 
Lapo Luchini
[EMAIL PROTECTED] (PGP & X.509 keys available)
http://www.lapo.it (ICQ UIN: 529796)

Re: setup RFC: Ditch homegrown http/ftp code and use a library?

[Gerrit P. Haase]

Brian Dessent wrote:
Max Bowsher wrote:
Indeed, libcurl was one which came to mind.
I've yet to play with it, but one thing I'm a little worried about is it
being *too* extensive - I don't want setup.exe to grow very much.
Sure, that's a concern.  However current setup.exe is only ~270kb when
stripped, and even if that, say, doubled I don't see what the big deal
would be.  It seems to me that being anorexic about the .exe size at the
cost of robust net/url code is a premature optimization.
Is it already compressed (ie. upx executable compressor)?  It reduces 
size and works stand-alone with built-in decompressor.

Gerrit
--
=^..^=