Re: Complaining after fair notice of dropping XP support. Was: Not very nice at all.

[Andrey Repin]

Greetings, Erik Soderquist!

> The automation details are primarily what I'm after, though it could
> have been something like ZFS with deduplication turned on

You can, indeed, achieve similar level of magic of a general purpose savings
with ZFS or BTRFS.
I'm using BTRFS snapshots to keep intermediate copies of a working partition.
It's nearly 400GB of data, and I keep ~10 copies of it (a week worth of daily
plus 3 for month of weekend copies). All on ~800GB of RAID6 space.

> I'm wondering if I could apply your automations to a few background
> projects I'm working on

> Would also be interesting to see how cleanly the existing setup could
> be mirrored knowing the details of the automation... rsync would
> happily copy the symlinks, and once created, a Time Machine mirror
> *should* be able to "stay in sync" with the original time machine by
> pulling from the cygwin main sources in the same manner.

I can imagine so.


-- 
With best regards,
Andrey Repin
Friday, November 11, 2016 17:31:48

Sorry for my terrible english...


--
Problem reports:   http://cygwin.com/problems.html
FAQ:   http://cygwin.com/faq/
Documentation: http://cygwin.com/docs.html
Unsubscribe info:  http://cygwin.com/ml/#unsubscribe-simple

Re: Complaining after fair notice of dropping XP support. Was: Not very nice at all.

[Erik Soderquist]

On Thu, Nov 10, 2016 at 7:38 PM, Peter A. Castro wrote:
> Greetings, Erik,
>
> A good magician never reveals how the trick works.  8^)
>
> Sadly I'm not a good magician.
>
> It's not all that complicated, if you think about it.  Each time a new
> setup.ini is generated only a hand full of packages was actually updated, so
> really, day-to-day, there's not all that much change.
>
> I keep a delta database and each time I grab a new setup.ini I compare all
> of the packages listed to my existing database.  Anything new, I pull down
> and add to the archive.  Anything already present I don't re-pull.  Think of
> it as de-duplication on a package level (though I do this for more than just
> setup.ini, but that's a separate trick).
>
> It's only if you decide you want a complete copy of all packages for any
> given circa that the amount of data instantiating is alot.
>
> Then, from the setup.ini, I create a new circa directory and, really, the
> smoke-and-mirrors of the trick is that it's all symlinks to the real package
> storage, of which I have exactly one copy.  Hence, 500Gb instead of 535Tb.
>
> So, um, "Ta-da"! :-)
>
> Oh, this is all automated, btw.  I hardly touch it except to clean out old
> logs and do backups from time to time.
>
> Sorry, it wasn't all that good a trick, was it.

The automation details are primarily what I'm after, though it could
have been something like ZFS with deduplication turned on

I'm wondering if I could apply your automations to a few background
projects I'm working on

Would also be interesting to see how cleanly the existing setup could
be mirrored knowing the details of the automation... rsync would
happily copy the symlinks, and once created, a Time Machine mirror
*should* be able to "stay in sync" with the original time machine by
pulling from the cygwin main sources in the same manner.



-- Erik

--
Problem reports:   http://cygwin.com/problems.html
FAQ:   http://cygwin.com/faq/
Documentation: http://cygwin.com/docs.html
Unsubscribe info:  http://cygwin.com/ml/#unsubscribe-simple

Re: Complaining after fair notice of dropping XP support. Was: Not very nice at all.

[Peter A. Castro]

On Thu, 10 Nov 2016, Erik Soderquist wrote:


Date: Thu, 10 Nov 2016 17:16:53 -0500
From: Erik Soderquist 
To: cygwin@cygwin.com
Subject: Re: Complaining after fair notice of dropping XP support. Was: Not
very nice at all.


Greetings, Erik,


On Thu, Nov 10, 2016 at 2:19 PM, Peter A. Castro wrote:

And for those who think keeping a private mirror is trivial, let me give you
some stats.  A full snapshot of Cygwin, 32-bit + 64-bit (+noarch) + source
packages (Current and Previous) is about 137Gb.  Don't believe me? Your
setup.ini has a size (and a hash) for every package.  Add them up yourself.
Now consider that I have about 4000 snapshots (and counting!) of Cygwin.
Oh!  Quick quiz:
  What's 137Gb * 4000?
  Answer: 535Tb.
Now, do you really think I have that amount of storage?  Of course not. The
Time Machine isn't organized like that (don't be silly).  It's currently
about 500Gb (which is quite a savings, if you think about it :-)


I would dearly love to know more about how you did this specific piece...


A good magician never reveals how the trick works.  8^)

Sadly I'm not a good magician.

It's not all that complicated, if you think about it.  Each time a new 
setup.ini is generated only a hand full of packages was actually updated, 
so really, day-to-day, there's not all that much change.


I keep a delta database and each time I grab a new setup.ini I compare all 
of the packages listed to my existing database.  Anything new, I pull down 
and add to the archive.  Anything already present I don't re-pull.  Think 
of it as de-duplication on a package level (though I do this for more 
than just setup.ini, but that's a separate trick).


It's only if you decide you want a complete copy of all packages for any
given circa that the amount of data instantiating is alot.

Then, from the setup.ini, I create a new circa directory and, really, the 
smoke-and-mirrors of the trick is that it's all symlinks to the real 
package storage, of which I have exactly one copy.  Hence, 500Gb 
instead of 535Tb.


So, um, "Ta-da"! :-)

Oh, this is all automated, btw.  I hardly touch it except to clean out 
old logs and do backups from time to time.


Sorry, it wasn't all that good a trick, was it.


-- Erik


--
--=> Peter A. Castro
Email: doctor at fruitbat dot org / Peter dot Castro at oracle dot com
"Cats are just autistic Dogs" -- Dr. Tony Attwood

--
Problem reports:   http://cygwin.com/problems.html
FAQ:   http://cygwin.com/faq/
Documentation: http://cygwin.com/docs.html
Unsubscribe info:  http://cygwin.com/ml/#unsubscribe-simple

Re: Complaining after fair notice of dropping XP support. Was: Not very nice at all.

[Erik Soderquist]

On Thu, Nov 10, 2016 at 2:19 PM, Peter A. Castro wrote:
> And for those who think keeping a private mirror is trivial, let me give you
> some stats.  A full snapshot of Cygwin, 32-bit + 64-bit (+noarch) + source
> packages (Current and Previous) is about 137Gb.  Don't believe me? Your
> setup.ini has a size (and a hash) for every package.  Add them up yourself.
> Now consider that I have about 4000 snapshots (and counting!) of Cygwin.
> Oh!  Quick quiz:
>   What's 137Gb * 4000?
>   Answer: 535Tb.
> Now, do you really think I have that amount of storage?  Of course not. The
> Time Machine isn't organized like that (don't be silly).  It's currently
> about 500Gb (which is quite a savings, if you think about it :-)


I would dearly love to know more about how you did this specific piece...

-- Erik

--
Problem reports:   http://cygwin.com/problems.html
FAQ:   http://cygwin.com/faq/
Documentation: http://cygwin.com/docs.html
Unsubscribe info:  http://cygwin.com/ml/#unsubscribe-simple

Re: Complaining after fair notice of dropping XP support. Was: Not very nice at all.

[Peter A. Castro]

On Thu, 10 Nov 2016, Brian Inglis wrote:


Date: Thu, 10 Nov 2016 02:45:32 -0700
From: Brian Inglis
Subject: Re: Complaining after fair notice of dropping XP support. Was: Not
very nice at all.

On 2016-11-10 00:24, Fergus wrote:

1. Use the following version of setup*.exe:
32-bit:

ftp://www.fruitbat.org/pub/cygwin/setup/snapshots/setup-x86-2.874.exe

2. Run setup*.exe with the -X option, using the following
mirror:
32-bit:
ftp://www.fruitbat.org/pub/cygwin/circa/2016/08/30/104223

Q1  Any ideas of what might be de-railing this simple operation?


See notice on throttling Cygwin Time Machine because of abuse:
http://www.fruitbat.org/Cygwin/timemachine.html
advises using only setup on as few required packages as possible
at a time, until abuse stops.


Greetings, All,
  (Time Machine owner/operator here! :-)


Q2  [... virtual(?);] Could one instead use wget on
ftp://www.fruitbat.org/pub/cygwin/circa/2016/08/30/104223?


Brian is correct about robots and LIST, but what he (anyone) doesn't know 
is why.  I generally keep to myself and don't post non-Cygwin specific 
things to this list so I've never really explain why I do what I do. 
And, no, it's not "BWJM" (https://cygwin.com/acronyms/#BWAM) either. :)



Wget honours robots.txt, and LIST has been disabled for Cygwin
Time Machine directories, so you can not even see the
directories, and no FTP download requests other than GET will
work.

https://www.google.ca/search?q=lack+of+planning+on+your+part+does+not+constitute+an+emergency=imghp=isch=u=univ=X

Please remember that both the Cygwin and Time Machine
projects are volunteer efforts with few resources.
Each must manage their projects as best they can fit effort
into their time, and may withdraw their efforts and
facilities at any time.


While it is true I have limited resources, it's unlikely I'll ever 
withdraw the Time Machine.  I've been doing this for over 10 years and 
will continue so until Windows finally becomes Skynet.  :-)  At that point 
I figure we'll all be dead or used as living batteries, so it won't 
matter.



Following the mailing lists/groups and participation earlier
could perhaps have delayed the final implementation to allow
more time for people to plan and execute final downloads.

Complaining after advance notice was given publicly and making
could/should have suggestions after the fact is pointless, and
could demotivate the volunteers to drop the projects, make
access private, or charge for it.

Fairly recently, the owner of gmane.org, that many of us had
used to browse and post to mailing lists via the web, shut
down his web site, although he kept his mail-news gateway up.
How many people over the years expressed appreciation for his
free service to the community? Or were gracious when letting
him know his site had a problem. And thanked him when it was
fixed. Not enough probably!

If you, a bunch of Cygwin XP dependents banded together, or
your company  have the space, bandwidth, server(s), money to
provide a public mirror of the final Cygwin XP release from
the Time Machine, you could contact the owner and make a
proposal to offload his site, or upgrade his facilities and
help run them.


Ugh.  Please don't.  I already have plans in motion to move the Time 
Machine to a service with greater network bandwidth and I'm really not 
willing to consider other options.  In the process of that I'll be 
restricting the existing site even further to not-so-subtly prod existing 
users over to the new system, when it is live.  People wanting to make 
their own special, private copy really should wait, please, pretty please.


And for those who think keeping a private mirror is trivial, let me give 
you some stats.  A full snapshot of Cygwin, 32-bit + 64-bit (+noarch) + 
source packages (Current and Previous) is about 137Gb.  Don't believe me? 
Your setup.ini has a size (and a hash) for every package.  Add them up 
yourself.  Now consider that I have about 4000 snapshots (and counting!) 
of Cygwin.

Oh!  Quick quiz:
  What's 137Gb * 4000?
  Answer: 535Tb.
Now, do you really think I have that amount of storage?  Of course not. 
The Time Machine isn't organized like that (don't be silly).  It's 
currently about 500Gb (which is quite a savings, if you think about it 
:-).  It is for this reason I won't let webcrawlers on the site.  They 
would see each "virtual" slice as wholely separate and attempt to pull it 
all.  That is what was happening before I disabled FTP LIST.



Or your company could ask MS and Redhat for XP and Cygwin XP
support quotes if it has the money ;^>


I see what you did there.  Ha Ha. :)


There's an opportunity for some of you XP folks to make
money off the others by providing dedicated repos of
outdated software with support ;^>

If you are working for a company that decided to build products
for XP dependent on Cygwin, maybe it's time to tell them that
Cygwin has reached EOL on the EOL XP.
If you are supporting Cygwin based products on XP, maybe it's

Re: Complaining after fair notice of dropping XP support. Was: Not very nice at all.

[Corinna Vinschen]

On Nov 10 02:45, Brian Inglis wrote:
> On 2016-11-10 00:24, Fergus wrote:
> > > > 1. Use the following version of setup*.exe:
> > > > 32-bit:
> > ftp://www.fruitbat.org/pub/cygwin/setup/snapshots/setup-x86-2.874.exe
> > > > 2. Run setup*.exe with the -X option, using the following
> > > > mirror:
> > > > 32-bit:
> > > > ftp://www.fruitbat.org/pub/cygwin/circa/2016/08/30/104223
> > Q1  Any ideas of what might be de-railing this simple operation?
> 
> See notice on throttling Cygwin Time Machine because of abuse:
> http://www.fruitbat.org/Cygwin/timemachine.html
> advises using only setup on as few required packages as possible
> at a time, until abuse stops.
> 
> > Q2  [... virtual(?);] Could one instead use wget on
> > ftp://www.fruitbat.org/pub/cygwin/circa/2016/08/30/104223?
> 
> Wget honours robots.txt, and LIST has been disabled for Cygwin
> Time Machine directories, so you can not even see the
> directories, and no FTP download requests other than GET will
> work.
> 
> https://www.google.ca/search?q=lack+of+planning+on+your+part+does+not+constitute+an+emergency=imghp=isch=u=univ=X
> 
> Please remember that both the Cygwin and Time Machine
> projects are volunteer efforts with few resources.
> Each must manage their projects as best they can fit effort
> into their time, and may withdraw their efforts and
> facilities at any time.
> 
> Following the mailing lists/groups and participation earlier
> could perhaps have delayed the final implementation to allow
> more time for people to plan and execute final downloads.
> 
> Complaining after advance notice was given publicly and making
> could/should have suggestions after the fact is pointless, and
> could demotivate the volunteers to drop the projects, make
> access private, or charge for it.
> 
> Fairly recently, the owner of gmane.org, that many of us had
> used to browse and post to mailing lists via the web, shut
> down his web site, although he kept his mail-news gateway up.
> How many people over the years expressed appreciation for his
> free service to the community? Or were gracious when letting
> him know his site had a problem. And thanked him when it was
> fixed. Not enough probably!
> 
> If you, a bunch of Cygwin XP dependents banded together, or
> your company  have the space, bandwidth, server(s), money to
> provide a public mirror of the final Cygwin XP release from
> the Time Machine, you could contact the owner and make a
> proposal to offload his site, or upgrade his facilities and
> help run them.
> Or your company could ask MS and Redhat for XP and Cygwin XP
> support quotes if it has the money ;^>

https://cygwin.com/ml/cygwin/2016-11/msg00068.html

Red Hat does not provide Cygwin Support any longer since we hadn't
enough support customers to get the revenue supporting the business
model.  Apparently too many people were happy with upstream as is.

As Stephen wrote, Cygwin is a volunteer driven project now.  One of the
reasons we relaxed the license was to make the project easier accessible
and maybe get more volunteers fixing bugs and stuff.  But then again,
Cygwin isn't Linux so it's not as sexy by far.

> There's an opportunity for some of you XP folks to make
> money off the others by providing dedicated repos of
> outdated software with support ;^>
> 
> If you are working for a company that decided to build products
> for XP dependent on Cygwin, maybe it's time to tell them that
> Cygwin has reached EOL on the EOL XP.
> If you are supporting Cygwin based products on XP, maybe it's
> time to tell your customers you can't any longer, as both are
> unsupported.
> 
> You can download source and binary packages that you need that
> predate 2.6 from the Cygwin mirrors, and include all the build
> dependencies, starting with cygport.
> That will enable you to use setup to download from the Time
> Machine only those packages recently updated on Cygwin
> mirrors to use 2.6.
> You could run setup unattended installing packages one at a
> time, in a loop driven by the packages needed from
> installed.db, to honour the site owner's request.
> You will then be in a position to monitor upstream sources,
> so you can download new upstream patches and releases as
> they become available, so you have and can apply them when
> needed, to rebuild the updated packages.
> 
> You could also try upgrading to W10 and working with single
> user Ubuntu under WSL ;^>

Interesting point, but where's the fun in that from a dev perspective :)


Corinna

-- 
Corinna Vinschen  Please, send mails regarding Cygwin to
Cygwin Maintainer cygwin AT cygwin DOT com
Red Hat


signature.asc
Description: PGP signature

Re: Complaining after fair notice of dropping XP support. Was: Not very nice at all.

[Brian Inglis]

On 2016-11-10 00:24, Fergus wrote:

1. Use the following version of setup*.exe:
32-bit:

ftp://www.fruitbat.org/pub/cygwin/setup/snapshots/setup-x86-2.874.exe

2. Run setup*.exe with the -X option, using the following
mirror:
32-bit:
ftp://www.fruitbat.org/pub/cygwin/circa/2016/08/30/104223

Q1  Any ideas of what might be de-railing this simple operation?


See notice on throttling Cygwin Time Machine because of abuse:
http://www.fruitbat.org/Cygwin/timemachine.html
advises using only setup on as few required packages as possible
at a time, until abuse stops.


Q2  [... virtual(?);] Could one instead use wget on
ftp://www.fruitbat.org/pub/cygwin/circa/2016/08/30/104223?


Wget honours robots.txt, and LIST has been disabled for Cygwin
Time Machine directories, so you can not even see the
directories, and no FTP download requests other than GET will
work.

https://www.google.ca/search?q=lack+of+planning+on+your+part+does+not+constitute+an+emergency=imghp=isch=u=univ=X

Please remember that both the Cygwin and Time Machine
projects are volunteer efforts with few resources.
Each must manage their projects as best they can fit effort
into their time, and may withdraw their efforts and
facilities at any time.

Following the mailing lists/groups and participation earlier
could perhaps have delayed the final implementation to allow
more time for people to plan and execute final downloads.

Complaining after advance notice was given publicly and making
could/should have suggestions after the fact is pointless, and
could demotivate the volunteers to drop the projects, make
access private, or charge for it.

Fairly recently, the owner of gmane.org, that many of us had
used to browse and post to mailing lists via the web, shut
down his web site, although he kept his mail-news gateway up.
How many people over the years expressed appreciation for his
free service to the community? Or were gracious when letting
him know his site had a problem. And thanked him when it was
fixed. Not enough probably!

If you, a bunch of Cygwin XP dependents banded together, or
your company  have the space, bandwidth, server(s), money to
provide a public mirror of the final Cygwin XP release from
the Time Machine, you could contact the owner and make a
proposal to offload his site, or upgrade his facilities and
help run them.
Or your company could ask MS and Redhat for XP and Cygwin XP
support quotes if it has the money ;^>

There's an opportunity for some of you XP folks to make
money off the others by providing dedicated repos of
outdated software with support ;^>

If you are working for a company that decided to build products
for XP dependent on Cygwin, maybe it's time to tell them that
Cygwin has reached EOL on the EOL XP.
If you are supporting Cygwin based products on XP, maybe it's
time to tell your customers you can't any longer, as both are
unsupported.

You can download source and binary packages that you need that
predate 2.6 from the Cygwin mirrors, and include all the build
dependencies, starting with cygport.
That will enable you to use setup to download from the Time
Machine only those packages recently updated on Cygwin
mirrors to use 2.6.
You could run setup unattended installing packages one at a
time, in a loop driven by the packages needed from
installed.db, to honour the site owner's request.
You will then be in a position to monitor upstream sources,
so you can download new upstream patches and releases as
they become available, so you have and can apply them when
needed, to rebuild the updated packages.

You could also try upgrading to W10 and working with single
user Ubuntu under WSL ;^>

--
Take care. Thanks, Brian Inglis, Calgary, Alberta, Canada

--
Problem reports:   http://cygwin.com/problems.html
FAQ:   http://cygwin.com/faq/
Documentation: http://cygwin.com/docs.html
Unsubscribe info:  http://cygwin.com/ml/#unsubscribe-simple