Re: svn.haxx.se is going away

[Nathan Hartman]

On Tue, Nov 24, 2020 at 8:04 PM Daniel Shahaf 
wrote:

> Nathan Hartman wrote on Tue, 24 Nov 2020 21:27 +00:00:
> > On Tue, Nov 24, 2020 at 2:56 AM Daniel Sahlberg
> >  wrote:
> > > Den tors 12 nov. 2020 kl 17:46 skrev Daniel Sahlberg <
> daniel.l.sahlb...@gmail.com>:
> > >> Could ASF provide this server space (basically a VirtualHost)? The
> archive is about 6.5 GB so it is not a huge amount.
> > >
> > > Any thoughts on this?
> >
> > I am looking into this; waiting for a reply...
>
> In the circumstances — it's Nov 25 and the site says it'll be taken down
> "in November 2020", not specifying a date — I'd say, better ask
> forgiveness than permission.

Let's go ahead and grab all the data we
> need to stand up the site (we have the mboxes, but not the mapping of
> *.shtml files to message-id's, nor any of the HTML/CSS/images), and if
> possible, also set it up (on svn-qavm.a.o or wherever) to ensure we've
> got everything and to prepare for a DNS repointing, if Daniel agrees.
> We can figure out the "paperwork", Puppet PRs, etc., later.


Just FYI it looks like yes, we will get the server space, but I don't know
details yet. The 1st order of business is to save the data...

@Daniel Sahlberg since you've previously reached out to the operator of
svn.haxx.se about saving the site, could you perhaps ask for a way to
download the data efficiently?

Nathan

Re: svn.haxx.se is going away

[Daniel Shahaf]

Nathan Hartman wrote on Tue, 24 Nov 2020 21:27 +00:00:
> On Tue, Nov 24, 2020 at 2:56 AM Daniel Sahlberg 
>  wrote:
> > Den tors 12 nov. 2020 kl 17:46 skrev Daniel Sahlberg 
> > :
> >> Could ASF provide this server space (basically a VirtualHost)? The archive 
> >> is about 6.5 GB so it is not a huge amount.
> > 
> > Any thoughts on this?
> 
> I am looking into this; waiting for a reply...

In the circumstances — it's Nov 25 and the site says it'll be taken down
"in November 2020", not specifying a date — I'd say, better ask
forgiveness than permission.  Let's go ahead and grab all the data we
need to stand up the site (we have the mboxes, but not the mapping of
*.shtml files to message-id's, nor any of the HTML/CSS/images), and if
possible, also set it up (on svn-qavm.a.o or wherever) to ensure we've
got everything and to prepare for a DNS repointing, if Daniel agrees.
We can figure out the "paperwork", Puppet PRs, etc., later.

I'd say the highest priority is to save the mapping of .shtml URLs to
message-id's (which are available as comments in the source HTML),
whether via a recursive wget(1) invocation, or by asking Daniel to run
an appropriate grep, or however else.  Without that info, we won't be
able to preserve old URLs.

Maybe there's also a button we can press to sic the archive.org spider
on svn.haxx.se.

(We can't derive the message<->.shtml mapping from the mboxes we have.
I only grabbed mboxes through the transition to ASF; for anything after
that point, the order of .shtml files would be the order in which list
mails reached haxx.se's MX, and we have no backups of that info.)

Cheers,

Daniel

P.S.  Yes, it's a bit https://m.xkcd.com/2337/ of me to refer to both
  Daniel and Daniel as "Daniel". :)

Re: svn.haxx.se is going away

[Nathan Hartman]

On Tue, Nov 24, 2020 at 2:56 AM Daniel Sahlberg 
wrote:

> Den tors 12 nov. 2020 kl 17:46 skrev Daniel Sahlberg <
> daniel.l.sahlb...@gmail.com>:
>
>> Could ASF provide this server space (basically a VirtualHost)? The
>> archive is about 6.5 GB so it is not a huge amount.
>>
>
> Any thoughts on this?
>

I am looking into this; waiting for a reply...

Thanks for the nudge.

Nathan

Re: svn.haxx.se is going away

[Daniel Sahlberg]

Den tors 12 nov. 2020 kl 17:46 skrev Daniel Sahlberg <
daniel.l.sahlb...@gmail.com>:

> Could ASF provide this server space (basically a VirtualHost)? The archive
> is about 6.5 GB so it is not a huge amount.
>

Any thoughts on this?

Re: svn.haxx.se is going away

[Daniel Sahlberg]

Den tors 5 nov. 2020 kl 15:31 skrev Julian Foad :

> Main point: Thanks to everyone helping this preservation effort.
>
> > * updating the 63+87 links in the site and source to point to links
> hosted on ASF hardware
> >
> Observation: s/hardware/domain/. While the ASF has long promoted "on our
> own hardware", the more critical and often under-valued key to keeping
> control of one's Internet assets is "on our own domain name". That's
> assumed in this context, but something to keep in mind elsewhere.
>

Agreeing with Julian's point on "on our own domain name", however this is
as it is. If we can get an agreement regarding keeping svn.haxx.se pointing
to a server where, at least, the old mailing list archive is available then
we would be better off.

Could ASF provide this server space (basically a VirtualHost)? The archive
is about 6.5 GB so it is not a huge amount.

Kind regards,
Daniel

Re: svn.haxx.se is going away

[Julian Foad]

Main point: Thanks to everyone helping this preservation effort.

> * updating the 63+87 links in the site and source to point to links hosted on 
> ASF hardware
> 
Observation: s/hardware/domain/. While the ASF has long promoted "on our own 
hardware", the more critical and often under-valued key to keeping control of 
one's Internet assets is "on our own domain name". That's assumed in this 
context, but something to keep in mind elsewhere.

- Julian

Re: svn.haxx.se is going away

[Daniel Shahaf]

Daniel Sahlberg wrote on Thu, 05 Nov 2020 11:16 +0100:
> Den ons 4 nov. 2020 kl 22:32 skrev Nathan Hartman  >:  
> 
> > On Wed, Nov 4, 2020 at 3:32 PM Mark Phippard  wrote:  
> > >
> > > Just a general fyi ... I went to https://svn.haxx.se/ today to search  
> > the lists and noticed there is a banner on the site saying it is going
> > offline forever soon.  
> > >
> > > I am not sure what the ramifications will be as I know there are a lot  
> > of historical links in the docs and site but I guess it is what it is.
> >
> > Daniel (danielsh) has been trying to get Infra to import the material
> > from pre-2009 (pre-migration to ASF) into lists.apache.org to avoid
> > losing the archives from the earliest period of development, which
> > arguably contain some of the most important development information.
> >
> > See the discussion here:
> >
> > https://lists.apache.org/thread.html/r97c9c5208af706b067fd8e67a7cbe79b37255958bb087bf699b722f8%40%3Cdev.subversion.apache.org%3E
> >

And https://issues.apache.org/jira/browse/INFRA-20213

> > Possibly it's still mirrored at home.apache.org but I can't check at the
> > moment.
> >
> > Nathan
> >  
> 
> Would it be considered a good thing if we manage to keep svn.haxx.se
> around? Even if Infra would get the old lists imported (I don't know what's
> holding them back), there are a bunch of references to the archives in the
> source (63 if I'm counting correctly), and in the website (87).
> 

Those in the website should be covered by
site/publish/.message-ids.tsv.  (See site/tools/ for the generating
scripts.)

The logic for converting the message-ids into URLs is embedded in [1]
(which I have tried to make discoverable, [2], but that seems to have
regressed, and I'm ENOTIME to chase it).

[1] 
https://svn.apache.org/repos/infra/infrastructure/trunk/projects/asf-generate-mail-archives-link
[2] https://issues.apache.org/jira/browse/INFRA-19422

> I have reached out to Daniel Stenberg and he seems willing to discuss to
> point the domain name to another server. I could probably volunteer to keep
> the site alive, provided there is an agreement within @Dev this is a good
> thing. Or is it better to just do the job and update the sources and
> website?

We should keep old links working, if possible.  Ideally, not only links
we happen to have lying around, but also other links (e.g., in people's
non-public branches of https://github.com/apache/subversion).

There's more than one way to preserve links (redirecting old URLs to
new URLs for the same messages; keeping the site online but not
updating; keeping the site online and updating, on ASF hardware, e.g.,
svn-qavm.a.o; etc.).  Any and all assistance would be most welcome!

> (Daniel S... seems to be a popular name!)

It is, yes.  And then there are people like danderson, who aren't named
"Daniel" but still get in the way of tab-completing Daniels ☺

Cheers,

Daniel

Re: svn.haxx.se is going away

[Daniel Shahaf]

Nathan Hartman wrote on Wed, 04 Nov 2020 16:32 -0500:
> On Wed, Nov 4, 2020 at 3:32 PM Mark Phippard  wrote:
> >
> > Just a general fyi ... I went to https://svn.haxx.se/ today to search the 
> > lists and noticed there is a banner on the site saying it is going offline 
> > forever soon.
> >
> > I am not sure what the ramifications will be as I know there are a lot of 
> > historical links in the docs and site but I guess it is what it is.  
> 
> Daniel (danielsh) has been trying to get Infra to import the material
> from pre-2009 (pre-migration to ASF) into lists.apache.org to avoid
> losing the archives from the earliest period of development, which
> arguably contain some of the most important development information.
> 
> See the discussion here:
> https://lists.apache.org/thread.html/r97c9c5208af706b067fd8e67a7cbe79b37255958bb087bf699b722f8%40%3Cdev.subversion.apache.org%3E
> 
> Possibly it's still mirrored at home.apache.org but I can't check at the 
> moment.

It is —

% ssh home.apache.org du -hs /home/danielsh/svn-haxx-se-mirror
245M/home/danielsh/svn-haxx-se-mirror
% ssh svn-qavm.apache.org du -hs /x1/svn-haxx-se-mirror 
245M/x1/svn-haxx-se-mirror

— but I don't know that either of these is backed up, so please someone
rsync either of those [they're identical] to their own hardware.

Cheers,

Daniel

Re: svn.haxx.se is going away

[Nathan Hartman]

On Thu, Nov 5, 2020 at 5:16 AM Daniel Sahlberg 
wrote:

>
> Would it be considered a good thing if we manage to keep svn.haxx.se
> around?
>

Yes, I would consider that a good thing.

Even if Infra would get the old lists imported (I don't know what's holding
> them back), there are a bunch of references to the archives in the source
> (63 if I'm counting correctly), and in the website (87).
>

There are many more links in emails, in log messages, etc.

IIRC Infra said there's some software-related reason that holds them back
from importing the old material.

I have reached out to Daniel Stenberg and he seems willing to discuss to
> point the domain name to another server. I could probably volunteer to keep
> the site alive, provided there is an agreement within @Dev this is a good
> thing. Or is it better to just do the job and update the sources and
> website?
>

Thank you for reaching out.

It would be ideal if 3 things happen:

* keep svn.haxx.se alive to prevent breaking the myriad links that exist
out there

* getting the early years' SVN dev & users archives (2000-2009) onto ASF
hardware one way or another; if it can't/won't be backfilled to
lists.apache.org for whatever reasons, maybe it can be put on Subversion's
website

* updating the 63+87 links in the site and source to point to links hosted
on ASF hardware

svn.haxx.se also has archives for TSVN and Subclipse dev and users, which
is another reason to keep that site alive if possible.

Nathan

Re: svn.haxx.se is going away

[Daniel Sahlberg]

Den ons 4 nov. 2020 kl 22:32 skrev Nathan Hartman :

> On Wed, Nov 4, 2020 at 3:32 PM Mark Phippard  wrote:
> >
> > Just a general fyi ... I went to https://svn.haxx.se/ today to search
> the lists and noticed there is a banner on the site saying it is going
> offline forever soon.
> >
> > I am not sure what the ramifications will be as I know there are a lot
> of historical links in the docs and site but I guess it is what it is.
>
> Daniel (danielsh) has been trying to get Infra to import the material
> from pre-2009 (pre-migration to ASF) into lists.apache.org to avoid
> losing the archives from the earliest period of development, which
> arguably contain some of the most important development information.
>
> See the discussion here:
>
> https://lists.apache.org/thread.html/r97c9c5208af706b067fd8e67a7cbe79b37255958bb087bf699b722f8%40%3Cdev.subversion.apache.org%3E
>
> Possibly it's still mirrored at home.apache.org but I can't check at the
> moment.
>
> Nathan
>

Would it be considered a good thing if we manage to keep svn.haxx.se
around? Even if Infra would get the old lists imported (I don't know what's
holding them back), there are a bunch of references to the archives in the
source (63 if I'm counting correctly), and in the website (87).

I have reached out to Daniel Stenberg and he seems willing to discuss to
point the domain name to another server. I could probably volunteer to keep
the site alive, provided there is an agreement within @Dev this is a good
thing. Or is it better to just do the job and update the sources and
website?

Kind regards
Daniel Sahlberg

(Daniel S... seems to be a popular name!)

Re: svn.haxx.se is going away

[Mark Phippard]

On Wed, Nov 4, 2020 at 4:32 PM Nathan Hartman 
wrote:

> On Wed, Nov 4, 2020 at 3:32 PM Mark Phippard  wrote:
> >
> > Just a general fyi ... I went to https://svn.haxx.se/ today to search
> the lists and noticed there is a banner on the site saying it is going
> offline forever soon.
> >
> > I am not sure what the ramifications will be as I know there are a lot
> of historical links in the docs and site but I guess it is what it is.
>
> Daniel (danielsh) has been trying to get Infra to import the material
> from pre-2009 (pre-migration to ASF) into lists.apache.org to avoid
> losing the archives from the earliest period of development, which
> arguably contain some of the most important development information.
>
> See the discussion here:
>
> https://lists.apache.org/thread.html/r97c9c5208af706b067fd8e67a7cbe79b37255958bb087bf699b722f8%40%3Cdev.subversion.apache.org%3E
>
> Possibly it's still mirrored at home.apache.org but I can't check at the
> moment.
>

Thanks Nathan. I am glad to hear we were aware of this. I had not seen any
discussion so just wanted to make sure interested parties maybe had some
time to act before it is too late. I notice we have a search function as
part of out website that uses their search. Hopefully that can be adapted
to the Apache list archives though not sure if it will work as well. I have
always used svn.haxx.se just for the search.

-- 
Thanks

Mark Phippard

Re: svn.haxx.se is going away

[Nathan Hartman]

On Wed, Nov 4, 2020 at 3:32 PM Mark Phippard  wrote:
>
> Just a general fyi ... I went to https://svn.haxx.se/ today to search the 
> lists and noticed there is a banner on the site saying it is going offline 
> forever soon.
>
> I am not sure what the ramifications will be as I know there are a lot of 
> historical links in the docs and site but I guess it is what it is.

Daniel (danielsh) has been trying to get Infra to import the material
from pre-2009 (pre-migration to ASF) into lists.apache.org to avoid
losing the archives from the earliest period of development, which
arguably contain some of the most important development information.

See the discussion here:
https://lists.apache.org/thread.html/r97c9c5208af706b067fd8e67a7cbe79b37255958bb087bf699b722f8%40%3Cdev.subversion.apache.org%3E

Possibly it's still mirrored at home.apache.org but I can't check at the moment.

Nathan

Re: svn.haxx.se is going away

[David Chapman]

On 11/4/2020 12:32 PM, Mark Phippard wrote:
Just a general fyi ... I went to https://svn.haxx.se/ 
 today to search the lists and noticed there is 
a banner on the site saying it is going offline forever soon.


I am not sure what the ramifications will be as I know there are a lot 
of historical links in the docs and site but I guess it is what it is.


--
Thanks

Mark Phippard


Daniel Stenberg is in the process of moving the Curl Web site from 
https://curl.haxx.se/ to https://www.curl.se/.  I'm not sure why 
https://svn.haxx.se/ is not following along (there is no 
https://svn.curl.se as of a few minutes ago), but then again I'm just a 
Curl user, not a dev.  Curl development is hosted on Github, so maybe he 
lost interest in a Subversion archive?


--
David Chapman  dcchap...@acm.org
Chapman Consulting -- San Jose, CA
EDA Software Developer, Expert Witness
www.chapman-consulting-sj.com
2018-2019 Chair, IEEE Consultants' Network of Silicon Valley

svn.haxx.se is going away

[Mark Phippard]

Just a general fyi ... I went to https://svn.haxx.se/ today to search the
lists and noticed there is a banner on the site saying it is going offline
forever soon.

I am not sure what the ramifications will be as I know there are a lot of
historical links in the docs and site but I guess it is what it is.

-- 
Thanks

Mark Phippard