Re: [Twisted-Python] bad news about twistedmatrix.com's hosting

[Evilham]

On dt., ag. 03 2021, Adi Roiban wrote:

On Sun, 1 Aug 2021 at 10:29, Evilham  
wrote:



\o

On dg., ag. 01 2021, Glyph wrote:

> Hello Twistors,
>
> Our venerable hosting provider, tummy.com 
> ,
> will be terminating their business operations on September 
> 30,

> 2021.
>
 datetime.date(2021, 9, 30) - datetime.date.today()
> datetime.timedelta(days=60)
>
> In addition to being a bit personally sad - Tummy has been an
> avid supporter of the Twisted community since the very 
> beginning

> - this means we have just about 60 days to fully migrate our
> website, mailing list, and DNS's hosting from Dornkirk, the
> machine where it's been running continuously since 2013, to 
> ...

> something else.


it is sad indeed when friendly businesses terminate operations,
but Twisted will go on :-).


> There are 2 problems here:
>
> We need a sponsor to help us find a new hosting or cloud
> provider where things can be hosted.  We do pay tummy 
> currently,

> and we can probably ask the SFC to pay a different hosting
> provider a similar price for equivalent services, but ideally
> we'd find someone willing to donate something so as not to 
> put a

> drain on those resources.


Let's define this so I and others can ask around with various
infra providers:

- What is the service currently? (e.g. VPS/Physical? Which
  resources?)



We only have a single server. It is  called "dornkirk"
I don't know if it's a VM or bare-metal... but it shouldn't 
matter.


- Are the resources allocated adequate? Can/should they be

  increased/reduced?



I don't understand this question. Sorry.
Can you please rephrase ? Thanks.



This was very poorly worded, my bad.
I meant how big of a machine (storage excluding ML archives, RAM, 
CPU, traffic) is currently used and whether or not it is 
over/under-provisioned.
Without knowing this concretely, it is hard to try to get an 
entity to commit to donating resources.


It is good to know that VM vs physical and location are not 
important factors.



- Is currently something on the "wishlist" for this service? 
(e.g.

  IPv6 support!)



Many things are on the wishlist.
The priority is to have the services up and running, even after 
dornkirk is

dead.


- Even if the goal is and should be for that expense to 
disappear
  or go down significantly: roughly what is the compensation 
  being

  offered for this service? (it might be easier to convince
  someone to offer it at cost/partial compensation than it is 
  to

  offer it for free)



I don't know what is the current cost to cover the expenses for 
dornkirk.





- Is location important? E.g. if I asked around and it were
  fruitful, it'd be EU or Switzerland-based and not US-based as 
  it

  is right now.



No. Location is not important.




> I will need significant help with the operational aspects of 
> the
> migration.  I typically do a bunch of behind-the-scenes 
> system
> administration for twistedmatrix.com 
> 
> to keep the whole thing clunking along, but after my brain 
> has

> been fried by a year and a half's worth of pandemic stress, I
> simply don't have the time, energy, or mental capacity to do
> this.  I know I've asked for help before and rarely received
> any, but if I have to do this entirely by myself, the most
> likely outcome is that I'll migrate DNS to some free cloud 
> thing
> so at least my email address doesn't break (I migrated SMTP, 
> and
> thereby everything associated with personal email addresses 
> to
> Mailgun some years ago, so Tummy is not in that loop for 
> that),
> and the website (and this mailing list) will simply stop 
> working

> sometime between now and October.  So if you'd like to keep
> www.twistedmatrix.com ,
> speed.twistedmatrix.com , 
> our

> IRC bot (which is offline anyway right now due to Freenode
> shutting down), https://twistedmatrix.com/highscores/
> , this mailing list, 
> or
> our dogfooding instance for Twisted Names DNS, please 
> volunteer

> so we can start to coordinate.


Thank you for both doing this so long and so far and for being
straightforward about not being able to do it as much now.

I'd like to help and would be particularly (aka more 
intensively)
able to do so during August, even if that's just helping 
document

what needs to be done and how.
On a day-to-day basis I do some other sysadmin work and if
necessary would be able to accommodate taking care of
Twisted-related things into that.



To coordinate migration and also day-to-day Twisted project 
things we use

the IRC Libera channel #twisted-dev

If you don't like IRC ... we are also experiencing with 
Gitter/Matrix

https://gitter.im/twisted/twisted

I start this to-do list 
https://github.com/twisted-infra/braid/issues/277



IM is particularly taxing these days.

The TODO list looks quite comprehensive, thank you.






This is the 

Re: [Twisted-Python] bad news about twistedmatrix.com's hosting

[Adi Roiban]

On Sun, 1 Aug 2021 at 10:29, Evilham  wrote:

> \o
>
> On dg., ag. 01 2021, Glyph wrote:
>
> > Hello Twistors,
> >
> > Our venerable hosting provider, tummy.com ,
> > will be terminating their business operations on September 30,
> > 2021.
> >
>  datetime.date(2021, 9, 30) - datetime.date.today()
> > datetime.timedelta(days=60)
> >
> > In addition to being a bit personally sad - Tummy has been an
> > avid supporter of the Twisted community since the very beginning
> > - this means we have just about 60 days to fully migrate our
> > website, mailing list, and DNS's hosting from Dornkirk, the
> > machine where it's been running continuously since 2013, to ...
> > something else.
>
>
> it is sad indeed when friendly businesses terminate operations,
> but Twisted will go on :-).
>
>
> > There are 2 problems here:
> >
> > We need a sponsor to help us find a new hosting or cloud
> > provider where things can be hosted.  We do pay tummy currently,
> > and we can probably ask the SFC to pay a different hosting
> > provider a similar price for equivalent services, but ideally
> > we'd find someone willing to donate something so as not to put a
> > drain on those resources.
>
>
> Let's define this so I and others can ask around with various
> infra providers:
>
> - What is the service currently? (e.g. VPS/Physical? Which
>   resources?)
>

We only have a single server. It is  called "dornkirk"
I don't know if it's a VM or bare-metal... but it shouldn't matter.

- Are the resources allocated adequate? Can/should they be
>   increased/reduced?
>

I don't understand this question. Sorry.
Can you please rephrase ? Thanks.


> - Is currently something on the "wishlist" for this service? (e.g.
>   IPv6 support!)
>

Many things are on the wishlist.
The priority is to have the services up and running, even after dornkirk is
dead.


> - Even if the goal is and should be for that expense to disappear
>   or go down significantly: roughly what is the compensation being
>   offered for this service? (it might be easier to convince
>   someone to offer it at cost/partial compensation than it is to
>   offer it for free)
>

I don't know what is the current cost to cover the expenses for dornkirk.



> - Is location important? E.g. if I asked around and it were
>   fruitful, it'd be EU or Switzerland-based and not US-based as it
>   is right now.
>

No. Location is not important.


>
> > I will need significant help with the operational aspects of the
> > migration.  I typically do a bunch of behind-the-scenes system
> > administration for twistedmatrix.com 
> > to keep the whole thing clunking along, but after my brain has
> > been fried by a year and a half's worth of pandemic stress, I
> > simply don't have the time, energy, or mental capacity to do
> > this.  I know I've asked for help before and rarely received
> > any, but if I have to do this entirely by myself, the most
> > likely outcome is that I'll migrate DNS to some free cloud thing
> > so at least my email address doesn't break (I migrated SMTP, and
> > thereby everything associated with personal email addresses to
> > Mailgun some years ago, so Tummy is not in that loop for that),
> > and the website (and this mailing list) will simply stop working
> > sometime between now and October.  So if you'd like to keep
> > www.twistedmatrix.com ,
> > speed.twistedmatrix.com , our
> > IRC bot (which is offline anyway right now due to Freenode
> > shutting down), https://twistedmatrix.com/highscores/
> > , this mailing list, or
> > our dogfooding instance for Twisted Names DNS, please volunteer
> > so we can start to coordinate.
>
>
> Thank you for both doing this so long and so far and for being
> straightforward about not being able to do it as much now.
>
> I'd like to help and would be particularly (aka more intensively)
> able to do so during August, even if that's just helping document
> what needs to be done and how.
> On a day-to-day basis I do some other sysadmin work and if
> necessary would be able to accommodate taking care of
> Twisted-related things into that.
>

To coordinate migration and also day-to-day Twisted project things we use
the IRC Libera channel #twisted-dev

If you don't like IRC ... we are also experiencing with Gitter/Matrix
https://gitter.im/twisted/twisted

I start this to-do list https://github.com/twisted-infra/braid/issues/277


> This is the documentation that I am aware of:
> https://twistedmatrix.com/trac/wiki/Infrastructure


There is not much documentation... we need to rely on memories :)
So don't bother to search for documentation, and just ask :)


>
> Particularly braid: https://github.com/twisted-infra/braid
>
>
Yes. braid is our deployment as code thing... is the best we have in terms
of documenting the infrastructure.

"braid" is (was?) the Twisted customization on top of fabric 

Re: [Twisted-Python] bad news about twistedmatrix.com's hosting

[Glyph]

> On Aug 1, 2021, at 2:49 AM, Adi Roiban  > wrote:
> 
> Hi Glyph
> 
> Thanks for sending the message.

As a wise computer once sung, "We do what we must, because we can".

> This is bad, but I am confident that we can do the migration in 60 days, as 
> long as we don't expect to have the exact same Twisted infrastructure.

It's certainly annoying (and I'm very sad to see tummy go, they were great) but 
I think we may be happy with the opportunity to reduce the amount of stuff we 
operate.

> We have a monthly credit of 150 USD from MS to spend on Azure services.

This is great news!  Presumably I … already have access to this, somehow?  Is 
it in the 1password team vault?  How do I see it?  Do we have a commitment from 
MS to actually maintain this credit?

> A quick hack would be to create a new VM in Azure with Ubuntu 18.04 and 
> migrate the existing server to it.

If we need to provision a new server anyway, could we provision it on the most 
recent ubuntu?  Presumably most of the stuff we require will still be on there? 
 If it's a hassle let's fall back to 18.04 but let's not set ourselves up to be 
outdated as long as we're doing this work.

> Do we have other servers hosted with tummy.com , other 
> than dornkirk.twistedmatrix.com  ?

Nope.  We used to have a VPS called Wolfwood, but in an act of nominative 
determinism, it had to die for the greater good.  (It was our source-control 
server, back when that was a thing you had to run yourself.  We shut it down 
when we migrated to github.)

> I will not have too much time to do the migration, but I can ask a sysadmin 
> from my work to help with that and my company will cover that costs.

Thanks Adi, this is very generous.  Let me know if I'm needed to give this 
sysadmin any requisite privileges.

> I think that for now, we should see who is interested with helping with the 
> migration and after 1 week start to work with that team.

Looks like a team is coming together :).

> Maybe we will need to schedule an online meeting, but I hope we can do async 
> work via GitHub Issues (braid) and IRC/Gitter.

Yeah, async seems to be working fine.

> I saw that you have already considered using containers for Twisted infra 
> https://github.com/twisted-infra/braid/issues/265 
> 
Yes. I'd love to do this, and if such a migration reduces effort, then let's do 
it by all means - I just don't want to get caught up in trying to do this and 
ending up with things broken when the clock runs out.

> In parallel with migrating  dornkirk.twistedmatrix.com 
> , we can look at outsourcing some of our 
> services.
> For example
> 
> * Move mailing list to the main Python mailing list server

Looks like this is full steam ahead!

> * Move DNS to a cloud provider (Azure DNS)

I've already pulled the trigger to migrate our DNS to Gandi, since they're our 
registrar and I already know I can provision certificates against their LiveDNS 
API :).  We do still have to get a few other domains off of this infrastructure 
though, I'll do the ones we control (Divunal) and contact folks at the others.

Longer-term, I would like to host some DNS, since I want to make sure that 
twisted.names does not bit-rot.  However, perhaps a prerequisite to setting 
this up again will be (A) containerizing it and (B) providing something like 
Sentry for us to track exceptions, so that the dog-fooding here is interesting 
and not just a tedious obligation that produces gigabytes full of 
tracback-laden log files that nobody ever looks at or fixes.

> * Move Trac Wiki to GitHub Wiki

The trac wiki's main purpose is not really to serve as a wiki-style editable 
document repository, but rather to host the website.  I think migrating it to 
something like Pelican, or some other static-site management thing might make 
more sense.

> * Move Trac Tickets to GitHub Issues

Sure, sure, as soon as we figure out a way to preserve all the links :).

> * Move static file hosting to GitHub pages

I wonder if it would make sense for www.twistedmatrix.com 
 itself to be hosted by 

> * Move highscore to a container and update it to pull info from GitHub hooks

+1.  This thing might be interesting 
> in that it might be a better 
starting point than our own code (which is not even open source because, among 
other reasons, it is such trash) this version gets the point of the scoring 
wrong, which is to weight 

> * IRC bot to some sort of integration to Gitter and then IRC bridge

Given that this is already broken right now, I'd say "don't bother migrating 
the IRC bot right away".  This also means we don't need amptrac, and we can 
rebuild the IRC bot in a more sustainable way, inside a container, later

> * 

Re: [Twisted-Python] bad news about twistedmatrix.com's hosting

[Adi Roiban]

On Sun, 1 Aug 2021 at 08:42, Glyph  wrote:

> Hello Twistors,
>
> Our venerable hosting provider, tummy.com, will be terminating their
> business operations on September 30, 2021.
>
> >>> datetime.date(2021, 9, 30) - datetime.date.today()
> datetime.timedelta(days=60)
>
>
> In addition to being a bit personally sad - Tummy has been an avid
> supporter of the Twisted community since the very beginning - this means we
> have just about 60 days to fully migrate our website, mailing list, and
> DNS's hosting from Dornkirk, the machine where it's been running
> continuously since 2013, to ... something else.
>
> There are 2 problems here:
>
>
>1. We need a sponsor to help us find a new hosting or cloud provider
>where things can be hosted.  We do pay tummy currently, and we can probably
>ask the SFC to pay a different hosting provider a similar price for
>equivalent services, but ideally we'd find someone willing to donate
>something so as not to put a drain on those resources.
>
>2. I will need significant help with the operational aspects of the
>migration.  I typically do a bunch of behind-the-scenes system
>administration for twistedmatrix.com to keep the whole thing clunking
>along, but after my brain has been fried by a year and a half's worth of
>pandemic stress, I simply don't have the time, energy, or mental capacity
>to do this.  I know I've asked for help before and rarely received any, but
>if I have to do *this* entirely by myself, the most likely outcome is
>that I'll migrate DNS to some free cloud thing so at least my email address
>doesn't break (I migrated SMTP, and thereby everything associated with
>personal email addresses to Mailgun some years ago, so Tummy is not in that
>loop for that), and the website (and this mailing list) will simply stop
>working sometime between now and October.  So if you'd like to keep
>www.twistedmatrix.com, speed.twistedmatrix.com, our IRC bot (which is
>offline anyway right now due to Freenode shutting down),
>https://twistedmatrix.com/highscores/, this mailing list, or our
>dogfooding instance for Twisted Names DNS, please volunteer so we can start
>to coordinate.
>
>
> If you've been wanting to just use Github Issues and don't care about
> preserving any of the data in Trac, congratulations, you're about to win
> that argument by default ;-).
>
> In the worst case scenario, I will download a backup image before things
> get turned off in case someone wants to deal with this
>
>
Hi Glyph

Thanks for sending the message.
This is bad, but I am confident that we can do the migration in 60 days, as
long as we don't expect to have the exact same Twisted infrastructure.

We have a monthly credit of 150 USD from MS to spend on Azure services.

A quick hack would be to create a new VM in Azure with Ubuntu 18.04 and
migrate the existing server to it.

Do we have other servers hosted with tummy.com, other than
dornkirk.twistedmatrix.com ?

I will not have too much time to do the migration, but I can ask a sysadmin
from my work to help with that and my company will cover that costs.

I think that for now, we should see who is interested with helping with the
migration and after 1 week start to work with that team.

Maybe we will need to schedule an online meeting, but I hope we can do
async work via GitHub Issues (braid) and IRC/Gitter.

I saw that you have already considered using containers for Twisted infra
https://github.com/twisted-infra/braid/issues/265

In parallel with migrating  dornkirk.twistedmatrix.com, we can look at
outsourcing some of our services.
For example

* Move mailing list to the main Python mailing list server
* Move DNS to a cloud provider (Azure DNS)
* Move Trac Wiki to GitHub Wiki
* Move Trac Tickets to GitHub Issues
* Move static file hosting to GitHub pages
* Move highscore to a container and update it to pull info from GitHub hooks
* IRC bot to some sort of integration to Gitter and then IRC bridge
* Retire speed tests and buildbot

--

>From past experience, I would say that most active Twisted contributors are
software developers and not sysadmin.
So we don't have the resources to keep custom Twisted infrastructure.
And with the limited resources that we have, I would like to see Twisted
contributors work at doing code review, fixing bugs rather than doing
infrastructure work.

So I am +1 for outsourcing as many Twisted services as possible.

I think that we have already outsourced a few services:

* Automated testing to Azure Pipelines and GitHub Actions
* Git Hosting to GitHub
* Some Twisted sub-projects use GitHub Issues
* Python package downloads to PyPi
* Documentation hosting to Read The Docs

PS: for quick text messages if you don't like IRC, we are experimenting
with Gitter at https://gitter.im/twisted/twisted

-- 
Adi Roiban
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com

Re: [Twisted-Python] bad news about twistedmatrix.com's hosting

[Evilham]

\o

On dg., ag. 01 2021, Glyph wrote:


Hello Twistors,

Our venerable hosting provider, tummy.com , 
will be terminating their business operations on September 30, 
2021.



datetime.date(2021, 9, 30) - datetime.date.today()

datetime.timedelta(days=60)

In addition to being a bit personally sad - Tummy has been an 
avid supporter of the Twisted community since the very beginning 
- this means we have just about 60 days to fully migrate our 
website, mailing list, and DNS's hosting from Dornkirk, the 
machine where it's been running continuously since 2013, to ... 
something else.



it is sad indeed when friendly businesses terminate operations, 
but Twisted will go on :-).




There are 2 problems here:

We need a sponsor to help us find a new hosting or cloud 
provider where things can be hosted.  We do pay tummy currently, 
and we can probably ask the SFC to pay a different hosting 
provider a similar price for equivalent services, but ideally 
we'd find someone willing to donate something so as not to put a 
drain on those resources.



Let's define this so I and others can ask around with various 
infra providers:


- What is the service currently? (e.g. VPS/Physical? Which 
 resources?)
- Are the resources allocated adequate? Can/should they be 
 increased/reduced?
- Is currently something on the "wishlist" for this service? (e.g. 
 IPv6 support!)
- Even if the goal is and should be for that expense to disappear 
 or go down significantly: roughly what is the compensation being 
 offered for this service? (it might be easier to convince 
 someone to offer it at cost/partial compensation than it is to 
 offer it for free)
- Is location important? E.g. if I asked around and it were 
 fruitful, it'd be EU or Switzerland-based and not US-based as it 
 is right now.



I will need significant help with the operational aspects of the 
migration.  I typically do a bunch of behind-the-scenes system 
administration for twistedmatrix.com  
to keep the whole thing clunking along, but after my brain has 
been fried by a year and a half's worth of pandemic stress, I 
simply don't have the time, energy, or mental capacity to do 
this.  I know I've asked for help before and rarely received 
any, but if I have to do this entirely by myself, the most 
likely outcome is that I'll migrate DNS to some free cloud thing 
so at least my email address doesn't break (I migrated SMTP, and 
thereby everything associated with personal email addresses to 
Mailgun some years ago, so Tummy is not in that loop for that), 
and the website (and this mailing list) will simply stop working 
sometime between now and October.  So if you'd like to keep 
www.twistedmatrix.com , 
speed.twistedmatrix.com , our 
IRC bot (which is offline anyway right now due to Freenode 
shutting down), https://twistedmatrix.com/highscores/ 
, this mailing list, or 
our dogfooding instance for Twisted Names DNS, please volunteer 
so we can start to coordinate.



Thank you for both doing this so long and so far and for being 
straightforward about not being able to do it as much now.


I'd like to help and would be particularly (aka more intensively) 
able to do so during August, even if that's just helping document 
what needs to be done and how.
On a day-to-day basis I do some other sysadmin work and if 
necessary would be able to accommodate taking care of 
Twisted-related things into that.


This is the documentation that I am aware of: 
https://twistedmatrix.com/trac/wiki/Infrastructure

Particularly braid: https://github.com/twisted-infra/braid

Is there anything else to look at? Something known to not be 
supported in braid / only partially supported?
Would you rather schedule some online voice session where I poke 
your brain at what needs to be done (IIRC something similar was 
done about 5 years ago) and take notes that can be used (also by 
someone else) while you let your mind flow happily?



If you've been wanting to just use Github Issues and don't care 
about preserving any of the data in Trac, congratulations, 
you're about to win that argument by default ;-).


In the worst case scenario, I will download a backup image 
before things get turned off in case someone wants to deal with 
this



I personally hope this doesn't happen; particularly the mailing 
list is a great low-bandwidth way of staying mostly up-to-date 
with the project.


Cheers,
--
Evilham

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

[Twisted-Python] bad news about twistedmatrix.com's hosting

[Glyph]

Hello Twistors,

Our venerable hosting provider, tummy.com , will be 
terminating their business operations on September 30, 2021.

>>> datetime.date(2021, 9, 30) - datetime.date.today()
datetime.timedelta(days=60)

In addition to being a bit personally sad - Tummy has been an avid supporter of 
the Twisted community since the very beginning - this means we have just about 
60 days to fully migrate our website, mailing list, and DNS's hosting from 
Dornkirk, the machine where it's been running continuously since 2013, to ... 
something else.

There are 2 problems here:

We need a sponsor to help us find a new hosting or cloud provider where things 
can be hosted.  We do pay tummy currently, and we can probably ask the SFC to 
pay a different hosting provider a similar price for equivalent services, but 
ideally we'd find someone willing to donate something so as not to put a drain 
on those resources.

I will need significant help with the operational aspects of the migration.  I 
typically do a bunch of behind-the-scenes system administration for 
twistedmatrix.com  to keep the whole thing clunking 
along, but after my brain has been fried by a year and a half's worth of 
pandemic stress, I simply don't have the time, energy, or mental capacity to do 
this.  I know I've asked for help before and rarely received any, but if I have 
to do this entirely by myself, the most likely outcome is that I'll migrate DNS 
to some free cloud thing so at least my email address doesn't break (I migrated 
SMTP, and thereby everything associated with personal email addresses to 
Mailgun some years ago, so Tummy is not in that loop for that), and the website 
(and this mailing list) will simply stop working sometime between now and 
October.  So if you'd like to keep www.twistedmatrix.com 
, speed.twistedmatrix.com 
, our IRC bot (which is offline anyway right 
now due to Freenode shutting down), https://twistedmatrix.com/highscores/ 
, this mailing list, or our dogfooding 
instance for Twisted Names DNS, please volunteer so we can start to coordinate.

If you've been wanting to just use Github Issues and don't care about 
preserving any of the data in Trac, congratulations, you're about to win that 
argument by default ;-).

In the worst case scenario, I will download a backup image before things get 
turned off in case someone wants to deal with this 

-g

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python