Re: [Labs-l] [Labs-announce] Switching to new puppetmaster tomorrow (mostly done)

[Andrew Bogott]

This is mostly done.  All instances that I can reach and that had 
working puppet are now using the new puppetmaster.


I'm now going to go through the list of unreachable and broken 
instances, and will be contacting project admins directly about those.  
Otherwise, everything should be largely as it was.


Please let me know if you see unexpected behavior.

-Andrew


On 8/3/17 10:52 AM, Andrew Bogott wrote:
Tomorrow (2017-08-04) at around 15:00 UTC I'm going to start moving 
cloud instances to a new puppetmaster.  This shouldn't require any 
action from anyone else, and will probably not be noticeable.  If 
things go poorly you might get a bunch of puppet failure email spam or 
alerts through other channels.


This will only affect instances that use the standard upstream 
puppetmaster (currently labs-puppetmaster-eqiad.wikimedia.org). 
Instances using a project-local puppetmaster will be unaffected.


I'll send an 'all clear' email when I finish moving things over; after 
that I'll be interested in hearing about any ill effects that remain.


-Andrew




___
Labs-announce mailing list
labs-annou...@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-announce
___
Labs-l mailing list
Labs-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-l

[Labs-l] [Labs-announce] Switching to new puppetmaster tomorrow

[Andrew Bogott]

Tomorrow (2017-08-04) at around 15:00 UTC I'm going to start moving 
cloud instances to a new puppetmaster.  This shouldn't require any 
action from anyone else, and will probably not be noticeable.  If things 
go poorly you might get a bunch of puppet failure email spam or alerts 
through other channels.


This will only affect instances that use the standard upstream 
puppetmaster (currently labs-puppetmaster-eqiad.wikimedia.org). 
Instances using a project-local puppetmaster will be unaffected.


I'll send an 'all clear' email when I finish moving things over; after 
that I'll be interested in hearing about any ill effects that remain.


-Andrew


___
Labs-announce mailing list
labs-annou...@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-announce
___
Labs-l mailing list
Labs-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-l

Re: [Labs-l] [Labs-announce] ldap outage in progress (meaning: no logins for a bit) -- RESOLVED

[Andrew Bogott]

Ldap is back, and most services have returned to normal.  You might run 
into a few bad states for the next few minutes due to caching issues, 
but in general everything should be working fine.


Please file a ticket or let me know if you see any issues that persist 
for more than an hour or two.


-A


On 7/19/17 3:44 PM, Andrew Bogott wrote:
Both of our ldap servers are currently refusing most connections.  The 
problem is well-understood and a fix is in progress.


In the meantime: most currently running services should continue to 
work just fine.  Shell and web logins (and sudo) on most Tools and 
Labs/VPS infrastructure will fail until ldap service is restored, and 
there will be all kinds of weird warnings and alerts as puppet fails 
all over the place.


I'll send another email when normal service is restored.  It's 
unlikely that users will need to do anything to follow up.


Sorry for the interruption!

-Andrew




___
Labs-announce mailing list
labs-annou...@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-announce
___
Labs-l mailing list
Labs-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-l

[Labs-l] [Labs-announce] ldap outage in progress (meaning: no logins for a bit)

[Andrew Bogott]

Both of our ldap servers are currently refusing most connections.  The 
problem is well-understood and a fix is in progress.


In the meantime: most currently running services should continue to work 
just fine.  Shell and web logins (and sudo) on most Tools and Labs/VPS 
infrastructure will fail until ldap service is restored, and there will 
be all kinds of weird warnings and alerts as puppet fails all over the 
place.


I'll send another email when normal service is restored.  It's unlikely 
that users will need to do anything to follow up.


Sorry for the interruption!

-Andrew


___
Labs-announce mailing list
labs-annou...@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-announce
___
Labs-l mailing list
Labs-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-l

Re: [Labs-l] [Labs-announce] Mild but long-running Tools outage in process, about to get worse!

[Andrew Bogott]

After various failed measures, we're now trying to revert back to the 
older kernel and switching back between NFS servers yet again.  So Tools 
NFS (and various associated services) will probably break, at least for 
a few minutes.


With luck this will get us into a stable place, but I'll update again 
regardless.


-Andrew


On 6/29/17 3:27 PM, Andrew Bogott wrote:
The tools cluster is suffering from several maladies right now. 
Existing services seem to be mostly fine, but any kubernetes services 
that tried to restart in the last few hours probably failed to start, 
and new things are still failing to start.  Similarly, web services 
and other tools are failing to restart in several cases.


There are various theories as to what's going on -- most likely 
it's a kernel-version incompatibility with the newly upgraded NFS 
server.  There was an earlier ldap outage which is better understood 
and should be resolved by now.


We apologize for the inconvenience, and are working frantically to 
restore stability.  There will be a follow-up email when things are 
resolved.


-Andrew





___
Labs-announce mailing list
labs-annou...@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-announce
___
Labs-l mailing list
Labs-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-l

[Labs-l] [Labs-announce] Mild but long-running Tools outage in process

[Andrew Bogott]

The tools cluster is suffering from several maladies right now. 
Existing services seem to be mostly fine, but any kubernetes services 
that tried to restart in the last few hours probably failed to start, 
and new things are still failing to start.  Similarly, web services and 
other tools are failing to restart in several cases.


There are various theories as to what's going on -- most likely 
it's a kernel-version incompatibility with the newly upgraded NFS 
server.  There was an earlier ldap outage which is better understood and 
should be resolved by now.


We apologize for the inconvenience, and are working frantically to 
restore stability.  There will be a follow-up email when things are 
resolved.


-Andrew



___
Labs-announce mailing list
labs-annou...@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-announce
___
Labs-l mailing list
Labs-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-l

Re: [Labs-l] [Labs-announce] IMPORTANT: Security reboots everywhere tomorrow

[Andrew Bogott]

Good news:

I've now finished rebooting all the labvirt hosts, and the last few VMs 
should be in the process of spinning back up right now.


Bad news:

- We have a lot more (non-VM-hosting) servers to restart!  Expect brief 
outages in storage, horizon, wikitech, etc during the next few hours.


- Most VMs are still running insecure kernels.  I didn't upgrade them 
yet because it's not entirely clear that the current generation of 
security patches won't break random Java apps here and there.  Once 
there's a clear resolution to that issue there will be another round of 
reboots, which I will announce in advance.


- The restart process hit k8s nodes a bit harder than planned, so if 
your tools are running on Kubernetes you might want to verify that 
things are up and running properly.  If they aren't, a simple restart 
should resolve things.





On 6/21/17 8:56 AM, Andrew Bogott wrote:

Reminder:  These reboots will start in about 5 minutes.

On 6/20/17 11:38 AM, Andrew Bogott wrote:

Good morning!

In order to plug a newly-revealed security hole, we need to upgrade 
kernels everywhere and reboot everything.  I'll be doing this for all 
instances and VM hosts tomorrow, 2017-06-21, beginning at 14:00UTC 
(aka 7AM in San Francisco).


Project admins should expect instances to be shut down for 10-15 
minutes and then restarted at some point tomorrow.  I don't expect 
the process to take absolutely all day, but it might.  If you have an 
instance that is a special case and you need it to be restarted 
during a more specific window, please respond to me directly with 
details about what you need and we'll try to make it happen.


These reboots should have minimal impact on things running in Tools, 
although it's always good to keep an eye out -- some tools are unable 
to recover from a shift between nodes and need a manual restart.


Sorry for the inconvenience!

-Andrew







___
Labs-announce mailing list
labs-annou...@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-announce
___
Labs-l mailing list
Labs-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-l

Re: [Labs-l] [Labs-announce] IMPORTANT: Security reboots everywhere tomorrow

[Andrew Bogott]

Reminder:  These reboots will start in about 5 minutes.

On 6/20/17 11:38 AM, Andrew Bogott wrote:

Good morning!

In order to plug a newly-revealed security hole, we need to upgrade 
kernels everywhere and reboot everything.  I'll be doing this for all 
instances and VM hosts tomorrow, 2017-06-21, beginning at 14:00UTC 
(aka 7AM in San Francisco).


Project admins should expect instances to be shut down for 10-15 
minutes and then restarted at some point tomorrow.  I don't expect the 
process to take absolutely all day, but it might.  If you have an 
instance that is a special case and you need it to be restarted during 
a more specific window, please respond to me directly with details 
about what you need and we'll try to make it happen.


These reboots should have minimal impact on things running in Tools, 
although it's always good to keep an eye out -- some tools are unable 
to recover from a shift between nodes and need a manual restart.


Sorry for the inconvenience!

-Andrew





___
Labs-announce mailing list
labs-annou...@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-announce
___
Labs-l mailing list
Labs-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-l

[Labs-l] [Labs-announce] Experimental Debian Stretch image now available

[Andrew Bogott]

Good morning!

I've just installed a new public base image, ' debian-9.0-stretch 
(experimental)' and made it available for all projects.  It should 
appear in the standard 'Source' UI in Horizon any time you create a new VM.


Please take heed of the word 'experimental' in the title.  Stretch is 
not yet an official Debian release, and may still contain unexpected 
bugs and security issues.  Additionally, the operations puppet repo has 
only begun to be Stretch-aware, so many roles and classes will likely 
fail or engage in otherwise undefined behavior.


So...

- Please DO use this image to test and update puppet classes, and to get 
a preview of things to come.


- Please DO NOT use this image to build any instances that you plan to 
expose to the public, or that you want to keep around for more than a 
few months.


When Stretch is officially released and we've ironed out some more 
puppet issues I will delete this base image and purge those VMs that 
were built from it in order to avoid the debt of having to support 
'experimental' VMs for eternity.


Happy testing!

-Andrew


___
Labs-announce mailing list
labs-annou...@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-announce
___
Labs-l mailing list
Labs-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-l

Re: [Labs-l] [Labs-announce] Instance management disabled for a bit (resolved, but happening again tomorrow)

[Andrew Bogott]

The switch maintenance that we were bracing for didn't happen today.  
I've re-enabled normal functionality on Horizon and Wikitech, but will 
disable them again during the window tomorrow (currently proposed for 
16:00 UTC).


I probably won't spam the list with any more on-and-off messages about 
this tomorrow unless something truly unusual happens.


-Andrew


On 4/26/17 11:49 AM, Andrew Bogott wrote:
This is turning out to take much longer than I expected.  I don't know 
how long, in full, but I still expect things to be wrapped up by the 
end of the day (in 5-6 hours max.)


Sorry again for any inconvenience caused.

-Andrew


On 4/26/17 8:33 AM, Andrew Bogott wrote:
There's some datacenter work going on today which is likely to 
break new instance creation, so I've disabled instance management on 
Horizon to prevent confusion. Due to the slapdash nature of this 
change, the UI will be a bit cryptic; if there are buttons missing 
from your Horizon panels don't worry, they should return in an hour 
or two.


Sorry for the inconvenience!


-Andrew






___
Labs-announce mailing list
labs-annou...@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-announce
___
Labs-l mailing list
Labs-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-l

Re: [Labs-l] [Labs-announce] Instance management disabled for a bit

[Andrew Bogott]

This is turning out to take much longer than I expected.  I don't know 
how long, in full, but I still expect things to be wrapped up by the end 
of the day (in 5-6 hours max.)


Sorry again for any inconvenience caused.

-Andrew


On 4/26/17 8:33 AM, Andrew Bogott wrote:
There's some datacenter work going on today which is likely to 
break new instance creation, so I've disabled instance management on 
Horizon to prevent confusion.  Due to the slapdash nature of this 
change, the UI will be a bit cryptic; if there are buttons missing 
from your Horizon panels don't worry, they should return in an hour or 
two.


Sorry for the inconvenience!


-Andrew




___
Labs-announce mailing list
labs-annou...@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-announce
___
Labs-l mailing list
Labs-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-l

[Labs-l] [Labs-announce] Instance management disabled for a bit

[Andrew Bogott]

There's some datacenter work going on today which is likely to 
break new instance creation, so I've disabled instance management on 
Horizon to prevent confusion.  Due to the slapdash nature of this 
change, the UI will be a bit cryptic; if there are buttons missing from 
your Horizon panels don't worry, they should return in an hour or two.


Sorry for the inconvenience!


-Andrew


___
Labs-announce mailing list
labs-annou...@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-announce
___
Labs-l mailing list
Labs-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-l

[Labs-l] [Labs-announce] Brief Horizon/nova outage coming up

[Andrew Bogott]

As part of a necessary service failover, I'm going to disable the 
openstack API for a few minutes, in a few minutes.  This will mean that 
new instance creation/deletion will be prevent, as well as most other 
features on Horizon.


Sorry for the short notice!  This should be over fairly quickly.

-Andrew


___
Labs-announce mailing list
labs-annou...@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-announce
___
Labs-l mailing list
Labs-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-l

[Labs-l] [Labs-announce] Does anyone care about service groups?

[Andrew Bogott]

Back when we created tools, we implemented tool accounts as a 
general labs-wide feature, 'service groups.'  A tool is a service group, 
but any other project can also create a single-use group or user via the 
'Manager Service Groups' link in Wikitech.


I just ran a big report of all uses of this feature outside of 
tools, and reached out to many of the projects that contain service 
groups.  The use cases I encountered were:


1) "I'm a Labs Op and am using that to test a feature for the tools project"
2) "I used that feature by mistake/I was just clicking around at random"
3) "That feature seemed like it might be useful but it wasn't and I 
never cleaned up afterwards"


So!  I propose deprecate and remove the service group feature 
outside of Tools.  Specifically:


- The new tools admin tool (https://toolsadmin.wikimedia.org/) will 
implement a tool request/approval/creation system that replaces the 
wikitech 'service groups' feature for tool creation purposes.


- Once that is done, I will remove the 'Manage Service Groups' link from 
wikitech, and disable all wikitech features related to service groups.


- I will delete all existing service groups outside of case 1 above

- The backend implementation of service groups will remain the same as 
it is now, and by special request (mostly, again, for case 1), operators 
will still be able to create/manipulate service groups outside of the 
Tools project via arduous by-hand commandline machinations.


Does that sound OK to everyone?  Does anyone want to speak up in 
favor of preserving this feature?


-Andrew


___
Labs-announce mailing list
labs-annou...@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-announce
___
Labs-l mailing list
Labs-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-l

Re: [Labs-l] [Labs-announce] Ubuntu Precise instances will be shut down at the end of this month (Done, and thank you!)

[Andrew Bogott]

I just deleted the last Precise instances that were remaining in Labs, 
and we've closed https://phabricator.wikimedia.org/T143349 and related 
subtasks.


I'm very impressed at how responsive and cooperative you, the users, 
were with this awkward process.  We were able to locate responsible 
people for every single affected project, and folks were especially 
gracious and helpful with the upgrades and migrations.  Thank you!


Keep in mind that the same process is likely to happen again with Trusty 
in 2019.  So if you're building new systems today, best to start with 
Jessie (which is slated to last until at least 2020) and postpone the 
headache.


Nice work, everyone!

-Andrew + all of the Labs team


On 3/27/17 9:28 AM, Andrew Bogott wrote:

There are now four days remaining until the Ubuntu Precise deadline.

On Friday the 31st I WILL DELETE remaining precise instances, both 
those that are currently running and those that are in a SHUTOFF state.


The good news is, we're almost to the goal.  The remaining hosts that 
I know about are:


analytics: limn1
openstack: labs-vmbuilder-precise  (which is mine!)
wildcat: dannyb


Please let me know if you need additional assistance with any of the 
above.  Documentation for this process can be found at 
https://phabricator.wikimedia.org/T143349.


-Andrew




___
Labs-announce mailing list
labs-annou...@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-announce
___
Labs-l mailing list
Labs-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-l

[Labs-l] [Labs-announce] Super Reminder: Ubuntu Precise instances will be shut down at the end of this month

[Andrew Bogott]

There are now four days remaining until the Ubuntu Precise deadline.

On Friday the 31st I WILL DELETE remaining precise instances, both those 
that are currently running and those that are in a SHUTOFF state.


The good news is, we're almost to the goal.  The remaining hosts that I 
know about are:


analytics: limn1
openstack: labs-vmbuilder-precise  (which is mine!)
wildcat: dannyb


Please let me know if you need additional assistance with any of the 
above.  Documentation for this process can be found at 
https://phabricator.wikimedia.org/T143349.


-Andrew


___
Labs-announce mailing list
labs-annou...@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-announce
___
Labs-l mailing list
Labs-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-l

Re: [Labs-l] [Labs-announce] rolling outages and reboots on Wednesday, including tools-bastion-02

[Andrew Bogott]

Reminder:  All of this will start happening in about 30 minutes.

On 3/20/17 6:19 PM, Andrew Bogott wrote:
One of our virtualization hosts (labvirt1001) is exhibiting troubling 
behavior.  Nothing hosted on that box is currently affected, but to be 
on the safe side we're going to evacuate its VMs and run it through 
some tests.


The evacuation process is a bit slow and haphazard -- while being 
transferred, any one VM may experience 30-60 minutes of downtime, and 
will be fully rebooted at the end.  The VMs that will be affected by 
this process are:


tools-bastion-02 | tools
citoidtest   | services
clm-web-01   | community-labs-monitoring
db   | cognate
elastic1 | analytics
etherpad01   | etherpad
maps-tiles2  | maps
osmit-tre| osmit
perf-testing | mobile
puppet-mailman   | puppet
test-spm-1   | project-proxy
wikidata-mobile  | wikidata-dev
wmt-exec | wmt


Note that tools-bastion-02 is on that list!  I will be moving that 
system first, at 14:00 UTC on Wednesday the 22nd (that's 7AM San 
Francisco time.)


The other instances will be evacuated over the course of that day, in 
no particular order and at no particular time.  If you require a more 
specific migration window, please respond and let me know. 
Alternatively, if you totally don't care about downtime and are happy 
to have your instance migrated tomorrow instead of Wednesday, that 
would be good to know as well.


Thank you, and sorry for the inconvenience.

-Andrew + the Labs Team




___
Labs-announce mailing list
labs-annou...@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-announce
___
Labs-l mailing list
Labs-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-l

[Labs-l] [Labs-announce] rolling outages and reboots on Wednesday, including tools-bastion-02

[Andrew Bogott]

One of our virtualization hosts (labvirt1001) is exhibiting troubling 
behavior.  Nothing hosted on that box is currently affected, but to be 
on the safe side we're going to evacuate its VMs and run it through some 
tests.


The evacuation process is a bit slow and haphazard -- while being 
transferred, any one VM may experience 30-60 minutes of downtime, and 
will be fully rebooted at the end.  The VMs that will be affected by 
this process are:


tools-bastion-02 | tools
citoidtest   | services
clm-web-01   | community-labs-monitoring
db   | cognate
elastic1 | analytics
etherpad01   | etherpad
maps-tiles2  | maps
osmit-tre| osmit
perf-testing | mobile
puppet-mailman   | puppet
test-spm-1   | project-proxy
wikidata-mobile  | wikidata-dev
wmt-exec | wmt


Note that tools-bastion-02 is on that list!  I will be moving that 
system first, at 14:00 UTC on Wednesday the 22nd (that's 7AM San 
Francisco time.)


The other instances will be evacuated over the course of that day, in no 
particular order and at no particular time.  If you require a more 
specific migration window, please respond and let me know.  
Alternatively, if you totally don't care about downtime and are happy to 
have your instance migrated tomorrow instead of Wednesday, that would be 
good to know as well.


Thank you, and sorry for the inconvenience.

-Andrew + the Labs Team


___
Labs-announce mailing list
labs-annou...@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-announce
___
Labs-l mailing list
Labs-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-l

[Labs-l] [Labs-announce] Reminder: Ubuntu Precise instances will be shut down at the end of this month

[Andrew Bogott]

There are now 25 days remaining until the Ubuntu Precise deadline.  
After that all remaining Labs VMs running Precise will be deleted.


Ideally, of course, owners of those VMs would be cleaned up by their 
owners before then so I can stop worrying about them.  The complete list 
of remaining instances is:


analytics: limn1
bots: wm-bot
huggle: huggle
integration: integration-slave-precise-1002
integration: integration-slave-precise-1012
integration: integration-slave-precise-1011
maps: maps-warper
openstack: labs-vmbuilder-precise
snuggle: snuggle-en
utrs: utrs-primary
wildcat: dannyb

Please let me know if you need additional assistance with any of the 
above.  The 'Huggle' instance remains unclaimed -- it is being tracked 
in https://phabricator.wikimedia.org/T157710.


Documentation for this process can be found at 
https://phabricator.wikimedia.org/T143349.


-Andrew


___
Labs-announce mailing list
labs-annou...@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-announce
___
Labs-l mailing list
Labs-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-l

Re: [Labs-l] labs-l:如何处理员工违纪问题? 6dkn7m

[Andrew Bogott]

On 2/16/17 8:07 AM, Maximilian Doerr wrote:

How’d it get on the list?  Isn’t it moderated?
By default, list members are able to post unmoderated.  Posts from 
non-list-members ought to be automatically discarded.  I'm not clear how 
this particular message got through -- I don't see evidence that that 
user has an account but I also don't trust mailman to handle non-ascii 
characters properly.


If we start getting more than one spam message per year, I'll 
investigate further :)


-Andrew




Cyberpower678
English Wikipedia Account Creation Team
English Wikipedia Administrator
Global User Renamer

On Feb 16, 2017, at 09:05, Brad Jorsch (Anomie) 
> wrote:


2017-02-15 23:38 GMT-05:00 Maximilian Doerr 
>:


What the hell is this?


Probably a macro virus trying to spread itself, although it's 
possible it's just a completely off-topic post (Google Translate 
makes the text sound like some workers' rights thing).



--
Brad Jorsch (Anomie)
Senior Software Engineer
Wikimedia Foundation
___
Labs-l mailing list
Labs-l@lists.wikimedia.org 
https://lists.wikimedia.org/mailman/listinfo/labs-l




___
Labs-l mailing list
Labs-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-l



___
Labs-l mailing list
Labs-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-l

Re: [Labs-l] [Labs-announce] Horizon upgrade tomorrow (Tuesday), 16:00 UTC (DONE)

[Andrew Bogott]

This upgrade is now complete.  I see a few immediate clumsy spots in the 
GUI that I'm going to work on ironing out today, but everything should 
now be usable.  Let me know if you run into trouble!


-Andrew


On 2/13/17 11:58 AM, Andrew Bogott wrote:
Tomorrow I'm going to upgrade https://horizon.wikimedia.org/ from 
version 'Liberty' to version 'Mitaka'.  I've scheduled an hour-long 
window for the upgrade, although the actual switch-over should be much 
shorter.  Downtime should be minimal; I'll be clearing caches as part 
of the upgrade, so login sessions will be interrupted, and performance 
may suffer a bit while the cache is cold.


The Mitaka version contains a lot of new Javascript front-end 
code.  Some of the new interfaces (in particular the one for creating 
new VMs) are a bit strange and will take some getting used to.  
Overall, though, the Mitaka upgrade is an improvement, and I have high 
hopes that it will considerably improve performance.


I've tested the new version extensively, but there may well be 
breakages around the corners of some of our custom features, so don't 
hesitate to notify me if you see new, bad behavior.


-Andrew




___
Labs-announce mailing list
labs-annou...@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-announce
___
Labs-l mailing list
Labs-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-l

[Labs-l] [Labs-announce] Horizon upgrade tomorrow (Tuesday), 16:00 UTC

[Andrew Bogott]

Tomorrow I'm going to upgrade https://horizon.wikimedia.org/ from 
version 'Liberty' to version 'Mitaka'.  I've scheduled an hour-long 
window for the upgrade, although the actual switch-over should be much 
shorter.  Downtime should be minimal; I'll be clearing caches as part of 
the upgrade, so login sessions will be interrupted, and performance may 
suffer a bit while the cache is cold.


The Mitaka version contains a lot of new Javascript front-end 
code.  Some of the new interfaces (in particular the one for creating 
new VMs) are a bit strange and will take some getting used to.  Overall, 
though, the Mitaka upgrade is an improvement, and I have high hopes that 
it will considerably improve performance.


I've tested the new version extensively, but there may well be 
breakages around the corners of some of our custom features, so don't 
hesitate to notify me if you see new, bad behavior.


-Andrew


___
Labs-announce mailing list
labs-annou...@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-announce
___
Labs-l mailing list
Labs-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-l

Re: [Labs-l] Mail to shell account name not working

[Andrew Bogott]

On 2/7/17 5:14 PM, MarcoAurelio wrote:
Thanks Tim. However when I go to the message I sent, I just see the 
mail I sent and not any received and marked as read. If you say the 
system is working well I'll trust you though. Thanks for your time. 
Regards.
In my experience when I send an email via gmail to a list that I'm 
subscribed to, I NEVER receive a copy of the message that I sent in my 
inbox.  I think that's google being efficient and preventing you from 
holding multiple copies.


Here's some rambling on the subject: 
https://productforums.google.com/forum/#!topic/gmail/MHEZDkqCHEY


This is the behavior you're seeing, is it not?  It drives me crazy.

-Andrew





El El mar, 7 feb 2017 a las 21:10, Tim Landscheidt 
> escribió:


MarcoAurelio > wrote:

> I've tested sending an email to maure...@tools.wmflabs.org
 to see if that
> worked, but I've not received any mail despite having verified
my wikitech
> email account. Are the docs outdated here or there's an error in
the mail
> processor?

/var/log/exim4/mainlog on tools-mail shows:

| […]
| 2017-02-07 19:12:10 1cbBBS-0005Ym-Lw DKIM: d=gmail.com
 s=20161025 c=relaxed/relaxed a=rsa-sha256
[verification succeeded]
| 2017-02-07 19:12:10 1cbBBS-0005Ym-Lw <= strig...@gmail.com
 H=mail-ot0-f173.google.com
 [74.125.82.173] P=esmtp S=2424
id=[…]
| 2017-02-07 19:12:11 1cbBBS-0005Ym-Lw => strig...@gmail.com
 > R=dnslookup T=remote_smtp
H=gmail-smtp-in.l.google.com 
[74.125.22.26] X=TLS1.2:RSA_AES_128_CBC_SHA1:128 C="250 2.0.0 OK
1486494731 e124si3678103qkf.17 - gsmtp"
| […]

I. e., the mail was successfully received from Gmail and
sent back to Gmail.  IIRC Gmail marks mails sent by you as
read ("feature").

A test for s...@tools.wmflabs.org 
worked fine.

Tim


___
Labs-l mailing list
Labs-l@lists.wikimedia.org 
https://lists.wikimedia.org/mailman/listinfo/labs-l

--
M. A.


___
Labs-l mailing list
Labs-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-l



___
Labs-l mailing list
Labs-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-l

[Labs-l] [Labs-announce] REMINDER: Ubuntu Precise instances on Labs will be shut down at the end of March

[Andrew Bogott]

This is a periodic reminder about the upcoming deprecation of Ubuntu 
Precise.  Details below, with a list of affected instances at the very 
bottom.


If you are affected by this issue, please update 
https://phabricator.wikimedia.org/T143349 with details about your plans.


Thank you!

---

On March 31st, 2017, all Precise instances running on Labs will be shut 
down.


In April, Ubuntu 12.04 Precise will reach its official End of Life. That 
means no more security upgrades, so after that date servers and VMs 
running Precise will no longer be welcome on Wikimedia infrastructure.


Toollabs users are already on notice that web services and tools need to 
be upgraded to run on Trusty, Jessie, or Kubernetes hosts. Outside of 
Toollabs, there are still 26 Precise instances running on Labs, listed 
below.


If you are responsible for any of these VMs, please begin migrating your 
work to new instances as soon as possible.  Due to security concerns, 
this is a hard deadline -- instances WILL be shut down on March 31st, 
and their data destroyed a few weeks later.  The Labs team will be 
dedicating quite a bit of time to assisting with this migration in the 
coming months, so don't hesitate to ask for help if you run into trouble 
or the migration process is unclear.  Do start thinking about this now 
so you don't get lost in a stampede of support requests in March.


If possible you'll want to migrate to instances running Debian Jessie. 
Trusty will sunset in 2019 but support for Jessie (both from the WMF and 
from upstream maintainers) is likely to continue well after that.


---

Here are all the instances subject to shutdown:

account-creation-assistance: accounts-db2
analytics: limn1
bots: wm-bot
contributors: contributors-metrics
editor-engagement: docs
editor-engagement: mwui
huggle: huggle
maps: maps-tiles1
mediahandler-tests: mediahandler-tests-static
openstack: labs-vmbuilder-precise
otrs: otrs-test2
signwriting: signwriting-ase-wiki
snuggle: snuggle-en
social-tools: social-tools1
sugarcrm: officetools
utrs: utrs-primary
visualeditor: towtruck
wikidata-dev: wikidata-wdq-mm
wikisource-dev: wikisource-dev
wikisource-tools: wsexport
wikistream: wikistream-web
wildcat: dannyb-large
wildcat: dannyb-test
wildcat: dannyb
wlmjudging: wlm-mysql-master
wlmjudging: wlm-apache1


___
Labs-announce mailing list
labs-annou...@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-announce
___
Labs-l mailing list
Labs-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-l

[Labs-l] [Labs-announce] Correction: Ubuntu PRECISE instances on Labs will be shut down at the end of March

[Andrew Bogott]

Apologies, I just noticed that this email was sent with an incorrect 
subject line.  It's only Ubuntu Precise instances that will be shut 
down... Trusty will live on.


On 12/19/16 11:47 AM, Andrew Bogott wrote:
On March 31st, 2017, all Precise instances running on Labs will be 
shut down.


In April, Ubuntu 12.04 Precise will reach its official End of Life. 
That means no more security upgrades, so after that date servers and 
VMs running Precise will no longer be welcome on Wikimedia 
infrastructure.


Toollabs users are already on notice that web services and tools need 
to be upgraded to run on Trusty, Jessie, or Kubernetes hosts. Outside 
of Toollabs, there are still 30-some Precise instances running on 
Labs, listed below.


If you are responsible for any of these VMs, please begin migrating 
your work to new instances as soon as possible.  Due to security 
concerns, this is a hard deadline -- instances WILL be shut down on 
March 31st, and their data destroyed a few weeks later.  The Labs team 
will be dedicating quite a bit of time to assisting with this 
migration in the coming months, so don't hesitate to ask for help if 
you run into trouble or the migration process is unclear.  Do start 
thinking about this now so you don't get lost in a stampede of support 
requests in March.


If possible you'll want to migrate to instances running Debian 
Jessie.  Trusty will sunset in 2019 but support for Jessie (both from 
the WMF and from upstream maintainers) is likely to continue well 
after that.


Here are all the instances subject to shutdown:

account-creation-assistance: accounts-db2
analytics: limn1
bots: wm-bot
contributors: contributors-metrics
editor-engagement: docs
editor-engagement: ee-flow-extra
editor-engagement: ee-flow
editor-engagement: mwui
fastcci: fastcci-master
fastcci: fastcci-worker1
huggle: huggle
maps: maps-tiles1
maps: maps-wma1
mediahandler-tests: mediahandler-tests-static
monitoring: filippo-test-precise2
otrs: otrs-test2
signwriting: signwriting-ase-wiki
snuggle: snuggle-en
social-tools: social-tools1
sugarcrm: officetools
utrs: utrs-primary
visualeditor: towtruck
wikidata-dev: wikidata-suggester
wikidata-dev: wikidata-wdq-mm
wikimania-support: wikimania-scholarships
wikisource-dev: wikisource-dev
wikisource-tools: wsexport
wikistream: wikistream-web
wildcat: dannyb-large
wildcat: dannyb-test
wildcat: dannyb
wlmjudging: wlm-mysql-master
wlmjudging: wlm-apache1




___
Labs-announce mailing list
labs-annou...@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-announce
___
Labs-l mailing list
Labs-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-l

[Labs-l] [Labs-announce] Ubuntu instances on Labs will be shut down at the end of March

[Andrew Bogott]

On March 31st, 2017, all Precise instances running on Labs will be shut 
down.


In April, Ubuntu 12.04 Precise will reach its official End of Life. That 
means no more security upgrades, so after that date servers and VMs 
running Precise will no longer be welcome on Wikimedia infrastructure.


Toollabs users are already on notice that web services and tools need to 
be upgraded to run on Trusty, Jessie, or Kubernetes hosts. Outside of 
Toollabs, there are still 30-some Precise instances running on Labs, 
listed below.


If you are responsible for any of these VMs, please begin migrating your 
work to new instances as soon as possible.  Due to security concerns, 
this is a hard deadline -- instances WILL be shut down on March 31st, 
and their data destroyed a few weeks later.  The Labs team will be 
dedicating quite a bit of time to assisting with this migration in the 
coming months, so don't hesitate to ask for help if you run into trouble 
or the migration process is unclear.  Do start thinking about this now 
so you don't get lost in a stampede of support requests in March.


If possible you'll want to migrate to instances running Debian Jessie.  
Trusty will sunset in 2019 but support for Jessie (both from the WMF and 
from upstream maintainers) is likely to continue well after that.


Here are all the instances subject to shutdown:

account-creation-assistance: accounts-db2
analytics: limn1
bots: wm-bot
contributors: contributors-metrics
editor-engagement: docs
editor-engagement: ee-flow-extra
editor-engagement: ee-flow
editor-engagement: mwui
fastcci: fastcci-master
fastcci: fastcci-worker1
huggle: huggle
maps: maps-tiles1
maps: maps-wma1
mediahandler-tests: mediahandler-tests-static
monitoring: filippo-test-precise2
otrs: otrs-test2
signwriting: signwriting-ase-wiki
snuggle: snuggle-en
social-tools: social-tools1
sugarcrm: officetools
utrs: utrs-primary
visualeditor: towtruck
wikidata-dev: wikidata-suggester
wikidata-dev: wikidata-wdq-mm
wikimania-support: wikimania-scholarships
wikisource-dev: wikisource-dev
wikisource-tools: wsexport
wikistream: wikistream-web
wildcat: dannyb-large
wildcat: dannyb-test
wildcat: dannyb
wlmjudging: wlm-mysql-master
wlmjudging: wlm-apache1


___
Labs-announce mailing list
labs-annou...@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-announce
___
Labs-l mailing list
Labs-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-l

Re: [Labs-l] [Labs-announce] More reboots today, 18:00 -- FINISHED

[Andrew Bogott]

All reboots are complete now.  Sorry for the commotion!

-Andrew


On 11/2/16 12:57 PM, Andrew Bogott wrote:

This is starting now.

On 11/2/16 10:43 AM, Andrew Bogott wrote:
Good news:  I've completed patching all the individual VMs listed in 
previous emails.


Bad news:  We need to reboot a couple of the physical hosts yet 
again. A complete list of instances that will be affected follows below.


The reboots will start at 18:00 UTC, in about two and a half hours.

As usual, Tools users can mostly ignore this, but should expect the 
occasional hiccup.


Affected instances:


accounts-db3   | account-creation-assistance | ACTIVE | - 
  | Running | public=10.68.20.23  |
af-master02| automation-framework| ACTIVE | - 
  | Running | public=10.68.22.243 |
af-puppetmaster01  | automation-framework| ACTIVE | - 
  | Running | public=10.68.19.175 |
boo  | discovery-stats | ACTIVE | - | 
Running | public=10.68.21.207 |
captcha-static-01| privpol-captcha | ACTIVE | - | 
Running | public=10.68.20.188 |
ci-jessie-wikimedia-408952 | contintcloud| ACTIVE | - 
  | Running | public=10.68.16.180 |
ci-trusty-wikimedia-408958 | contintcloud| ACTIVE | - 
  | Running | public=10.68.17.103 |
danny  | gitblit | ACTIVE | - 
  | Running | public=10.68.16.58  |
deployment-apertium02| deployment-prep | ACTIVE | - | 
Running | public=10.68.22.254 |
deployment-db03| deployment-prep | ACTIVE | - 
  | Running | public=10.68.23.30  |
deployment-db04  | deployment-prep | ACTIVE | - | 
Running | public=10.68.18.35  |
deployment-jobrunner02   | deployment-prep | ACTIVE | - | 
Running | public=10.68.19.42  |
deployment-kafka05   | deployment-prep | ACTIVE | - | 
Running | public=10.68.21.106 |
deployment-mediawiki05 | deployment-prep | ACTIVE | - 
  | Running | public=10.68.22.21  |
deployment-mediawiki06   | deployment-prep | ACTIVE | - | 
Running | public=10.68.19.241 |
deployment-pdfrender   | deployment-prep | ACTIVE | - 
  | Running | public=10.68.17.167 |
deployment-phab02| deployment-prep | ACTIVE | - | 
Running | public=10.68.19.232 |
deployment-prometheus01| deployment-prep | ACTIVE | - 
  | Running | public=10.68.20.247 |
deployment-tin | deployment-prep | ACTIVE | - 
  | Running | public=10.68.21.205 |
dewiktionary   | cognate | ACTIVE | - 
  | Running | public=10.68.20.214 |
discovery-prototypes | shiny-r | ACTIVE | - | 
Running | public=10.68.16.238 |
drmf2017   | math| ACTIVE | - 
  | Running | public=10.68.19.68  |
encoding02 | video   | ACTIVE | - 
  | Running | public=10.68.21.74  |
encoding03   | video   | ACTIVE | - | 
Running | public=10.68.20.55  |
etcd06   | etcd| ACTIVE | - | 
Running | public=10.68.18.90  |
fawp | fa-wp   | ACTIVE | - | 
Running | public=10.68.17.108 |
fawp-tts   | fa-wp   | ACTIVE | - 
  | Running | public=10.68.19.133 |
frwiktionary   | cognate | ACTIVE | - 
  | Running | public=10.68.20.14  |
gerrit-mysql | git | ACTIVE | - | 
Running | public=10.68.23.211 |
gerrit-test3   | git | ACTIVE | - 
  | Running | public=10.68.23.148, 208.80.155.149 |
integration-puppetmaster01   | integration | ACTIVE | - | 
Running | public=10.68.22.41  |
integration-slave-jessie-1003| integration | ACTIVE | - | 
Running | public=10.68.21.145 |
integration-slave-jessie-android | integration | ACTIVE | - | 
Running | public=10.68.19.239 |
jenkins-slave-01 | git | ACTIVE | - | 
Running | public=10.68.22.0   |
kafka501   | analytics   | ACTIVE | - 
  | Running | public=10.68.20.29  |
kafka601 | analytics   | ACTIVE | - | 
Running | public=10.68.21.56  |
labs-traefikbuild  | openstack   | ACTIVE | - 
  | Running | public=10.68.19.222 |
language-mleb-legacy   | language| ACTIVE

Re: [Labs-l] [Labs-announce] More reboots today, 18:00 UTC :(

[Andrew Bogott]

This is starting now.

On 11/2/16 10:43 AM, Andrew Bogott wrote:
Good news:  I've completed patching all the individual VMs listed in 
previous emails.


Bad news:  We need to reboot a couple of the physical hosts yet again. 
A complete list of instances that will be affected follows below.


The reboots will start at 18:00 UTC, in about two and a half hours.

As usual, Tools users can mostly ignore this, but should expect the 
occasional hiccup.


Affected instances:


accounts-db3   | account-creation-assistance | ACTIVE | - 
  | Running | public=10.68.20.23  |
af-master02| automation-framework| ACTIVE | - 
  | Running | public=10.68.22.243 |
af-puppetmaster01  | automation-framework| ACTIVE | - 
  | Running | public=10.68.19.175 |
boo  | discovery-stats | ACTIVE | - | 
Running | public=10.68.21.207 |
captcha-static-01| privpol-captcha | ACTIVE | - | 
Running | public=10.68.20.188 |
ci-jessie-wikimedia-408952 | contintcloud| ACTIVE | - 
  | Running | public=10.68.16.180 |
ci-trusty-wikimedia-408958 | contintcloud| ACTIVE | - 
  | Running | public=10.68.17.103 |
danny  | gitblit | ACTIVE | - 
  | Running | public=10.68.16.58  |
deployment-apertium02| deployment-prep | ACTIVE | - | 
Running | public=10.68.22.254 |
deployment-db03| deployment-prep | ACTIVE | - 
  | Running | public=10.68.23.30  |
deployment-db04  | deployment-prep | ACTIVE | - | 
Running | public=10.68.18.35  |
deployment-jobrunner02   | deployment-prep | ACTIVE | - | 
Running | public=10.68.19.42  |
deployment-kafka05   | deployment-prep | ACTIVE | - | 
Running | public=10.68.21.106 |
deployment-mediawiki05 | deployment-prep | ACTIVE | - 
  | Running | public=10.68.22.21  |
deployment-mediawiki06   | deployment-prep | ACTIVE | - | 
Running | public=10.68.19.241 |
deployment-pdfrender   | deployment-prep | ACTIVE | - 
  | Running | public=10.68.17.167 |
deployment-phab02| deployment-prep | ACTIVE | - | 
Running | public=10.68.19.232 |
deployment-prometheus01| deployment-prep | ACTIVE | - 
  | Running | public=10.68.20.247 |
deployment-tin | deployment-prep | ACTIVE | - 
  | Running | public=10.68.21.205 |
dewiktionary   | cognate | ACTIVE | - 
  | Running | public=10.68.20.214 |
discovery-prototypes | shiny-r | ACTIVE | - | 
Running | public=10.68.16.238 |
drmf2017   | math| ACTIVE | - 
  | Running | public=10.68.19.68  |
encoding02 | video   | ACTIVE | - 
  | Running | public=10.68.21.74  |
encoding03   | video   | ACTIVE | - | 
Running | public=10.68.20.55  |
etcd06   | etcd| ACTIVE | - | 
Running | public=10.68.18.90  |
fawp | fa-wp   | ACTIVE | - | 
Running | public=10.68.17.108 |
fawp-tts   | fa-wp   | ACTIVE | - 
  | Running | public=10.68.19.133 |
frwiktionary   | cognate | ACTIVE | - 
  | Running | public=10.68.20.14  |
gerrit-mysql | git | ACTIVE | - | 
Running | public=10.68.23.211 |
gerrit-test3   | git | ACTIVE | - 
  | Running | public=10.68.23.148, 208.80.155.149 |
integration-puppetmaster01   | integration | ACTIVE | - | 
Running | public=10.68.22.41  |
integration-slave-jessie-1003| integration | ACTIVE | - | 
Running | public=10.68.21.145 |
integration-slave-jessie-android | integration | ACTIVE | - | 
Running | public=10.68.19.239 |
jenkins-slave-01 | git | ACTIVE | - | 
Running | public=10.68.22.0   |
kafka501   | analytics   | ACTIVE | - 
  | Running | public=10.68.20.29  |
kafka601 | analytics   | ACTIVE | - | 
Running | public=10.68.21.56  |
labs-traefikbuild  | openstack   | ACTIVE | - 
  | Running | public=10.68.19.222 |
language-mleb-legacy   | language| ACTIVE | - 
  | Running | public=10.68.22.180 |
maps-graphoid  | maps-team

Re: [Labs-l] [Labs-announce] Many instance reboots Tuesday, 2016-11-01 at 18:00 UTC

[Andrew Bogott]

This is now mostly done.

Unfortunately, there are quite a few instances with broken packages, 
full disks, etc. that I was unable to patch automatically, so I'm 
logging into them one-by-one and fixing things by hand.  I'm about 
half-way done with that, and I don't expect to have this done tonight, 
so the instances listed below should expect a reboot sometime in the 
next 24 hours or so.


I'll send another email when I'm well and truly done with this.

Instances still in need of patching:

deployment-zotero01
limn1
lsg-01
map
maps-tiles1
mediahandler-tests-static
mw-base
mw-expt
mwui
nomad
os-instance-control-01
os-instance-db-01
osmit-due
phab
phab-03
phab-tin
phabricator
phlogiston-3
pole
proofreadpage
reading-web-research
reading-web-staging-3
relforge-search
sentry-alpha
sm-puppetmaster-trusty2
sm1
snuggle-en
st-trusty
striker-deploy03
striker-uwsgi02
telnet2
toolsbeta-puppetmaster3
trusty
trusty-update
undeltest
utrs-secondary
vem
vem2
vicky
visualeditor-switch
visualeditor-test
wbdocs
wikidata-suggester
wikimania-scholarships
wm-bot
xtools-live01




On 10/26/16 3:32 PM, Andrew Bogott wrote:
In order to address a security issue, we are patching and rebooting 
many physical Labs hosts.  The final round of reboots will take place 
on Tuesday, beginning at 18:00 UTC (11:00 AM San Francisco time).


All virtualization hosts except for labvirt1001 will be rebooted on 
Tuesday.  That means that all instances EXCEPT those on labvirt1001* 
will be rebooted.  In general a reboot involves 5-20 minutes of 
downtime for the instance; in unusual cases a second reboot may be 
necessary.


The labs operations team will manage tool queues and services, so 
interruption to services on Tools should be minimal.


Apologies in advance for the downtime.

-Andrew





___
Labs-announce mailing list
labs-annou...@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-announce
___
Labs-l mailing list
Labs-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-l

Re: [Labs-l] [Labs-announce] Many instance reboots Tuesday, 2016-11-01 at 18:00 UTC

[Andrew Bogott]

Reminder:  This is happening in about 90 minutes.

On 10/26/16 3:32 PM, Andrew Bogott wrote:
In order to address a security issue, we are patching and rebooting 
many physical Labs hosts.  The final round of reboots will take place 
on Tuesday, beginning at 18:00 UTC (11:00 AM San Francisco time).


All virtualization hosts except for labvirt1001 will be rebooted on 
Tuesday.  That means that all instances EXCEPT those on labvirt1001* 
will be rebooted.  In general a reboot involves 5-20 minutes of 
downtime for the instance; in unusual cases a second reboot may be 
necessary.


The labs operations team will manage tool queues and services, so 
interruption to services on Tools should be minimal.


Apologies in advance for the downtime.

-Andrew






* Instances on labvirt1001 have already been rebooted.  They are:

 citoidtest (services)
 deployment-elastic08 (deployment-prep)
 deployment-pdf01 (deployment-prep)
 deployment-urldownloader (deployment-prep)
 druid1 (analytics)
 ee-flow-extra (editor-engagement)
 elastic1 (analytics)
 etcd01 (etcd)
 etcd03 (etcd)
 etherpad01 (etherpad)
 huggle-d2 (huggle)
 language-replag-slave (language)
 maps-tiles2 (maps)
 novaproxy-01 (project-proxy)
 osmit-tre (osmit)
 paws-base-01 (paws)
 perf-testing (mobile)
 precise (debdeploy)
 puppet-mailman (puppet)
 tools-bastion-02 (tools)
 toolsbeta-exec-1209 (toolsbeta)
 toolsbeta-vagrant3-scfc (toolsbeta)
 wikidata-mobile (wikidata-dev)
 wmt-exec (wmt)






___
Labs-announce mailing list
labs-annou...@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-announce
___
Labs-l mailing list
Labs-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-l

Re: [Labs-l] [Labs-announce] Labs and Wikitech outages Friday, 2016-10-28 at 15:00 UTC

[Andrew Bogott]

All done.  Sorry for the interruption!

-Andrew

On 10/26/16 3:22 PM, Andrew Bogott wrote:
In order to address a security issue, we are patching and rebooting 
many physical Labs hosts.  The next round of this will begin at 15:00 
UTC on Friday, which is 8:00AM San Francisco time.


Interruptions will include:

- A labs-wide network outage

- Wikitech downtime

- Horizon downtime

- Possible failures in new instance creation

None of the above should last more than a few minutes.  For the most 
part Labs instances will be unaware of the outages, and no action is 
required from users.


Apologies for any inconvenience caused!


-Andrew




___
Labs-announce mailing list
labs-annou...@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-announce
___
Labs-l mailing list
Labs-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-l

Re: [Labs-l] [Labs-announce] Labs and Wikitech outages Friday, 2016-10-28 at 15:00 UTC

[Andrew Bogott]

Reminder:  These reboots will start in 30 minutes.


On 10/26/16 3:22 PM, Andrew Bogott wrote:
In order to address a security issue, we are patching and rebooting 
many physical Labs hosts.  The next round of this will begin at 15:00 
UTC on Friday, which is 8:00AM San Francisco time.


Interruptions will include:

- A labs-wide network outage

- Wikitech downtime

- Horizon downtime

- Possible failures in new instance creation

None of the above should last more than a few minutes.  For the most 
part Labs instances will be unaware of the outages, and no action is 
required from users.


Apologies for any inconvenience caused!


-Andrew




___
Labs-announce mailing list
labs-annou...@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-announce
___
Labs-l mailing list
Labs-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-l

[Labs-l] [Labs-announce] Many instance reboots Tuesday, 2016-11-01 at 18:00 UTC

[Andrew Bogott]

In order to address a security issue, we are patching and rebooting many 
physical Labs hosts.  The final round of reboots will take place on 
Tuesday, beginning at 18:00 UTC (11:00 AM San Francisco time).


All virtualization hosts except for labvirt1001 will be rebooted on 
Tuesday.  That means that all instances EXCEPT those on labvirt1001* 
will be rebooted.  In general a reboot involves 5-20 minutes of downtime 
for the instance; in unusual cases a second reboot may be necessary.


The labs operations team will manage tool queues and services, so 
interruption to services on Tools should be minimal.


Apologies in advance for the downtime.

-Andrew






* Instances on labvirt1001 have already been rebooted.  They are:

 citoidtest (services)
 deployment-elastic08 (deployment-prep)
 deployment-pdf01 (deployment-prep)
 deployment-urldownloader (deployment-prep)
 druid1 (analytics)
 ee-flow-extra (editor-engagement)
 elastic1 (analytics)
 etcd01 (etcd)
 etcd03 (etcd)
 etherpad01 (etherpad)
 huggle-d2 (huggle)
 language-replag-slave (language)
 maps-tiles2 (maps)
 novaproxy-01 (project-proxy)
 osmit-tre (osmit)
 paws-base-01 (paws)
 perf-testing (mobile)
 precise (debdeploy)
 puppet-mailman (puppet)
 tools-bastion-02 (tools)
 toolsbeta-exec-1209 (toolsbeta)
 toolsbeta-vagrant3-scfc (toolsbeta)
 wikidata-mobile (wikidata-dev)
 wmt-exec (wmt)




___
Labs-announce mailing list
labs-annou...@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-announce
___
Labs-l mailing list
Labs-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-l

[Labs-l] [Labs-announce] Labs and Wikitech outages Friday, 2016-10-28 at 15:00 UTC

[Andrew Bogott]

In order to address a security issue, we are patching and rebooting many 
physical Labs hosts.  The next round of this will begin at 15:00 UTC on 
Friday, which is 8:00AM San Francisco time.


Interruptions will include:

- A labs-wide network outage

- Wikitech downtime

- Horizon downtime

- Possible failures in new instance creation

None of the above should last more than a few minutes.  For the most 
part Labs instances will be unaware of the outages, and no action is 
required from users.


Apologies for any inconvenience caused!


-Andrew


___
Labs-announce mailing list
labs-annou...@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-announce
___
Labs-l mailing list
Labs-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-l

[Labs-l] new puppetmaster version on Labs

[Andrew Bogott]

This email is purely informational and doesn't require any action on 
your part :)



Operations has recently upgraded our production puppetmaster to version 
3.8.5.  In order to keep things in sync, I'm doing the following:


- (done) Upgrading the general Labs puppetmaster to 3.8.5

- (done) Updating the official puppetmaster (and client) packages in our 
repo to 3.8.5.  That means that any newly created puppetmasters will get 
those packages by default.


- (doing) Actively upgrading all existing in-labs Trusty puppetmasters 
(using role::puppet::self or puppetmaster::standalone) to 3.8.5.  
(Jessie puppetmasters should all be running the latest version already.)


That last change will result in two changes in puppet behavior on local 
puppetmasters:


1) Puppet runs may throw a deprecation warning, "Warning: Setting 
templatedir is deprecated."  That can be safely ignored until we 
rearrange the puppet repo to resolve it.


2) Puppet runs may include one additional action, 
"Puppetmaster::Passenger/Service[puppetmaster]/ensure: ensure changed 
'running' to 'stopped'"  This is dumb, and caused by a bug in the 
puppetmaster packages, but may also be safely ignored.


If you encounter any bad behavior apart from those last two items, 
please let me know!  In general, though, I expect this change to have no 
significant effect on anyone.



-Andrew


___
Labs-l mailing list
Labs-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-l

Re: [Labs-l] [Labs-announce] Rescheduled: Some instance reboots next Tuesday, 2016-10-25 at 18:00 UTC DONE

[Andrew Bogott]

This is done now -- everything should be back up and running as usual.  
Please let me know if you find an instance that didn't survive the 
operation.


-Andrew


On 10/25/16 12:51 PM, Andrew Bogott wrote:
Reminder:  This is happening in 10 minutes.  If you are using 
tools-bastion-02 best to finish up and log off now.


-A


On 10/20/16 11:27 AM, Andrew Bogott wrote:
This reboot window has been rescheduled for next Tuesday.  Sorry for 
the confusion!


-Andrew


On 10/14/16 3:12 PM, Andrew Bogott wrote:
During the upcoming weeks we'll be performing kernel upgrades on the 
Labs virtualization hosts. The first host will be upgraded this 
Thursday at 18:00 UTC (11AM PDT).


In general, Tools users will be unaffected by this reboot.  It will, 
however, include a reboot of tools-bastion-02, so active login 
sessions may be interrupted.


In addition to Tools instances, the following instances will also be 
rebooted:


citoidtest (services)
deployment-elastic08 (deployment-prep)
deployment-pdf01 (deployment-prep)
deployment-urldownloader (deployment-prep)
druid1 (analytics)
ee-flow-extra (editor-engagement)
elastic1 (analytics)
etcd01 (etcd)
etcd03 (etcd)
etherpad01 (etherpad)
huggle-d2 (huggle)
language-replag-slave (language)
maps-tiles2 (maps)
novaproxy-01 (project-proxy)
osmit-tre (osmit)
paws-base-01 (paws)
perf-testing (mobile)
precise (debdeploy)
puppet-mailman (puppet)
tools-bastion-02 (tools)
toolsbeta-exec-1209 (toolsbeta)
toolsbeta-vagrant3-scfc (toolsbeta)
wikidata-mobile (wikidata-dev)
wmt-exec (wmt)









___
Labs-announce mailing list
labs-annou...@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-announce
___
Labs-l mailing list
Labs-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-l

[Labs-l] [Labs-announce] Rescheduled: Some instance reboots next Tuesday, 2016-10-25 at 18:00 UTC

[Andrew Bogott]

This reboot window has been rescheduled for next Tuesday.  Sorry for the 
confusion!


-Andrew


On 10/14/16 3:12 PM, Andrew Bogott wrote:
During the upcoming weeks we'll be performing kernel upgrades on the 
Labs virtualization hosts.  The first host will be upgraded this 
Thursday at 18:00 UTC (11AM PDT).


In general, Tools users will be unaffected by this reboot.  It will, 
however, include a reboot of tools-bastion-02, so active login 
sessions may be interrupted.


In addition to Tools instances, the following instances will also be 
rebooted:


citoidtest (services)
deployment-elastic08 (deployment-prep)
deployment-pdf01 (deployment-prep)
deployment-urldownloader (deployment-prep)
druid1 (analytics)
ee-flow-extra (editor-engagement)
elastic1 (analytics)
etcd01 (etcd)
etcd03 (etcd)
etherpad01 (etherpad)
huggle-d2 (huggle)
language-replag-slave (language)
maps-tiles2 (maps)
novaproxy-01 (project-proxy)
osmit-tre (osmit)
paws-base-01 (paws)
perf-testing (mobile)
precise (debdeploy)
puppet-mailman (puppet)
tools-bastion-02 (tools)
toolsbeta-exec-1209 (toolsbeta)
toolsbeta-vagrant3-scfc (toolsbeta)
wikidata-mobile (wikidata-dev)
wmt-exec (wmt)





___
Labs-announce mailing list
labs-annou...@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-announce
___
Labs-l mailing list
Labs-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-l

Re: [Labs-l] [Labs-announce] New Puppet configuration GUI on horizon (UPDATE)

[Andrew Bogott]

On 9/21/16 2:22 PM, Andrew Bogott wrote:


== What's coming up ==

Early in October (probably around the 5th), I'll be removing the 
puppet and hiera interfaces from wikitech, and migrating any settings 
or content from there to the new Horizon system.  With luck the 
transitional period (when the wikitech settings are still in force but 
not actually visible) will be brief.  After that, we'll have a single, 
Horizon-based config that should generally work more consistently than 
either the old Wikitech system or the current mishmash.


We've just migrated all puppet role settings from Wikitech to Horizon, 
and removed the Wikitech puppet configuration UI.  So, from now on, 
Horizon holds the one canonical representation of what roles are applied 
to a given instance.


Hiera config is still a composite of the Wikitech hiera pages and the 
Horizon settings.  We may actually keep it this way indefinitely since 
there are some advantages to having free-form hiera settings stored on a 
wiki page with an edit history.  I'm still on the fence about this, so I 
welcome your thoughts.


-Andrew

___
Labs-announce mailing list
labs-annou...@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-announce
___
Labs-l mailing list
Labs-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-l

[Labs-l] [Labs-announce] Some instance reboots next Thursday, 2016-10-20 at 18:00 UTC

[Andrew Bogott]

During the upcoming weeks we'll be performing kernel upgrades on the 
Labs virtualization hosts.  The first host will be upgraded this 
Thursday at 18:00 UTC (11AM PDT).


In general, Tools users will be unaffected by this reboot.  It will, 
however, include a reboot of tools-bastion-02, so active login sessions 
may be interrupted.


In addition to Tools instances, the following instances will also be 
rebooted:


citoidtest (services)
deployment-elastic08 (deployment-prep)
deployment-pdf01 (deployment-prep)
deployment-urldownloader (deployment-prep)
druid1 (analytics)
ee-flow-extra (editor-engagement)
elastic1 (analytics)
etcd01 (etcd)
etcd03 (etcd)
etherpad01 (etherpad)
huggle-d2 (huggle)
language-replag-slave (language)
maps-tiles2 (maps)
novaproxy-01 (project-proxy)
osmit-tre (osmit)
paws-base-01 (paws)
perf-testing (mobile)
precise (debdeploy)
puppet-mailman (puppet)
tools-bastion-02 (tools)
toolsbeta-exec-1209 (toolsbeta)
toolsbeta-vagrant3-scfc (toolsbeta)
wikidata-mobile (wikidata-dev)
wmt-exec (wmt)



___
Labs-announce mailing list
labs-annou...@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-announce
___
Labs-l mailing list
Labs-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-l

Re: [Labs-l] [Labs-announce] [DONE] We are removing inactive projects.

[Andrew Bogott]

This cleanup is now complete. We freed the following resources:

27 projects (12.5% of total projects)
49 instances (7% of all instances labs-wide)
87 virtual CPUs (5% of all used)
172.5 GiB of allocated RAM (5% of all allocated RAM)

In addition to the things that I removed according to the formal 
process, many other instances were voluntarily cleared from active 
projects as a result of my prompting, so the actual savings are quite a 
bit larger than the numbers reported above.


Thanks for everyone's help and patience with this process!  The Labs 
team will probably institute a formal, recurring (most likely annual) 
reclamation process like this one to prevent the future backlog from 
getting so big.  I welcome your comments, both positive and negative, 
about how this went.


-Andrew


On 10/5/16 10:31 AM, Andrew Bogott wrote:

Update:

It's a month later, and I'm going to start deleting instances in 
unclaimed projects.  Here is the list of projects subject to deletion:


<https://wikitech.wikimedia.org/wiki/Purge_2016#SHUTDOWN_wlmjurytool>cuos
cvresearch
experiment
fatg
ganglia
glam
icinga
mediawiki-docker
mwreview
netflow
nginx
opengrok
oxygenguide
popcorn
pubsubhubbub
safesandbox
structured-wikiquote
tool-renewal
ve-languagetoolwikibrain
wikidata-quality
wlmjurytool

Please speak up NOW if you need any of those projects preserved.

-Andrew


On 8/30/16 10:19 AM, Andrew Bogott wrote:

Update:

There are still 30 projects that remain unclaimed on 
https://wikitech.wikimedia.org/wiki/Purge_2016.


I've emailed the admins of each of these projects directly. If I 
don't hear anything I will start to shut down instances on Thursday.



Many thanks to the vast majority of admins who provided me with 
prompt and useful feedback about project preservation and cleanup!


-Andrew




On 7/8/16 11:09 AM, Andrew Bogott wrote:
If you are exclusively a user of tool labs, you can ignore this 
email.  If you use or administer another labs project, this email 
REQUIRES ACTION ON YOUR PART.


We are reclaiming unused resources due to an ongoing shortage.

Visit this page and add a signature under projects you know to be 
active:


https://wikitech.wikimedia.org/wiki/Purge_2016

Associated wmflabs.org <http://wmflabs.org> domains are included to 
identify projects by offered services.


We are not investigating why projects are needed at this time.  If 
one person votes to preserve then we will do so in this round of 
cleanup.


In a month, projects and associated instances not claimed will be 
suspended or shutdown.  A month later if no one complains these 
projects will be deleted..


- Andrew (on behalf of all Labs Admins everywhere) 







___
Labs-announce mailing list
labs-annou...@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-announce
___
Labs-l mailing list
Labs-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-l

[Labs-l] [Labs-announce] New Puppet configuration GUI on horizon

[Andrew Bogott]

I've just deployed some new Horizon panels for puppet configuration.  If 
you care about puppetizing your instances, read on.  If you're not 
really sure what Puppet is, you can probably ignore this.



== Whats new ==

The new interface can be found in three places in horizon:

For instance-specific config:

Project->Compute->Instances->(instance name)->Puppet Configuration

For project-wide config:

Project->Puppet->Project Puppet

To configure all instances in a project beginning with a given prefix:

Project->Puppet->Prefix Puppet

As you can imagine, this can get a bit complex, with config cascading 
through some or all of the above configs (and multiple prefixes to boot).


Adding to the complexity, for now:  The old wikitech-based instance 
configuration pages (and the wiki-based hiera pages) are still active. 
You can configure your instances or projects from ALL of the places, and 
all of the different configurations will be applied.  All roles or 
classes included in any of the above interfaces will be applied to a 
given instance.  For hiera, there's an order of precedence (where the 
last things in this list clobber anything earlier in the list):


wikitech project hiera

wikitech instance hiera

horizon project hiera

horizon prefix hiera (longest prefix to shortest)

horizon instance hiera

If you want to see what's actually happening on your instance, you can 
investigate with


$ curl labcontrol1001.wikimedia.org:8100/v1//node/


== What's coming up ==

Early in October (probably around the 5th), I'll be removing the puppet 
and hiera interfaces from wikitech, and migrating any settings or 
content from there to the new Horizon system.  With luck the 
transitional period (when the wikitech settings are still in force but 
not actually visible) will be brief.  After that, we'll have a single, 
Horizon-based config that should generally work more consistently than 
either the old Wikitech system or the current mishmash.



== What you need to do ==

Please explore the new interface, and let me know what issues you find. 
(It's slow and ugly, but should be feature-complete.) Barring serious 
bugs in the new UI, any future changes to puppet or hiera configuration 
should be made using the new Horizon panels.


If you want to get ahead of the game, please go ahead and migrate 
settings from wikitech to the new interface.  That will make my upcoming 
migration task considerably easier.


The new interface is entirely role-based.  That means that if you 
currently rely on puppet classes that are /not/ role classes (e.g. they 
don't begin with 'role::' ) they will not be selectable in the new 
interfaces.  For such cases, please notify me, open a phab task, or 
submit a patch that wraps the class in a proper role.



== Self Hosted Puppetmaster / Project Puppetmaster ==

If you are running a project puppetmaster / self hosted puppetmaster, 
make sure they are running the latest puppet successfully. If not, 
changes made in horizon might not take effect



Have fun!


-Andrew


___
Labs-announce mailing list
labs-annou...@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-announce
___
Labs-l mailing list
Labs-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-l

[Labs-l] [Labs-announce] No more puppet globals in labs

[Andrew Bogott]

This email is only of interest to Labs project admins; Tool-Labs users 
can disregard this entirely.


tl;dr:

Don't set instance puppet variables using the puppet config page; 
instead use the hiera pages of the form 
https://wikitech.wikimedia.org/wiki/Hiera:/host/



The whole story:


Until recently, the recommended way to specify e.g. a puppet server on 
Labs was via a global variable set on the 'configure' panel in Wikitech.


That panel was designed several Puppet versions ago; modern puppet 
discourages use of global variables.  Instead, we're going to configure 
everything via hiera, which is the built-in cascading database puppet 
uses to configure classes.


This will have quite a few advantages, but the main one is that this 
gets us a bit closer to being able to adopt the new Horizon-based puppet 
management system that I'm in the process of writing.  It's also one of 
the last steps on the road to eliminating a whole pile of ldap dependencies.


Right now I'm hunting down all the puppet globals currently set, 
removing them, and adding equivalent hiera settings.  In a day or two 
the puppet variable feature on wikitech will disappear entirely, but 
please don't use it in the meantime.  If you're hunting for a setting, 
look on the hiera page for your project or instance.


For project-wide hiera config, pages have this format:

https://wikitech.wikimedia.org/wiki/Hiera:

For instance-specific hiera, the url looks like this:

https://wikitech.wikimedia.org/wiki/Hiera:/host/

Those hiera pages are simple yaml, and setting a single param is even 
simpler.  Here's an example, from 
https://wikitech.wikimedia.org/wiki/Hiera:Ttmserver/host/ttmserver-mediawiki01:


---
"role::salt::minions::salt_master": ttmserver-salt01.eqiad.wmflabs
"role::salt::minions::salt_finger": 
42:bb:24:ad:a5:75:86:95:db:da:dd:33:c5:90:5d:3e

That's it!

I realize this is fairly arcane -- don't hesitate to respond with 
questions or find me on IRC if you get stuck.  In a few weeks we should 
be back to having a friendly, web-based system, one that should make a 
bit more sense.


-Andrew

___
Labs-announce mailing list
labs-annou...@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-announce
___
Labs-l mailing list
Labs-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-l

Re: [Labs-l] [Labs-announce] Labs network maintenance (and outages) Wednesday 2016-08-31, 15:00 UTC (COMPLETE)

[Andrew Bogott]

This is finished, and all Labs and Tools services should be back to normal.

We didn't do a great job of disabling alerts before this started, so I 
apologize for the flood of alarming emails in your inbox.


-Andrew

On 8/31/16 9:02 AM, Andrew Bogott wrote:

Reminder:  This is starting in one hour.

On 8/25/16 8:41 AM, Andrew Bogott wrote:

Hello!

Next Wednesday we will be applying security patches to a few Labs 
hosts, including the Labs network node.  We'll do our best to avoid 
downtime, but it's very likely that there will be some brief network 
outages, possibly for as long as several minutes at a time.  This 
will affect all network traffic for all instances, both in Tools and 
in other Labs projects.


No action is required on your part, although you may want to notify 
users of your services who will be affected by downtime.


The scheduled maintenance window is:

Wednesday, 2016-08-31

15:00-17:00 UTC

08:00-10:00 PDT


-Andrew






___
Labs-announce mailing list
labs-annou...@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-announce
___
Labs-l mailing list
Labs-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-l

Re: [Labs-l] [Labs-announce] Labs network maintenance (and outages) Wednesday 2016-08-31, 15:00 UTC

[Andrew Bogott]

Reminder:  This is starting in one hour.

On 8/25/16 8:41 AM, Andrew Bogott wrote:

Hello!

Next Wednesday we will be applying security patches to a few Labs 
hosts, including the Labs network node.  We'll do our best to avoid 
downtime, but it's very likely that there will be some brief network 
outages, possibly for as long as several minutes at a time.  This will 
affect all network traffic for all instances, both in Tools and in 
other Labs projects.


No action is required on your part, although you may want to notify 
users of your services who will be affected by downtime.


The scheduled maintenance window is:

Wednesday, 2016-08-31

15:00-17:00 UTC

08:00-10:00 PDT


-Andrew




___
Labs-announce mailing list
labs-annou...@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-announce
___
Labs-l mailing list
Labs-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-l

Re: [Labs-l] [Labs-announce] [action required] We are removing inactive projects.

[Andrew Bogott]

Update:

There are still 30 projects that remain unclaimed on 
https://wikitech.wikimedia.org/wiki/Purge_2016.


I've emailed the admins of each of these projects directly.  If I don't 
hear anything I will start to shut down instances on Thursday.



Many thanks to the vast majority of admins who provided me with prompt 
and useful feedback about project preservation and cleanup!


-Andrew




On 7/8/16 11:09 AM, Andrew Bogott wrote:
If you are exclusively a user of tool labs, you can ignore this email. 
If you use or administer another labs project, this email REQUIRES 
ACTION ON YOUR PART.


We are reclaiming unused resources due to an ongoing shortage.

Visit this page and add a signature under projects you know to be active:

https://wikitech.wikimedia.org/wiki/Purge_2016

Associated wmflabs.org <http://wmflabs.org> domains are included to 
identify projects by offered services.


We are not investigating why projects are needed at this time.  If one 
person votes to preserve then we will do so in this round of cleanup.


In a month, projects and associated instances not claimed will be 
suspended or shutdown.  A month later if no one complains these 
projects will be deleted..


- Andrew (on behalf of all Labs Admins everywhere) 



___
Labs-announce mailing list
labs-annou...@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-announce
___
Labs-l mailing list
Labs-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-l

[Labs-l] [Labs-announce] Labs network maintenance (and outages) Wednesday 2016-08-31, 15:00 UTC

[Andrew Bogott]

Hello!

Next Wednesday we will be applying security patches to a few Labs hosts, 
including the Labs network node.  We'll do our best to avoid downtime, 
but it's very likely that there will be some brief network outages, 
possibly for as long as several minutes at a time.  This will affect all 
network traffic for all instances, both in Tools and in other Labs projects.


No action is required on your part, although you may want to notify 
users of your services who will be affected by downtime.


The scheduled maintenance window is:

Wednesday, 2016-08-31

15:00-17:00 UTC

08:00-10:00 PDT


-Andrew


___
Labs-announce mailing list
labs-annou...@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-announce
___
Labs-l mailing list
Labs-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-l

Re: [Labs-l] [Labs-announce] Instance creation and security group handling issues in Liberty

[Andrew Bogott]

The issues discussed in this email are now resolved.

Instance creation on Horizon and Wikitech has been re-enabled. After 
some discussion we've decided to enable security group editing on 
Horizon but leave it disabled on Wikitech -- the Horizon interface is 
generally nicer, more feature-rich, and more reliable.  Please go to 
Horizon for any future security group needs.


There were two bugs that triggered this incident.  One of them[1] 
prevented enforcement of firewall rules in certain cases, and the 
other[2] enforced rules but updated them very haphazardly.  Both issues 
are now well understood, with patches in place and proper long-term 
solutions underway.


We have not yet written a full incident report, but when we do it will 
most likely be here: 
https://wikitech.wikimedia.org/wiki/Incident_documentation/20160805-LabsSecurityGroups


Sorry for the inconvenience!

-Andrew

[1]  https://phabricator.wikimedia.org/T142388

[2]  https://phabricator.wikimedia.org/T142165



On 8/5/16 3:21 PM, Chase Pettet wrote:

Currently running instances within Labs are fine.

This week we upgraded to Openstack Liberty[1][2].  Thursday (8/4) we 
had reports of issues involving new instances[3].  We have now 
determined there is errant behavior with Liberty managing source 
groups.  We use this to allow instances within the same project to 
communicate with each other.  Attempts to resolve this behavior for 
the Tool Labs project resulted in a short issue today[4].  Requests 
via the web proxy were failing to connect.  Tools and bots within Tool 
Labs were still running.


Currently:
* Newly created instances are not being integrated into their security 
groups appropriately
* We have disabled the self-serve options for instance creation 
temporarily
* Modifying security groups can result in existing instances 
experiencing issues
* We have disabled the self-serve options for security group 
management temporarily as well


We'll update the task[3] as we have more information.  An incident 
report will be filed as well.   As always, we can be found at labs-l 
or on IRC in #wikimedia-labs.


Thanks,

Chase Pettet (on behalf of the Labs team)


[1] https://www.openstack.org/software/liberty/
[2] https://lists.wikimedia.org/pipermail/labs-l/2016-July/004564.html
[3] https://phabricator.wikimedia.org/T142165
[4] https://lists.wikimedia.org/pipermail/labs-l/2016-August/004575.html


___
Labs-announce mailing list
labs-annou...@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-announce



___
Labs-l mailing list
Labs-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-l

Re: [Labs-l] [Labs-announce] Labs Openstack upgrade next Tuesday, 2016-08-02, 16:00 UTC

[Andrew Bogott]

Re-reminder:  I'm starting this in 10 minutes.  CI tests in Gerrit will 
be disabled for the duration.


-A


On 8/1/16 11:55 AM, Andrew Bogott wrote:

Reminder:  This is happening tomorrow.

On 7/25/16 11:57 AM, Andrew Bogott wrote:
I'm going to upgrade our Openstack install from version 'Kilo' to 
version 'Liberty' next Tuesday. I've scheduled a 3-hour window for 
the process, although noticeable service interruptions should be 
shorter.  Here's what to expect:


- Tools services and existing Labs uses should be unaffected apart 
from brief ( < 1 minute) interruptions in network service.


- Continuous Integration tests (Jenkins, Zuul, etc.) will be disabled 
for most of the upgrade window.


- Creation/Deletion of new instances will be disabled for most of the 
window.


- Wikitech and Horizon may error out occasionally and/or display 
inconsistent information.  Users may need to refresh their web logins 
after the upgrade.


Apart from fixing a few seldom-seen bugs, this upgrade shouldn't 
result in noticeable changes for Labs users.  It will lay the 
groundwork for an upcoming Horizon upgrade, but that will be 
announced in a future email.


-Andrew







___
Labs-announce mailing list
labs-annou...@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-announce
___
Labs-l mailing list
Labs-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-l

[Labs-l] [Labs-announce] DONE: Labs Openstack upgrade next Tuesday, 2016-08-02, 16:00 UTC

[Andrew Bogott]

This is all done.  I'm still chasing down a few straggling issues, but 
all normal Labs and CI behavior should be restored.


CI will be backed up for a while until it catches up.  Please inform me 
of any other unexpected behavior.


-andrew


On 7/25/16 11:57 AM, Andrew Bogott wrote:
I'm going to upgrade our Openstack install from version 'Kilo' to 
version 'Liberty' next Tuesday.  I've scheduled a 3-hour window for 
the process, although noticeable service interruptions should be 
shorter.  Here's what to expect:


- Tools services and existing Labs uses should be unaffected apart 
from brief ( < 1 minute) interruptions in network service.


- Continuous Integration tests (Jenkins, Zuul, etc.) will be disabled 
for most of the upgrade window.


- Creation/Deletion of new instances will be disabled for most of the 
window.


- Wikitech and Horizon may error out occasionally and/or display 
inconsistent information.  Users may need to refresh their web logins 
after the upgrade.


Apart from fixing a few seldom-seen bugs, this upgrade shouldn't 
result in noticeable changes for Labs users.  It will lay the 
groundwork for an upcoming Horizon upgrade, but that will be announced 
in a future email.


-Andrew





___
Labs-announce mailing list
labs-annou...@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-announce
___
Labs-l mailing list
Labs-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-l

Re: [Labs-l] [Labs-announce] Labs Openstack upgrade next Tuesday, 2016-08-02, 16:00 UTC

[Andrew Bogott]

Reminder:  This is happening tomorrow.

On 7/25/16 11:57 AM, Andrew Bogott wrote:
I'm going to upgrade our Openstack install from version 'Kilo' to 
version 'Liberty' next Tuesday.  I've scheduled a 3-hour window for 
the process, although noticeable service interruptions should be 
shorter.  Here's what to expect:


- Tools services and existing Labs uses should be unaffected apart 
from brief ( < 1 minute) interruptions in network service.


- Continuous Integration tests (Jenkins, Zuul, etc.) will be disabled 
for most of the upgrade window.


- Creation/Deletion of new instances will be disabled for most of the 
window.


- Wikitech and Horizon may error out occasionally and/or display 
inconsistent information.  Users may need to refresh their web logins 
after the upgrade.


Apart from fixing a few seldom-seen bugs, this upgrade shouldn't 
result in noticeable changes for Labs users.  It will lay the 
groundwork for an upcoming Horizon upgrade, but that will be announced 
in a future email.


-Andrew





___
Labs-announce mailing list
labs-annou...@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-announce
___
Labs-l mailing list
Labs-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-l

[Labs-l] [Labs-announce] Labs Openstack upgrade next Tuesday, 2016-08-02, 16:00 UTC

[Andrew Bogott]

I'm going to upgrade our Openstack install from version 'Kilo' to 
version 'Liberty' next Tuesday.  I've scheduled a 3-hour window for the 
process, although noticeable service interruptions should be shorter.  
Here's what to expect:


- Tools services and existing Labs uses should be unaffected apart from 
brief ( < 1 minute) interruptions in network service.


- Continuous Integration tests (Jenkins, Zuul, etc.) will be disabled 
for most of the upgrade window.


- Creation/Deletion of new instances will be disabled for most of the 
window.


- Wikitech and Horizon may error out occasionally and/or display 
inconsistent information.  Users may need to refresh their web logins 
after the upgrade.


Apart from fixing a few seldom-seen bugs, this upgrade shouldn't result 
in noticeable changes for Labs users.  It will lay the groundwork for an 
upcoming Horizon upgrade, but that will be announced in a future email.


-Andrew



___
Labs-announce mailing list
labs-annou...@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-announce
___
Labs-l mailing list
Labs-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-l

Re: [Labs-l] [action required] We are removing inactive projects.

[Andrew Bogott]

Reminder:

There are still 73 unclaimed projects listed on 
https://wikitech.wikimedia.org/wiki/Purge_2016. Please claim your 
projects today!  (Or, better yet, mark them as unused so I can start 
reclaiming space.)


Thanks!

-Andrew

On 7/8/16 11:09 AM, Andrew Bogott wrote:
If you are exclusively a user of tool labs, you can ignore this email. 
If you use or administer another labs project, this email REQUIRES 
ACTION ON YOUR PART.


We are reclaiming unused resources due to an ongoing shortage.

Visit this page and add a signature under projects you know to be active:

https://wikitech.wikimedia.org/wiki/Purge_2016

Associated wmflabs.org <http://wmflabs.org> domains are included to 
identify projects by offered services.


We are not investigating why projects are needed at this time.  If one 
person votes to preserve then we will do so in this round of cleanup.


In a month, projects and associated instances not claimed will be 
suspended or shutdown.  A month later if no one complains these 
projects will be deleted..


- Andrew (on behalf of all Labs Admins everywhere) 



___
Labs-l mailing list
Labs-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-l

Re: [Labs-l] [Labs-announce] [action required] We are removing inactive projects.

[Andrew Bogott]

On 7/8/16 4:02 PM, Maarten Dammers wrote:

Hi Andrew,

I know resources are limited but I very very strongly object with this 
course of action. A lot of people are not on these mailing list, just 
not reading it or on holiday because it's summer. 
I always make an attempt to personally contact project admins before 
taking steps that affect a project.  That said:  If you are a consumer 
of Labs resources, it is your /responsibility/ to subscribe to this list 
and read announcements from admins.  Any unattended and unmaintained 
server is not just a wasted expense; it is a potential security risk to 
all of Labs.


Suspending, shutting down and deletion projects is a good way to piss 
off and scare away volunteers.
And spending thousands of dollars of donor money on resources for 
projects that no one has looked at in years is a good way to ensure that 
we never have any new volunteers, because we don't have any resources to 
give them.  It's clearly a balancing act, but I'm not clear on what the 
alternatives are to having these periodic clean-ups.


-A


I'm sure that's not your intended outcome.

Maarten

On 08-07-16 18:09, Andrew Bogott wrote:
If you are exclusively a user of tool labs, you can ignore this 
email. If you use or administer another labs project, this email 
REQUIRES ACTION ON YOUR PART.


We are reclaiming unused resources due to an ongoing shortage.

Visit this page and add a signature under projects you know to be active:

https://wikitech.wikimedia.org/wiki/Purge_2016

Associated wmflabs.org <http://wmflabs.org> domains are included to 
identify projects by offered services.


We are not investigating why projects are needed at this time.  If 
one person votes to preserve then we will do so in this round of cleanup.


In a month, projects and associated instances not claimed will be 
suspended or shutdown.  A month later if no one complains these 
projects will be deleted..


- Andrew (on behalf of all Labs Admins everywhere)


___
Labs-announce mailing list
labs-annou...@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-announce


___
Labs-l mailing list
Labs-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-l




___
Labs-l mailing list
Labs-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-l



___
Labs-l mailing list
Labs-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-l

[Labs-l] [Labs-announce] [action required] We are removing inactive projects.

[Andrew Bogott]

If you are exclusively a user of tool labs, you can ignore this email.  
If you use or administer another labs project, this email REQUIRES 
ACTION ON YOUR PART.


We are reclaiming unused resources due to an ongoing shortage.

Visit this page and add a signature under projects you know to be active:

https://wikitech.wikimedia.org/wiki/Purge_2016

Associated wmflabs.org  domains are included to 
identify projects by offered services.


We are not investigating why projects are needed at this time.  If one 
person votes to preserve then we will do so in this round of cleanup.


In a month, projects and associated instances not claimed will be 
suspended or shutdown.  A month later if no one complains these projects 
will be deleted..


- Andrew (on behalf of all Labs Admins everywhere)
___
Labs-announce mailing list
labs-annou...@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-announce
___
Labs-l mailing list
Labs-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-l

Re: [Labs-l] [Labs-announce] Issues with instances on labvirt1011 -- RESOLVED

[Andrew Bogott]

I now believe this issue to be resolved.  As far as I know, users were 
largely unaffected by the problem, but let me know if you're seeing bad 
behavior on any of the instances listed below.


-Andrew

On 7/6/16 10:34 PM, Chase Pettet wrote:
We have been having periodic issues with the host labvirt1011.  This 
is the normal spare host that is in service as we are at capacity in 
Labs.  We are in the process now of integrating more hardware.  
Because of this situation, we have fewer options for moving around 
instances.  However, so far we have not seen any issues with these 
hosted instances.


Instances that are hosted on labvirt1011:

am-01
am02
captcha-logging-01
git-test-3
ids-testing
phab-01
phab-05
phab-tin
social-tools2
test-prometheus2
tools-exec-1212
tools-exec-1216
tools-web-static-02
toolsbeta-valhallasw-puppet-compiler
visualeditor-test
xtools-dev02
xtools-live01

This is a notice only at this point.  If one of these instances is 
experiencing issues it could be us and not you.  We will resolve the 
underlying issue when we are able or move the instances when we have 
capacity.


Thanks,

Chase Pettet


___
Labs-announce mailing list
labs-annou...@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-announce



___
Labs-l mailing list
Labs-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-l

[Labs-l] [Labs-announce] New instance creation temporarily disabled

[Andrew Bogott]

We're running into serious capacity issues this week, so I have 
temporarily disabled the creation of new labs instances.  We have new 
hardware sitting in the datacenter already, so at worst we'll be back to 
normal sometime next week.


Sorry for the inconvenience.  And, as usual, if you have instances you 
aren't using, please delete them!


-Andrew



___
Labs-announce mailing list
labs-annou...@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-announce
___
Labs-l mailing list
Labs-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-l

[Labs-l] [Labs-announce] Reduced staff support for Labs, 2016-06-20 through 206-07-04

[Andrew Bogott]

Hello!

Due to Wikimania and associated travel, there will be fewer Labs 
operations staff on duty during the end of June and the beginning of 
July.  Someone should always be available in case of emergencies, but 
fewer people will be around and response times may be slower.


Bryan, Yuvi and I will all be at Wikimania from the 20th through the 
27th.  During that week we should be reachable via email but may not be 
present on IRC; all three of us will be in the UTC+1 timezone.  The 
following week I will be traveling and largely unreachable; Yuvi will 
remain in Europe but will be working as usual.  Chase will be in North 
America working a normal schedule throughout.


If you have any pressing issues, please bring them to my attention this 
week while I still have the attention span to deal with them.  And, if 
you are visiting Wikimania, come and see us!


-Andrew


___
Labs-announce mailing list
labs-annou...@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-announce
___
Labs-l mailing list
Labs-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-l

[Labs-l] [Labs-announce] Finished: Labs-wide security updates tomorrow, 2016-06-08

[Andrew Bogott]

This is (finally) done.  A handful of instances didn't handle the 
suspends properly and had to be rebooted, but everything should be up 
and running now.  Please let me know if you have instances with new issues.


-Andrew


On 6/7/16 5:26 PM, Andrew Bogott wrote:
This didn't go very well, and then I got pulled into a related 
emergency[1] (one that was related but largely invisible to users.)


So, I may have another go at the suspending/resuming tomorrow 
morning.  As before, if we wind up needing to do something more 
intrusive than suspend/resume I will provide additional notice.


Sorry about the endless maintenance window :(

-Andrew

[1] https://phabricator.wikimedia.org/T137241

On 6/6/16 9:02 AM, Andrew Bogott wrote:
Moritz and I will be applying some security patches to the 
virtualization hosts tomorrow, beginning at 14:00 UTC.  We're going 
to try to apply this change via suspend/resume cycles rather than via 
reboots -- that means that each VM should experience a minute or two 
(or three) of hesitation but should not register any reboots.


If this goes badly and it turns out that actual reboots are 
needed I will reschedule the maintenance for a later date with more 
notice.  For now, though, no action is required from any of you.


-Andrew






___
Labs-announce mailing list
labs-annou...@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-announce
___
Labs-l mailing list
Labs-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-l

[Labs-l] [Labs-announce] More to come! Labs-wide security updates tomorrow, 2016-06-08

[Andrew Bogott]

This didn't go very well, and then I got pulled into a related 
emergency[1] (one that was related but largely invisible to users.)


So, I may have another go at the suspending/resuming tomorrow morning.  
As before, if we wind up needing to do something more intrusive than 
suspend/resume I will provide additional notice.


Sorry about the endless maintenance window :(

-Andrew

[1] https://phabricator.wikimedia.org/T137241

On 6/6/16 9:02 AM, Andrew Bogott wrote:
Moritz and I will be applying some security patches to the 
virtualization hosts tomorrow, beginning at 14:00 UTC.  We're going to 
try to apply this change via suspend/resume cycles rather than via 
reboots -- that means that each VM should experience a minute or two 
(or three) of hesitation but should not register any reboots.


If this goes badly and it turns out that actual reboots are needed 
I will reschedule the maintenance for a later date with more notice.  
For now, though, no action is required from any of you.


-Andrew




___
Labs-announce mailing list
labs-annou...@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-announce
___
Labs-l mailing list
Labs-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-l

Re: [Labs-l] [Labs-announce] Labs-wide security updates tomorrow, 2016-06-07

[Andrew Bogott]

Update:  This is in progress now.  So far, the pauses that instances are 
experiencing have been quite brief (seconds rather than minutes.)


On 6/6/16 9:02 AM, Andrew Bogott wrote:
Moritz and I will be applying some security patches to the 
virtualization hosts tomorrow, beginning at 14:00 UTC.  We're going to 
try to apply this change via suspend/resume cycles rather than via 
reboots -- that means that each VM should experience a minute or two 
(or three) of hesitation but should not register any reboots.


If this goes badly and it turns out that actual reboots are needed 
I will reschedule the maintenance for a later date with more notice.  
For now, though, no action is required from any of you.


-Andrew




___
Labs-announce mailing list
labs-annou...@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-announce
___
Labs-l mailing list
Labs-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-l

[Labs-l] [Labs-announce] Labs-wide security updates tomorrow, 2016-06-07

[Andrew Bogott]

Moritz and I will be applying some security patches to the 
virtualization hosts tomorrow, beginning at 14:00 UTC.  We're going to 
try to apply this change via suspend/resume cycles rather than via 
reboots -- that means that each VM should experience a minute or two (or 
three) of hesitation but should not register any reboots.


If this goes badly and it turns out that actual reboots are needed 
I will reschedule the maintenance for a later date with more notice.  
For now, though, no action is required from any of you.


-Andrew


___
Labs-announce mailing list
labs-annou...@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-announce
___
Labs-l mailing list
Labs-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-l

[Labs-l] [Labs-announce] tools webservice outage

[Andrew Bogott]

Due to a bug with the web proxy setup, all tools webservices have 
been displaying 502s and/or 503s for the last 90 minutes or so.  The 
issue seems to be resolved now, and services are gradually coming back 
up.  The problem was the result of a complex series of coincidences and 
so is unlikely to recur, but I'll leave the details for Yuvi to document.


If you find that your webservice is still failing a few minutes 
from now, feel free to do a 'webservice restart' which will most likely 
fix things.


-Andrew



___
Labs-announce mailing list
labs-annou...@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-announce
___
Labs-l mailing list
Labs-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-l

Re: [Labs-l] [Labs-announce] Possible reboots and/or outages -- please read

[Andrew Bogott]

On 5/20/16 10:45 AM, Maximilian Doerr wrote:

Does this unusual behavior, also cause erroneous fingerprints being given 
during authentication.  Yesterday when I was SSHing into Cyberbot-exec-01 via 
bastion I got a fingerprint mismatch with the exec node.  This was using WinSCP.
That's almost certainly not related.  I'm not sure why that would 
happen, although there are a few possible failure cases in our network 
setup where traffic gets routed not to your VM but directly to the 
nova-network host.  In that case you'd be seeing the wrong key because 
it's the key for labnet1002 instead of for your VM.  The timing doesn't 
hold up but there was a brief network outage a couple of days ago that 
could have caused that.


Do let me know if you see this issue repeatedly -- in particular, if you 
can open a phabricator ticket which includes both host keys (the 
erroneous one and the correct one) then that will help us diagnose things.


-Andrew




To clarify, it got into Bastion just fine but not into my node, because I 
aborted the authentication process.  I waited 5 minutes and tried again and it 
worked just fine.

Cyberpower678
English Wikipedia Account Creation Team
ACC Mailing List Moderator
Global User Renamer


On May 20, 2016, at 11:10, Andrew Bogott <abog...@wikimedia.org> wrote:

Note:  Tools users can ignore this message

We are seeing some unusual behavior on labvirt1003, which hosts a large 
number of labs instances.  The problem is not yet diagnosed, but it is likely a 
hardware problem that will require reboots or downtime.  Here is a complete 
list of labs instances currently living on labvirt1003:

https://phabricator.wikimedia.org/P3159

If you have any hosts on that box that cannot survive a reboot, please 
either let me know, or take steps to minimize the damage.  I've removed 
labvirt1003 from the scheduler, so if you want to build a new instance and 
migrate services to it you can be assured that the new instance will be 
isolated from the coming chaos.

A simple reboot shouldn't produce more than 5-10 minutes of downtime.  If a 
major outage seems likely, I'll follow up with additional warning.

-Andrew


___
Labs-announce mailing list
labs-annou...@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-announce
___
Labs-l mailing list
Labs-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-l

___
Labs-l mailing list
Labs-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-l




___
Labs-l mailing list
Labs-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-l

[Labs-l] [Labs-announce] Possible reboots and/or outages -- please read

[Andrew Bogott]

Note:  Tools users can ignore this message

We are seeing some unusual behavior on labvirt1003, which hosts a 
large number of labs instances.  The problem is not yet diagnosed, but 
it is likely a hardware problem that will require reboots or downtime.  
Here is a complete list of labs instances currently living on labvirt1003:


https://phabricator.wikimedia.org/P3159

If you have any hosts on that box that cannot survive a reboot, 
please either let me know, or take steps to minimize the damage.  I've 
removed labvirt1003 from the scheduler, so if you want to build a new 
instance and migrate services to it you can be assured that the new 
instance will be isolated from the coming chaos.


A simple reboot shouldn't produce more than 5-10 minutes of 
downtime.  If a major outage seems likely, I'll follow up with 
additional warning.


-Andrew


___
Labs-announce mailing list
labs-annou...@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-announce
___
Labs-l mailing list
Labs-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-l

[Labs-l] [Labs-announce] IMPORTANT! Insecure (non-HTTPS) API Requests to become unsupported starting 2016-06-12

[Andrew Bogott]

If your tool/bot/service/whatever consumes Wikimedia APIs, please read 
this forwarded message! Access to APIs under http:// rather than 
https:// is going to start breaking next month.


-Andrew



 Forwarded Message 
Subject: 	[Wikitech-l] Insecure (non-HTTPS) API Requests to become 
unsupported starting 2016-06-12

Date:   Fri, 13 May 2016 22:34:20 +
From:   Brandon Black 
Reply-To:   Wikimedia developers 
To: 	mediawiki-api-annou...@lists.wikimedia.org, 
mediawiki-...@lists.wikimedia.org, Wikimedia developers 





TL;DR:

* All access to Wikimedia production sites/APIs should use https://
URLs, not http:// -- your bot/tool will break in the near future if it
does not!
* 2016-06-12 - insecure access is unsupported; starting on this date
we plan to break (deny with 403) 10% of all insecure requests randomly
as a wake-up call.
* 2016-07-12 - we plan to break all insecure requests.


Hi all,

As you may remember, all production Wikimedia wikis switched to
HTTPS-only for all canonical domainnames nearly a year ago:
https://blog.wikimedia.org/2015/06/12/securing-wikimedia-sites-with-https/

Since way back then, we've been forcing insecure HTTP requests to our
canonical domains over to HTTPS by using redirects and
Strict-Transport-Security, which is effective for the vast majority of
access from humans using browsers and apps.

In the time since, we've been chasing down various corner-case issues
where loopholes may arise in our HTTPS standards and enforcement.  One
of the most-difficult loopholes to close has been the "Insecure POST"
loophole, which is discussed in our ticket system here:
https://phabricator.wikimedia.org/T105794 .

To briefly recap the "Insecure POST" issue:

* Most of our humans using browser UAs are not affected by it.  They
start out doing GET traffic to our sites, their GETs get redirected to
HTTPS if necessary, and then any POSTs issued by their browser use
protocol-relative URIs which are also HTTPS.
* However, many automated/code UAs (bots, tools, etc) access the APIs
using initial POST requests to hardcoded service URLs using HTTP
(rather than HTTPS).
* For all of the code/library UAs out there in the world, there is no
universally-compatible way to redirect them to HTTPS.  There are
different ways that work for some UAs, but many UAs used for APIs
don't handle redirects at all.
* Regardless of the above, even if we could reliably redirect POST
requests, that doesn't fix the security problem like it does with GET.
The private data has already been leaked in the initial insecure
request before we have a chance to redirect it.  If we did some kind
of redirect first, we'd still just be putting off the inevitable
future date where we have to go through a breaking transition to
secure the data.

Basically, we're left with no good way to upgrade these insecure
requests without breaking them.  The only way it gets fixed is if all
of our API clients in the world use explicit https:// URLs for
Wikimedia sites in all of their code and configuration, and the only
way we can really force them to do so is to break insecure POST
requests by returning a 403 error to tools that don't.

Back in July 2015, I began making some efforts to statistically sample
the User-Agent fields of clients doing "Insecure POST" and tracking
down the most-prominent offenders.  We were able to find and fix many
clients along the way since.

A few months ago Bryan Davis got us further when he committed a
MediaWiki core change to let our sites directly warn offending
clients.  I believe that went live on Jan 29th of this year (
https://gerrit.wikimedia.org/r/#/c/266958 ).  It allows insecure POSTs
to still succeed, but sends the clients a standard warning that says
"HTTP used when HTTPS was expected".

This actually broke some older clients that weren't prepared to handle
warnings at all, and caused several clients to upgrade.  We've been
logging offending UAs and accounts which trigger the warning via
EventLogging since then, but after the initial impact the rate
flattened out again; clients and/or users that didn't notice the
warning fairly quickly likely never will.

Many of the remaining UAs we see in logs are simply un-updated.  For
example, https://github.com/mwclient/mwclient switched to
HTTPS-by-default in 0.8.0, released in early January, but we're still
getting lots of insecure POST from older mwclient versions installed
out there in the world.  Even in cases where the code is up to date
and supports HTTPS properly, bot/tool configurations may still have
hardcoded http:// site config URLs.

We're basically out of "soft" ways to finish up this part of the HTTPS
transition, and we've stalled long enough on this.

** 2016-06-12 is the selected support cutoff date **

After this date, insecure HTTP POST requests to our sites are
officially unsupported.  This date is:

* A year to day after the 

[Labs-l] Logins were broken for a bit

[Andrew Bogott]

Due to a puppet blunder, I just now broke ldap (and, hence, user 
logins) pretty much everywhere on labs for about 45 minutes. It was a 
simple mistake with a simple fix, and things should be recovering now, 
if they have not already.  If you encounter any unexpected problems more 
than ~30 minutes from now, please let me know.


Sorry!

-Andrew


___
Labs-l mailing list
Labs-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-l

[Labs-l] [Labs-announce] DNS maintenance window on Thursday, 14:00 UTC

[Andrew Bogott]

This Thursday we're going to do some fairly intrusive rearranging of the 
Labs DNS system[1].  No action is required on the part of Labs users, 
but be ready for the following:


- For some or all of the window I will be disabling instance creation 
and deletion.  Attempts to create or delete instances (or perform any 
other actions that use the nova API) will fail with unhelpful error 
messages.


- CI will be partially or completely broken (some CI tests require new 
instance creation)


- In theory there will not be DNS service interruptions, but in practice 
there may be.


This change is part of a series of planned DNS updates intended to 
improve redundancy and performance.  Expect an hour of weirdness 
beginning at 14:00 UTC, 7:00 AM San Francisco.



[1] https://phabricator.wikimedia.org/T128737


___
Labs-announce mailing list
labs-annou...@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-announce
___
Labs-l mailing list
Labs-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-l

Re: [Labs-l] [Labs-announce] IMPORTANT: Reboots (again!) on Wednesday

[Andrew Bogott]

Security updates are not something you can opt out of.  Most concretely: 
we have to keep our physical hardware secure and up-to-date, and 
rebooting a physical virtualization node has the side-effect of cycling 
power on the resident VMs.


Even if it weren't a natural consequence of our own maintenance, though, 
I would still be rolling out these updates.  Foundation staff can't 
engage in total supervision of all Labs use or activity but we are, 
nonetheless, effectively responsible for any malicious activities that 
might originate from within our cluster.  Naturally we need to take 
whatever general measures we can to prevent Labs from turning into a 
spam farm.  When a staff of three is managing more than 700 VMs we are 
always going to have to rely on general one-size-fits-all maintenance 
solutions.


In almost all cases it is trivial or near-trivial to set up a service so 
that it recovers gracefully from a reboot.  If the few minutes of 
downtime that result are unacceptable, we can provide resources for a 
fail-over instance and ensure that the two instances are spread out on 
different virt hosts so that they don't both go down at once.  Let me 
know if you need help with either of the above!


-A


On 4/3/16 5:23 PM, Petr Bena wrote:

Is it possible to somehow opt-out of these auto updates and reboots?
Majority of kernel exploits (like 99.%) of them can be exploited
only if the hacker has at least some access to remote system (eg. over
ssh) so that they can execute something that eventually get them
rights of user 0.

This makes sense for tool labs, but I see no reason why for example
wm-bot's instance need to get this patch, when all people who can
access it over ssh already have root. I am absolutely sure there is no
security risk while running obsolete kernel on that one for example.
The reboots on other hand cause much more harm.

Thanks

On Wed, Mar 23, 2016 at 11:42 PM, Andrew Bogott <abog...@wikimedia.org> wrote:

This is now done; all labs services should be back to normal.  As always,
it's a good idea to poke at your tools and make sure there aren't jobs that
need restarting.

-Andrew


On 3/23/16 8:23 AM, Andrew Bogott wrote:

Reminder: This is happening today, starting in about 40 minutes.

On 3/18/16 1:58 PM, Andrew Bogott wrote:

Yet another kernel exploit turned up this week, which means another round
of kernel updates and reboots. All labs instances will be rebooted (at
various and unpredictable times) this Wednesday, 2016-03-23, beginning
around 14:00 UTC.

We're getting pretty good at this :(  We'll pool- and de-pool exec
nodes as needed to minimize surprise endings for ToolLabs jobs, but as usual
there will be some stragglers that get cut short or don't get restarted
properly.  Keep an eye out on Wednesday, and let us know on IRC if you run
into trouble.

-Andrew



___
Labs-announce mailing list
labs-annou...@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-announce
___
Labs-l mailing list
Labs-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-l

___
Labs-l mailing list
Labs-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-l



___
Labs-l mailing list
Labs-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-l

Re: [Labs-l] [Labs-announce] Public DNS and Proxy changes coming up this weekend

[Andrew Bogott]

Update:

This is going fine.  There will be an unanticipated delay while we wait 
for our domain registrar to catch up with changes for the new DNS 
server; until that happens the horizon DNS gui will be disabled.  
Currently some users are using the new DNS server and some the old one; 
the data is in sync so you shouldn't notice any problems, but if you 
find something unexpectedly broken please notify me.


I predict that the upstream changes will be fully settled down in around 
48 hours, at which point I'll re-enable the proxy and DNS interfaces.


-Andrew


On 3/23/16 5:44 PM, Andrew Bogott wrote:

== Executive Summary ==

Creation of new web proxies and DNS records will be disabled over the 
weekend; Labs and Tools will switch to a new public DNS system on 
Monday, with possible accompanying hiccups and interruptions.


Nothing will change for ToolLabs users, and no immediate action is 
required on the part of tool or project maintainers.  Starting Monday, 
however, Labs project maintainers will need to use the Horizon[1] web 
UI to manage proxies, manage public DNS records, and assign floating 
IPs to instances.


Labs Project Admins:  Two-factor authentication will be required to 
access the new Horizon interface.  Please set up 2fa now (via 
Preferences-> User Profile on Wikitech) so that you aren't rudely 
surprised when trying to manage Horizon during future emergencies.


== The whole story ==

Currently the public DNS server that resolves things under wmflabs.org 
is running an old and creaky setup using ldap and powerdns.  These 
domains are configured using WMF-developed features on wikitech.  Labs 
internal dns (e.g. foo.bar.eqiad.wmflabs) is now managed by the 
OpenStack Designate project with a more modern mysql-based powerdns 
backend.


There's a ready-made upstream web UI for designate[2] that is part of 
the Horizon project.  So, we're going to standardize on Designate, 
Horizon, and mysql/powerdns, and rip out the old ldap/pdns/wikitech 
code[3].  Web proxy management is intimately linked with dns 
management, so the proxy UI will also move to Horizon, thanks to a 
custom Horizon panel written by Alex Monk.


Timeline:

This week:  Various web Domain and Proxy UIs will appear and disappear 
from horizon.wikimedia.org as we put the finishing touches on the new 
interface.  Changes made via these interfaces will have no effect on 
public DNS before Monday; on Monday some such changes may persist and 
some may be overwritten.


Friday, 2015-03-25:  A few sidebar links on wikitech will vanish: " 
Manage Addresses," and " Manage Web Proxies."  During the following 
days, public DNS will be effectively frozen so that we have time to 
safely migrate to the new setup.


Friday (and, possibly, weekend):  DNS migration to designate, testing

Monday, 2015-03-28, 18:00 UTC:  The public dns servers labs-ns0 and 
labs-ns1 will be moved to point to the new DNS service.  There may be 
brief interruptions to public DNS during the switch-over. The Horizon 
web UI for domains and proxies will be live and available to all 
project admins.




[1] https://wikitech.wikimedia.org/wiki/Help:Horizon_FAQ
[2] https://github.com/openstack/designate-dashboard
[3] https://phabricator.wikimedia.org/T124184


___
Labs-announce mailing list
labs-annou...@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-announce
___
Labs-l mailing list
Labs-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-l

Re: [Labs-l] [Labs-announce] Public DNS and Proxy changes coming up this weekend

[Andrew Bogott]

Slight corrections to the dates, below... all this is still happening 
this weekend, but this weekend is Friday 3/25-Monday 3/28.  My calendar 
was turned to the wrong page :)


-A


On 3/23/16 5:44 PM, Andrew Bogott wrote:

== Executive Summary ==

Creation of new web proxies and DNS records will be disabled over the 
weekend; Labs and Tools will switch to a new public DNS system on 
Monday, with possible accompanying hiccups and interruptions.


Nothing will change for ToolLabs users, and no immediate action is 
required on the part of tool or project maintainers.  Starting Monday, 
however, Labs project maintainers will need to use the Horizon[1] web 
UI to manage proxies, manage public DNS records, and assign floating 
IPs to instances.


Labs Project Admins:  Two-factor authentication will be required to 
access the new Horizon interface.  Please set up 2fa now (via 
Preferences-> User Profile on Wikitech) so that you aren't rudely 
surprised when trying to manage Horizon during future emergencies.


== The whole story ==

Currently the public DNS server that resolves things under wmflabs.org 
is running an old and creaky setup using ldap and powerdns.  These 
domains are configured using WMF-developed features on wikitech.  Labs 
internal dns (e.g. foo.bar.eqiad.wmflabs) is now managed by the 
OpenStack Designate project with a more modern mysql-based powerdns 
backend.


There's a ready-made upstream web UI for designate[2] that is part of 
the Horizon project.  So, we're going to standardize on Designate, 
Horizon, and mysql/powerdns, and rip out the old ldap/pdns/wikitech 
code[3].  Web proxy management is intimately linked with dns 
management, so the proxy UI will also move to Horizon, thanks to a 
custom Horizon panel written by Alex Monk.


Timeline:

This week:  Various web Domain and Proxy UIs will appear and disappear 
from horizon.wikimedia.org as we put the finishing touches on the new 
interface.  Changes made via these interfaces will have no effect on 
public DNS before Monday; on Monday some such changes may persist and 
some may be overwritten.


Friday, 2015-03-27:  A few sidebar links on wikitech will vanish: " 
Manage Addresses," and " Manage Web Proxies."  During the following 
days, public DNS will be effectively frozen so that we have time to 
safely migrate to the new setup.


Friday (and, possibly, weekend):  DNS migration to designate, testing

Monday, 2015-03-30, 18:00 UTC:  The public dns servers labs-ns0 and 
labs-ns1 will be moved to point to the new DNS service.  There may be 
brief interruptions to public DNS during the switch-over. The Horizon 
web UI for domains and proxies will be live and available to all 
project admins.




[1] https://wikitech.wikimedia.org/wiki/Help:Horizon_FAQ
[2] https://github.com/openstack/designate-dashboard
[3] https://phabricator.wikimedia.org/T124184


___
Labs-announce mailing list
labs-annou...@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-announce
___
Labs-l mailing list
Labs-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-l

Re: [Labs-l] Reboot post-mortem

[Andrew Bogott]

On 3/24/16 10:06 AM, Stephen E Slevinski Jr wrote:
Yesterday, during the reboot, one of my instances died.  It reported 
it was active, but the web-proxy failed with a 502 bad gateway and ssh 
failed with "channel 0: open failed: connect failed: No route to 
host".  Manual reboots appeared successful, but the problems 
continued.  On IRC, Yuvipanda was unable to debug the problem, so I 
created a new instance this morning.
The kernel updates applied yesterday required us to mess with grub -- 
it's possible that this instance suffered from a series of unlucky 
coincidences and lost its bootloader entirely.




I no longer need instance "signwriting-icon-server-2".  Should I 
delete the instance or does someone want to do an autopsy?
I don't have time right this minute but I'd like to investigate it; I'll 
delete it when I'm done.  Thanks!


-A




For project SignWriting, I was going to update the documentation to 
reflect the new instance name, but I'm having more problems. The "Edit 
documentation" link opens a blank page, rather than the project 
documentation form.

https://wikitech.wikimedia.org/wiki/Nova_Resource:Signwriting

Additionally, the "Instances for this project" section does not report 
the new instance yet, but this might be a caching issue that will 
resolve on it's own eventually.


Regards,
∼Steve

___
Labs-l mailing list
Labs-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-l



___
Labs-l mailing list
Labs-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-l

Re: [Labs-l] [Labs-announce] Public DNS and Proxy changes coming up this weekend

[Andrew Bogott]

On 3/24/16 9:19 AM, Moritz Schubotz wrote:

Hi Andrew,

now, it works nicely. I don't know what was wrong but there were no
messages the other day
for example instead of

That's great!  Whoever fixed that, take a bow :)

-A



Download a mobile app for two-factor authentication (such as Google
Authenticator) on to your phone.

it showed



So I'm now set up as two factor auth user.

Best
Moritz

On Thu, Mar 24, 2016 at 10:10 AM, Andrew Bogott <abog...@wikimedia.org> wrote:

On 3/23/16 6:42 PM, Moritz Schubotz wrote:

Hi Andrew,

enabling two factor authentification looks not very well documented.
I see no text just the i18n labels

Is there a guide?


Can you send me a screenshot of what you're seeing?  When I setup 2fa there
are clear step-by-step instructions on the page that displays the initial QR
code.

-A



Best
Moritz


On Wed, Mar 23, 2016 at 6:44 PM, Andrew Bogott <abog...@wikimedia.org>
wrote:

== Executive Summary ==

Creation of new web proxies and DNS records will be disabled over the
weekend; Labs and Tools will switch to a new public DNS system on Monday,
with possible accompanying hiccups and interruptions.

Nothing will change for ToolLabs users, and no immediate action is
required
on the part of tool or project maintainers.  Starting Monday, however,
Labs
project maintainers will need to use the Horizon[1] web UI to manage
proxies, manage public DNS records, and assign floating IPs to instances.

Labs Project Admins:  Two-factor authentication will be required to
access
the new Horizon interface.  Please set up 2fa now (via Preferences-> User
Profile on Wikitech) so that you aren't rudely surprised when trying to
manage Horizon during future emergencies.

== The whole story ==

Currently the public DNS server that resolves things under wmflabs.org is
running an old and creaky setup using ldap and powerdns.  These domains
are
configured using WMF-developed features on wikitech.  Labs internal dns
(e.g. foo.bar.eqiad.wmflabs) is now managed by the OpenStack Designate
project with a more modern mysql-based powerdns backend.

There's a ready-made upstream web UI for designate[2] that is part of the
Horizon project.  So, we're going to standardize on Designate, Horizon,
and
mysql/powerdns, and rip out the old ldap/pdns/wikitech code[3].  Web
proxy
management is intimately linked with dns management, so the proxy UI will
also move to Horizon, thanks to a custom Horizon panel written by Alex
Monk.

Timeline:

This week:  Various web Domain and Proxy UIs will appear and disappear
from
horizon.wikimedia.org as we put the finishing touches on the new
interface.
Changes made via these interfaces will have no effect on public DNS
before
Monday; on Monday some such changes may persist and some may be
overwritten.

Friday, 2015-03-27:  A few sidebar links on wikitech will vanish:  "
Manage
Addresses," and " Manage Web Proxies."  During the following days, public
DNS will be effectively frozen so that we have time to safely migrate to
the
new setup.

Friday (and, possibly, weekend):  DNS migration to designate, testing

Monday, 2015-03-30, 18:00 UTC:  The public dns servers labs-ns0 and
labs-ns1
will be moved to point to the new DNS service.  There may be brief
interruptions to public DNS during the switch-over.  The Horizon web UI
for
domains and proxies will be live and available to all project admins.



[1] https://wikitech.wikimedia.org/wiki/Help:Horizon_FAQ
[2] https://github.com/openstack/designate-dashboard
[3] https://phabricator.wikimedia.org/T124184

___
Labs-announce mailing list
labs-annou...@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-announce

___
Labs-l mailing list
Labs-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-l









___
Labs-l mailing list
Labs-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-l

[Labs-l] [Labs-announce] Public DNS and Proxy changes coming up this weekend

[Andrew Bogott]

== Executive Summary ==

Creation of new web proxies and DNS records will be disabled over the 
weekend; Labs and Tools will switch to a new public DNS system on 
Monday, with possible accompanying hiccups and interruptions.


Nothing will change for ToolLabs users, and no immediate action is 
required on the part of tool or project maintainers.  Starting Monday, 
however, Labs project maintainers will need to use the Horizon[1] web UI 
to manage proxies, manage public DNS records, and assign floating IPs to 
instances.


Labs Project Admins:  Two-factor authentication will be required to 
access the new Horizon interface.  Please set up 2fa now (via 
Preferences-> User Profile on Wikitech) so that you aren't rudely 
surprised when trying to manage Horizon during future emergencies.


== The whole story ==

Currently the public DNS server that resolves things under wmflabs.org 
is running an old and creaky setup using ldap and powerdns.  These 
domains are configured using WMF-developed features on wikitech.  Labs 
internal dns (e.g. foo.bar.eqiad.wmflabs) is now managed by the 
OpenStack Designate project with a more modern mysql-based powerdns backend.


There's a ready-made upstream web UI for designate[2] that is part of 
the Horizon project.  So, we're going to standardize on Designate, 
Horizon, and mysql/powerdns, and rip out the old ldap/pdns/wikitech 
code[3].  Web proxy management is intimately linked with dns management, 
so the proxy UI will also move to Horizon, thanks to a custom Horizon 
panel written by Alex Monk.


Timeline:

This week:  Various web Domain and Proxy UIs will appear and disappear 
from horizon.wikimedia.org as we put the finishing touches on the new 
interface.  Changes made via these interfaces will have no effect on 
public DNS before Monday; on Monday some such changes may persist and 
some may be overwritten.


Friday, 2015-03-27:  A few sidebar links on wikitech will vanish:  " 
Manage Addresses," and " Manage Web Proxies."  During the following 
days, public DNS will be effectively frozen so that we have time to 
safely migrate to the new setup.


Friday (and, possibly, weekend):  DNS migration to designate, testing

Monday, 2015-03-30, 18:00 UTC:  The public dns servers labs-ns0 and 
labs-ns1 will be moved to point to the new DNS service.  There may be 
brief interruptions to public DNS during the switch-over.  The Horizon 
web UI for domains and proxies will be live and available to all project 
admins.




[1] https://wikitech.wikimedia.org/wiki/Help:Horizon_FAQ
[2] https://github.com/openstack/designate-dashboard
[3] https://phabricator.wikimedia.org/T124184
___
Labs-announce mailing list
labs-annou...@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-announce
___
Labs-l mailing list
Labs-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-l

Re: [Labs-l] [Labs-announce] IMPORTANT: Reboots (again!) on Wednesday

[Andrew Bogott]

This is now done; all labs services should be back to normal.  As 
always, it's a good idea to poke at your tools and make sure there 
aren't jobs that need restarting.


-Andrew

On 3/23/16 8:23 AM, Andrew Bogott wrote:

Reminder: This is happening today, starting in about 40 minutes.

On 3/18/16 1:58 PM, Andrew Bogott wrote:
Yet another kernel exploit turned up this week, which means another 
round of kernel updates and reboots. All labs instances will be 
rebooted (at various and unpredictable times) this Wednesday, 
2016-03-23, beginning around 14:00 UTC.


   We're getting pretty good at this :(  We'll pool- and de-pool exec 
nodes as needed to minimize surprise endings for ToolLabs jobs, but 
as usual there will be some stragglers that get cut short or don't 
get restarted properly.  Keep an eye out on Wednesday, and let us 
know on IRC if you run into trouble.


-Andrew






___
Labs-announce mailing list
labs-annou...@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-announce
___
Labs-l mailing list
Labs-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-l

Re: [Labs-l] [Labs-announce] IMPORTANT: Reboots (again!) on Wednesday

[Andrew Bogott]

Reminder:  This is happening today, starting in about 40 minutes.

On 3/18/16 1:58 PM, Andrew Bogott wrote:
Yet another kernel exploit turned up this week, which means another 
round of kernel updates and reboots.  All labs instances will be 
rebooted (at various and unpredictable times) this Wednesday, 
2016-03-23, beginning around 14:00 UTC.


   We're getting pretty good at this :(  We'll pool- and de-pool exec 
nodes as needed to minimize surprise endings for ToolLabs jobs, but as 
usual there will be some stragglers that get cut short or don't get 
restarted properly.  Keep an eye out on Wednesday, and let us know on 
IRC if you run into trouble.


-Andrew




___
Labs-announce mailing list
labs-annou...@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-announce
___
Labs-l mailing list
Labs-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-l

Re: [Labs-l] Potential new public DNS/proxy naming policies

[Andrew Bogott]

On 3/9/16 10:24 AM, Andrew Bogott wrote:
Merlijn has just pointed out that my scheme will not work AT ALL for 
http proxies.  I think there's a work-around for that, so feel free to 
mentally insert 'except for proxies which will stay the same' whenever 
necessary while reading this.
To follow up on this last comment... this means that my change should 
only affect domain names bound to public IPs that are assigned within 
your project.  You can check the complete list of such addresses by 
visiting https://wikitech.wikimedia.org/wiki/Special:NovaAddress for 
your project.



-A




On 3/9/16 9:46 AM, Andrew Bogott wrote:
We're in the process of moving our DNS manipulation web UI out of 
wikitech/OpenStackManager and adopting the upstream OpenStack tools 
and APIs.  As usual, though, our current security/user model is weird 
and not especially supported by the upstream models.


Rather than hacking away at Openstack, I'm considering just adopting 
their model.


Right now on wikitech, any project admin can:

1) Create records under wmflabs.org
2) Create records under any pre-existing subdomain of wmflabs.org
3) Bind a floating IP to any of the above records
4) Associate an http proxy with any of the above records
5) Ask an admin to create a new subdomain of wmflabs.org for use in 
option 2.


The thing that's hard to do with the OpenStack tools is item 1 and 2 
-- there's no real conception of a 'global' domain that's shared and 
editable among multiple projects.  So, I propose a new model where 
users can...


1) Create records under .wmflabs.org
2) Create records under pre-existing subdomains of wmflabs.org that 
belong to the project in question

3) Bind a floating IP to any of the above records
4) Associate an http proxy with any of the above records
5) Ask an admin to create a new project-specific subdomain of 
wmflabs.org for use in option 2 (not necessarily a subdomain of 
.wmflabs.org)


How is that different?

a) there will no longer be any foo.wmflabs.org records, only 
foo..wmflabs.org records.
b) Existing records using the foo.wmflabs.org scheme will have to be 
migrated to a project-specific domain, or remain in a weird 
in-between state where only admins can see and edit them.
c) If there are any existing subdomains that are shared between 
projects, they'll need to be untangled.



So, tell me:  How much will this change hurt you, and how much will 
it hurt your users?  Please be as detailed as possible so that I have 
what I need to come up with compromise solutions.


Thank you!

-Andrew





___
Labs-l mailing list
Labs-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-l

Re: [Labs-l] Potential new public DNS/proxy naming policies

[Andrew Bogott]

Merlijn has just pointed out that my scheme will not work AT ALL for 
http proxies.  I think there's a work-around for that, so feel free to 
mentally insert 'except for proxies which will stay the same' whenever 
necessary while reading this.


-A




On 3/9/16 9:46 AM, Andrew Bogott wrote:
We're in the process of moving our DNS manipulation web UI out of 
wikitech/OpenStackManager and adopting the upstream OpenStack tools 
and APIs.  As usual, though, our current security/user model is weird 
and not especially supported by the upstream models.


Rather than hacking away at Openstack, I'm considering just adopting 
their model.


Right now on wikitech, any project admin can:

1) Create records under wmflabs.org
2) Create records under any pre-existing subdomain of wmflabs.org
3) Bind a floating IP to any of the above records
4) Associate an http proxy with any of the above records
5) Ask an admin to create a new subdomain of wmflabs.org for use in 
option 2.


The thing that's hard to do with the OpenStack tools is item 1 and 2 
-- there's no real conception of a 'global' domain that's shared and 
editable among multiple projects.  So, I propose a new model where 
users can...


1) Create records under .wmflabs.org
2) Create records under pre-existing subdomains of wmflabs.org that 
belong to the project in question

3) Bind a floating IP to any of the above records
4) Associate an http proxy with any of the above records
5) Ask an admin to create a new project-specific subdomain of 
wmflabs.org for use in option 2 (not necessarily a subdomain of 
.wmflabs.org)


How is that different?

a) there will no longer be any foo.wmflabs.org records, only 
foo..wmflabs.org records.
b) Existing records using the foo.wmflabs.org scheme will have to be 
migrated to a project-specific domain, or remain in a weird in-between 
state where only admins can see and edit them.
c) If there are any existing subdomains that are shared between 
projects, they'll need to be untangled.



So, tell me:  How much will this change hurt you, and how much will it 
hurt your users?  Please be as detailed as possible so that I have 
what I need to come up with compromise solutions.


Thank you!

-Andrew



___
Labs-l mailing list
Labs-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-l

[Labs-l] Potential new public DNS/proxy naming policies

[Andrew Bogott]

We're in the process of moving our DNS manipulation web UI out of 
wikitech/OpenStackManager and adopting the upstream OpenStack tools and 
APIs.  As usual, though, our current security/user model is weird and 
not especially supported by the upstream models.


Rather than hacking away at Openstack, I'm considering just adopting 
their model.


Right now on wikitech, any project admin can:

1) Create records under wmflabs.org
2) Create records under any pre-existing subdomain of wmflabs.org
3) Bind a floating IP to any of the above records
4) Associate an http proxy with any of the above records
5) Ask an admin to create a new subdomain of wmflabs.org for use in 
option 2.


The thing that's hard to do with the OpenStack tools is item 1 and 2 -- 
there's no real conception of a 'global' domain that's shared and 
editable among multiple projects.  So, I propose a new model where users 
can...


1) Create records under .wmflabs.org
2) Create records under pre-existing subdomains of wmflabs.org that 
belong to the project in question

3) Bind a floating IP to any of the above records
4) Associate an http proxy with any of the above records
5) Ask an admin to create a new project-specific subdomain of 
wmflabs.org for use in option 2 (not necessarily a subdomain of 
.wmflabs.org)


How is that different?

a) there will no longer be any foo.wmflabs.org records, only 
foo..wmflabs.org records.
b) Existing records using the foo.wmflabs.org scheme will have to be 
migrated to a project-specific domain, or remain in a weird in-between 
state where only admins can see and edit them.
c) If there are any existing subdomains that are shared between 
projects, they'll need to be untangled.



So, tell me:  How much will this change hurt you, and how much will it 
hurt your users?  Please be as detailed as possible so that I have what 
I need to come up with compromise solutions.


Thank you!

-Andrew

___
Labs-l mailing list
Labs-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-l

Re: [Labs-l] Labs privacy policy questions

[Andrew Bogott]

Pine --

I was not involved in crafting the language of this policy, but I can at 
least start to answer your questions.


On 3/7/16 9:54 PM, Pine W wrote:

The Wikimetrics login screen [1] presents me with this information:

"By using this project, you agree that any private information you 
give to this project may be made publicly available and not be treated 
as confidential.


"By using this project, you agree that the volunteer administrators of 
this project will have access to any data you submit. This can include 
your IP address, your username/password combination for accounts 
created in Labs services, and any other information that you send. The 
volunteer administrators of this project are bound by the Wikimedia 
Labs Terms of Use, and are not allowed to share this information or 
use it in any non-approved way.


"Since access to this information is fundamental to the operation of 
Wikimedia Labs, these terms regarding use of your data expressly 
override the Wikimedia Foundation's Privacy Policy as it relates to 
the use and access of your personal information."


I have two questions to start.

1. Why would my IP, password, or other "private information" that I 
give to Labs ever "be made publicly available and not treated as 
confidential"?
Clearly that's a question for the project admins -- I don't know what 
they're planning to do with your data.  The main take-away is that you 
are giving your information to them, /not/ to NDA-bound WMF staff.  In 
my (non-legal) option, that means we're already outside the realm of 
'confidential'.  Furthermore, project membership is only informally 
managed, and most likely any member of the wikimetrics project can also 
access identifying information.


I can think of lots of good reasons why a user's IP address might be 
interesting as research data and might legitimately find it's way into 
public view, /if a user willingly discloses it/.  That's exactly why we 
have disclosures like this:  so that real, confidential Wikimedia 
projects don't quietly dump their traffic into a Labs back-end without 
seriously considering the possible breaches of privacy that might result.




2. Why would Labs volunteer administrators ever have access to my 
password? To the best of my knowledge, even WMF staff never have 
access to plaintext passwords of anyone but themselves unless someone 
chooses to disclose their password on a one-time basis.



I believe you're referring to this text:

"This can include your IP address, your username/password combination 
for accounts created in Labs services, and any other information that 
you send."


Again, the point is that you are logging into software that is created 
and maintained by volunteers -- therefore by definition your information 
is passing through their hands.  Clearly a well-made project will not 
present plaintext passwords to actual human eyes, but you are typing 
your password into a text field maintained by actual human volunteers, 
which in terms of security amounts to the same thing:  you are trusting 
those volunteers with your password.


I hope that helps!  I don't think there's a lot of wiggle-room here... 
if you are uncomfortable with the terms of use for a given labs project, 
best not to use it.


-Andrew



Thanks,

Pine


[1] 
https://metrics.wmflabs.org/login?next=%2Freports%2Fprogram-global-metrics



___
Labs-l mailing list
Labs-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-l


___
Labs-l mailing list
Labs-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-l

Re: [Labs-l] Spring Cleaning

[Andrew Bogott]

On 3/6/16 8:02 AM, Merlijn van Deen (valhallasw) wrote:
On 5 March 2016 at 20:18, Mr. Maximilian Doerr 
> wrote:


38G  523df61c-07f0-41ba-924d-e2b8e474b4d7  tools-exec-cyberbot
 tools marc

What on Earth could Cyberbot be generating? Nothing in its folder
amounts to that size


The storage is 'flexible', but it doesn't shrink. This means that if 
you have a 80GB disk, with only 20GB filled, it will use 20GB on the 
VM server. If you then fill the disk, it will use 80GB on the VM 
server. However, if you now delete 60GB of data, it will /still/ use 
80GB on the VM server. Andrew pointed out in an earlier email that 
it's possible to reclaim the space by shutting down the instance and 
compressing it, or by creating a new instance.
I ran a test of this on Friday and the recompressing process (qemu-img 
convert -O qcow2) didn't actually save me much space -- the instance had 
about 50 Gb of 'empty' space in it but recomressing only recaptured 
about 2Gb.  I'm not sure if there's a better process than the one I'm 
using;  it is nonetheless useful to delete unneeded files, though, since 
it prevents future footprint growth.
___
Labs-l mailing list
Labs-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-l

Re: [Labs-l] Spring Cleaning

[Andrew Bogott]

On 3/4/16 4:35 PM, Magnus Manske wrote:
Huh. WDQ seems to be the biggest problem, but I can't find the data - 
only ~20GB in total.
I'm not 100% sure about how our qcow2 setup works, but I suspect that if 
the storage use of an instance grows and then subsequently shrinks the 
new space is not automatically reclaimed.  So if you see specific 
instances that are huge on my report but small in reality, I can reclaim 
empty space by shutting the instances down briefly and recompressing.


Magnus (and, really, anyone) if you find cases like that please let me 
know the instance and project and if/when you could tolerate downtime 
for a few minutes.


Thanks!

-Andrew




Unless Yuvi has cleaned up in the meantime, or I don't have permissions...

On Fri, Mar 4, 2016 at 7:45 PM Brion Vibber <bvib...@wikimedia.org 
<mailto:bvib...@wikimedia.org>> wrote:


Thanks for the reminder ping! I still need ogvjs-testing for
ongoing work on the mobile side (the desktop integration is done
months ago) but I can probably trim its disk space on my end. (It
uses a lot of space because it's got local copies of some videos
from Commons, but the player can pull them directly from Commons
these days.)

-- brion

On Fri, Mar 4, 2016 at 10:59 AM, Andrew Bogott
<abog...@wikimedia.org <mailto:abog...@wikimedia.org>> wrote:

Disk space is always the most limited factor on the Labs
virtualization servers.  If you're using it for something
good, then no one minds!  If you're using it for logfiles that
no one ever reads, or using it for instances that have
long-since been forgotten and no one cares about, then I mind
a little bit :)

Attached is an epic list:  Every instance, sorted by disk
footprint.  Please take a moment to hit ctrl-F and scan the
list of instances that you created or that are part of a
project you work with.  Look at those instances, and ask
yourself if they would be missed.

If the answer is 'no,' please delete them!  Or, email me
and I'll delete them.  (Actually, even if you delete them
yourself, I'd appreciate knowing about it so I know whether or
not emails like this are useful.)  If the answer is 'maybe'
then please call my attention to the instances in question,
and I'll do the necessary pestering and verifying to determine
whether or not they can be safely deleted.

Benefits of good VM hygiene include:  Less
rebalancing-related downtime, better VM performance, happier
operators,  and more money for the Labs team to spend on
things more interesting than storage space.

Thanks for your attention!


-Andrew




___
Labs-l mailing list
Labs-l@lists.wikimedia.org <mailto:Labs-l@lists.wikimedia.org>
https://lists.wikimedia.org/mailman/listinfo/labs-l


___
Labs-l mailing list
Labs-l@lists.wikimedia.org <mailto:Labs-l@lists.wikimedia.org>
https://lists.wikimedia.org/mailman/listinfo/labs-l



___
Labs-l mailing list
Labs-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-l

Re: [Labs-l] [Labs-announce] abrupt tools login host changes

[Andrew Bogott]

On 2/24/16 1:25 PM, DeltaQuad Wikipedia wrote:
tools-bastion-05 seems to be moving pretty slow, I assume that's going 
to be the case till the new one is built?


As far as I can tell these issues are unrelated.  -05 has all the same 
resources as -01 and should serve as a permanent replacement.


In other news, there's been a lot of crazy busy traffic on -05 that most 
likely would have crippled -01 as well.  We're ongoingly engaging with 
this particular problem.


-A




Amanda - DeltaQuad
English Wikipedia Arbitrator
Notice: Anything sent from this email, unless otherwise noted, is in 
my individual capacity, and does not represent the Arbitration 
Committee at all.


On 22 February 2016 at 11:31, Andrew Bogott <abog...@wikimedia.org 
<mailto:abog...@wikimedia.org>> wrote:


Just a quick update, below...

On 2/21/16 8:03 PM, Andrew Bogott wrote:

On 2/21/16 7:00 PM, Brad Jorsch (Anomie) wrote:

On Sun, Feb 21, 2016 at 5:16 PM, Andrew Bogott
<abog...@wikimedia.org <mailto:abog...@wikimedia.org>> wrote:

I've already moved all of the affected hostnames (e.g.
tools-login.wmflabs.org <http://tools-login.wmflabs.org>) so
if you get kicked off of the bastion, just reconnect and all
will be more-or-less as it was.


It seems that the grid doesn't know about the new tools-login,
so I'm getting "host "tools-bastion-05.tools.eqiad.wmflabs" is
no submit host." whenever I try to do much there.

Indeed, apparently puppet doesn't know how to make a submit
host.  So, I've moved tools-login back to bastion-02.


I've now declared bastion-05 a submit host and moved traffic back
there.  Please let me know  if you run into any trouble.


-A



___
Labs-l mailing list
Labs-l@lists.wikimedia.org <mailto:Labs-l@lists.wikimedia.org>
https://lists.wikimedia.org/mailman/listinfo/labs-l




___
Labs-l mailing list
Labs-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-l

Re: [Labs-l] [Labs-announce] abrupt tools login host changes

[Andrew Bogott]

Just a quick update, below...

On 2/21/16 8:03 PM, Andrew Bogott wrote:

On 2/21/16 7:00 PM, Brad Jorsch (Anomie) wrote:
On Sun, Feb 21, 2016 at 5:16 PM, Andrew Bogott <abog...@wikimedia.org 
<mailto:abog...@wikimedia.org>> wrote:


I've already moved all of the affected hostnames (e.g.
tools-login.wmflabs.org <http://tools-login.wmflabs.org>) so if
you get kicked off of the bastion, just reconnect and all will be
more-or-less as it was.


It seems that the grid doesn't know about the new tools-login, so I'm 
getting "host "tools-bastion-05.tools.eqiad.wmflabs" is no submit 
host." whenever I try to do much there.
Indeed, apparently puppet doesn't know how to make a submit host. So, 
I've moved tools-login back to bastion-02.


I've now declared bastion-05 a submit host and moved traffic back 
there.  Please let me know  if you run into any trouble.



-A


___
Labs-l mailing list
Labs-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-l

Re: [Labs-l] [Labs-announce] sudo vulnerability in toollabs

[Andrew Bogott]

Just a quick follow-up to Marc's response...

On 2/22/16 7:43 AM, Marc-André Pelletier wrote:

On 16-02-22 03:26 AM, Ilya Korniyko wrote:

Configuration is created automatically by puppet, isn't it?
In this case, puppet is not involved.  Sudo policies are stored in ldap; 
they are applied more-or-less instantly after a change rather than 
pending a puppet run.



Does it also include automated tests for this scenarios? If not - why?
Thorough automated tests would have eliminated such mistakes.
Patches and phab tasks are always welcome :)  We do a lot of monitoring 
for tools, but most current tests are of the form "this should work; 
does it?"  We have fairly few tests of the form "this should not work; 
does it?"


Because writing tests (and, subsequently, sifting through false 
positives) is fairly labor-intensive, it's always a judgement call when 
to stop writing new ones.  That said, I agree that we would benefit from 
a suite of tests for basic access controls.




Not really in that particular case:  The short of what happened is that
a migration inadvertently caused the default sudo policy rules to return
to all projects - including for those where they had been explicitly
removed and replaced with something more restrictive.

The end result, of course, is that people were able to sudo to root that
weren't *intended* to, but they /technically/ did so correctly according
to the configuration in place.

-- Marc


___
Labs-l mailing list
Labs-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-l



___
Labs-l mailing list
Labs-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-l

Re: [Labs-l] [Labs-announce] sudo vulnerability in toollabs

[Andrew Bogott]

On 2/22/16 2:11 AM, Legoktm wrote:

Hi,

On 02/21/2016 07:46 PM, Andrew Bogott wrote:

 Most labs projects have permissive sudo policies by default.  A few
have locked down policies, and those projects have been closely checked.
  Nonetheless, for completeness here are projects that were temporarily
less secure:  'catgraph', 'translatesvg', 'toolsbeta', 'jawiki',
'wmve-techteam', 'utrs', 'wmt', 'bastion', 'project-proxy',
'mediawiki-verp', 'glam', 'wlmjudging', 'tools',
'account-creation-assistance'

To clarify, these projects should specifically be checked because they
don't have "permissive sudo policies"? Could you expand on what you mean
by that?


Yes, sorry, I'll try again :)

New labs projects by default provide complete sudo access to all 
members.  Most labs projects preserve those initial settings -- that 
means that most projects were untouched by this issue, because they 
/already/ had the policy that was inadvertently applied.


The above list is all of the projects that no longer retained the 
default permissive policy (as of late January) and therefore had their 
sudo policies expanded by the errant rules applied on the 12th.


Fortunately, most of the projects in the above list fall in to one or 
more of these categories:


- All users are projectadmins, thus effectively providing full access to 
all potential logins

- No active VMs, thus nothing to exploit

That combined with auth log auditing leaves me relatively unconcerned 
about projects other than bastion and tools (they being both active and 
containing a large number of non-root users).


I hope that makes more sense!

-Andrew




-- Legoktm

___
Labs-l mailing list
Labs-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-l



___
Labs-l mailing list
Labs-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-l

Re: [Labs-l] [Labs-announce] abrupt tools login host changes

[Andrew Bogott]

On 2/21/16 7:00 PM, Brad Jorsch (Anomie) wrote:
On Sun, Feb 21, 2016 at 5:16 PM, Andrew Bogott <abog...@wikimedia.org 
<mailto:abog...@wikimedia.org>> wrote:


I've already moved all of the affected hostnames (e.g.
tools-login.wmflabs.org <http://tools-login.wmflabs.org>) so if
you get kicked off of the bastion, just reconnect and all will be
more-or-less as it was.


It seems that the grid doesn't know about the new tools-login, so I'm 
getting "host "tools-bastion-05.tools.eqiad.wmflabs" is no submit 
host." whenever I try to do much there.
Indeed, apparently puppet doesn't know how to make a submit host. So, 
I've moved tools-login back to bastion-02.


-A
___
Labs-l mailing list
Labs-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-l

[Labs-l] [Labs-announce] abrupt tools login host changes

[Andrew Bogott]

This one is for Tools users only!

We're in the midst of an ongoing security incident.  At this point it's 
in the realm of 'potential compromise' rather than 'known compromise' 
but I am nonetheless shutting down the primary tools bastion as it is 
the most likely target of any potential tampering.


I've already moved all of the affected hostnames (e.g. 
tools-login.wmflabs.org) so if you get kicked off of the bastion, just 
reconnect and all will be more-or-less as it was.  Things will be a bit 
crowded until I have a new bastion built, but that should be done by 
tomorrow at the latest.


Sorry about the sudden shutdown!  I'll be writing up a complete incident 
report soon.


-Andrew


___
Labs-announce mailing list
labs-annou...@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-announce
___
Labs-l mailing list
Labs-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-l

Re: [Labs-l] Instance stuck in SHUTOFF state, unable to boot

[Andrew Bogott]

Andrew --

On 2/19/16 2:13 AM, Andrew "FastLizard4" Adams wrote:

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

Hello Labs folks,

One of our Labs instances for the account-creation-assistance project,
accounts-db2 (UUID 942cfe3f-8a7b-4e6f-ae5c-e5fcd5f16ed8) seems to be
stuck in the SHUTOFF state, and attempting to reboot it yields a "Failed
to reboot instance accounts-db2" message.  There's nothing in the
console output that indicates why the instance is down in the first place.
When I try to reboot on the commandline,  I'm told that this instance 
can't be rebooted
because it is in the 'stopped' state.  I did a 'start' on it rather than 
a 'reboot'

and it has accepted that and is now up and running.

The bad news is that the 'start' command is not currently available to 
non-Ops.

The good news is that my primary project right now is making a greater range
of OpenStack commands publicly available via Horizon.  So, in short, you 
couldn't
have fixed this yourself today, but should be able to deal with similar 
issues yourself

in a month or two.

Let me know if you have any other problems in the meantime!

-Andrew




Interestingly, this seems to be unrelated to the glibc-related reboots
earlier - users report that the account-creation-assistance instances
were unavailable for a while (presumably due to the reboots), then they
came back up again, then this particular instance went down again and is
still down.

Any assistance would be greatly appreciated.  Thanks in advance!
- -- 
Sincerely,

Andrew "FastLizard4" Adams



GPG Key ID: 0x221A627DD76E2616
-BEGIN PGP SIGNATURE-
Version: GnuPG v2.0.17 (MingW32)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQIcBAEBCAAGBQJWxs6TAAoJECIaYn3XbiYWA1cQALyzCJjF4CMWVnaMqTkGgXw5
PM9CFR0GxzvHEjSmibHM4L3NEoKnoKqZ33YKqMiizxDpSYaZCPCWbp8TWxpIHaGw
X3DY1fpsMkHOoi6zTrpA2O20V5ko5udiNlQ1y8KzvqZAOnDuKgndkH2o6WGjff3M
kl/JY6VgQWI3PNFohf3qd74MyyrfJIj/m9NWsHQFKsbA36IGcOpiTmtuODHths4j
mSsxGhGnzy4hyi4L1W4OW3ttFNCemiW1mjmmh6g9ZCW06OxRz857RtVcNAxafjf5
ObPjbmIFGxTni/14bymG38gl2uFpQATdL9HVS31kikDIEh0cQABXpfy5pTBrGSJy
6tKPlco53jbMPgiVfhdAuVkMEfeHRGjyR4ud44Qx6Sl+5kUUg9Y9ei9LwN2joEVs
x2LrXTxIRV1el/M25FS+EpmjnSFbZpEcXFBwE8SKbU+TV+OgLrob2D1MFsZuBF8I
sDsm6l3E1ZpNIE+Scy/CwTE3RP6M9ZvsHFD3aW8ohVZMZiZpGmbDOAExtiWo+6XP
fq6LF2Lxj3GugDpwl81kDOgHwMyGHd+OXbdFTzU32QC8qfBxdcdKyN+ffD+E+GlA
23xinGVO+4P9F6xoqqAG8+EogZmEmcg7jbc7ihHGDFyNyrOtaLRmgWFRriIdFpor
qb/k1U3BFjcDS/lbvVYk
=daub
-END PGP SIGNATURE-

___
Labs-l mailing list
Labs-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-l



___
Labs-l mailing list
Labs-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-l

[Labs-l] [Labs-announce] All Clear: Labs instance security reboots on Thursday, 2016-02-18 at 18:00 UTC

[Andrew Bogott]

Thanks to a long day spent by Chase and Yuvi, these reboots are now 
completed.


As always, it's probably a good idea to check your tools and/or web 
pages and make sure everything is still running.  Simple restarts should 
resolve most issues that you see.


-Andrew


On 2/16/16 10:24 AM, Andrew Bogott wrote:

Hello!

Due to a security update in glibc, we need to reboot all labs 
instances and virt hosts. The updates are currently being installed on 
all instances and the reboots will start at 18:00 UTC (10 AM in 
California) on Thursday.


A reboot is needed since the updated glibc packages are only effective 
once all processes using glibc have been restarted, and virtually all 
processes use glibc.


If you're running a labs instance which for some reason should not be 
rebooted currently, please email me directly with the details. We will 
do our best to insulate toollabs jobs from this reboot, but some 
long-running jobs may need to be restarted by hand after the reboots.


-Andrew




___
Labs-announce mailing list
labs-annou...@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-announce
___
Labs-l mailing list
Labs-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-l

[Labs-l] [Labs-announce] Labs instance security reboots on Thursday, 2016-02-18 at 18:00 UTC

[Andrew Bogott]

Hello!

Due to a security update in glibc, we need to reboot all labs instances 
and virt hosts. The updates are currently being installed on all 
instances and the reboots will start at 18:00 UTC (10 AM in California) 
on Thursday.


A reboot is needed since the updated glibc packages are only effective 
once all processes using glibc have been restarted, and virtually all 
processes use glibc.


If you're running a labs instance which for some reason should not be 
rebooted currently, please email me directly with the details. We will 
do our best to insulate toollabs jobs from this reboot, but some 
long-running jobs may need to be restarted by hand after the reboots.


-Andrew


___
Labs-announce mailing list
labs-annou...@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-announce
___
Labs-l mailing list
Labs-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-l

[Labs-l] [Labs-announce] Labsdb hardware failure: Some tools may a restart

[Andrew Bogott]

Hello!

One of our db servers (labsdb1002) just suffered a disk failure. We've 
rebalanced traffic to different servers, but depending on how tools are 
coded they may still be banging fruitlessly on the labsdb1002 door.  
Here are the databases that were moved:


'bgwiki'
'bgwiktionary',
'commonswiki'
'cswiki'
'dewiki'
'enwikiquote'
'enwiktionary'
'eowiki'
'fiwiki'
'idwiki'
'itwiki',
'nlwiki'
'nowiki'
'plwiki'
'ptwiki'
'svwiki'
'thwiki'
'trwiki'
'wikidatawiki'
'zhwiki'

If your tool relies on one of those databases, please verify that it is 
still up.  If it isn't, a simple restart should get you back on track, 
but you might also want to consider adding code that detects db failures 
and reconnects.


Sorry for the interruption!

-Andrew
___
Labs-announce mailing list
labs-annou...@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-announce
___
Labs-l mailing list
Labs-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-l

Re: [Labs-l] Labs-announce post from jamison.loftho...@gmail.com requires approval

[Andrew Bogott]

jamison.lofthouse@gmail.comwrote:

>> If all goes perfectly, Wikitech will be restored to its original 
state but be slightly slower.


> Could you explain this a bit more? Is this due to the new role data 
storage backend? How much of a slowdown should we expect?


Things will be slower in large part because of limitations in the 
keystone api -- in some cases where we were able to do one big ldap 
query to answer a question like "what projects am I in?" we now have to 
do clumsy things like ask each individual project "am I a member of 
you?" which adds up to a lot of REST calls.


Most everything is cached, though, so the slow page loads should only 
happen occasionally.  If you run into a case where performance is 
unbearably slow, let me know and I'll see about optimizing.


-Andrew


___
Labs-l mailing list
Labs-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-l

Re: [Labs-l] [Labs-announce] Wikitech maintenance and CI downtime tomorrow, 2015-02-12 16:00 UTC

[Andrew Bogott]

Update:

This migration is complete.  There are a bunch of bugs with labs 
features in wikitech which I'm now hunting down one by one.  In the 
meantime, CI should be working again, and most common wikitech actions 
should mostly work.


I'll send another 'all clear' once I've stamped out some of the more 
obvious bugs.  In the meantime maybe hold off on filing phab tickets 
since most likely we already know :)


-Andrew


On 2/11/16 9:54 AM, Andrew Bogott wrote:

Executive summary:

CI tests won't run for a while tomorrow.  Also you won't be able 
to log into Wikitech.  This will begin at 16:00 UTC (8:00 AM in 
California) and may take a couple of hours.


Full story:

As part of a long-overdue tech-debt payment[0], I'll be migrating 
all of the Labs project, membership and role data from ldap into 
mysql[1].  Keystone (the Labs Openstack authentication layer) will be 
shut off briefly, and subsequently will have incomplete data while 
things are gradually copied over from Ldap.
At the same time, Alex will be rolling out a bunch of 
OpenStackManager patches[2] to cope with the new reality.  I've tested 
these quite a lot, but no doubt there will be unexpected issues with 
such a big refactor.


Existing, running labs and tools sessions should not be affected.  
New instance creation will be disabled for part of the window, which 
will prevent CI from starting new tests.  Wikitech login will be 
disabled, and users will have to login afresh after logins are restored.


There is, unfortunately, no user-facing improvement associated 
with this update.  If all goes perfectly, Wikitech will be restored to 
its original state but be slightly slower.  If, after the update, 
anyone encounters new Wikitech issues (and I emphasize the NEW), 
please open a phab ticket and inform me immediately.


As always, in the worst case scenario you can refer to our doc 
site of last resort, https://wikitech-static.wikimedia.org



-Andrew


[0] https://phabricator.wikimedia.org/T115029
[1] https://wikitech.wikimedia.org/wiki/Labs_keystone_roles
[2] The patchset begins with https://gerrit.wikimedia.org/r/#/c/252615/



___
Labs-announce mailing list
labs-annou...@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-announce
___
Labs-l mailing list
Labs-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-l

Re: [Labs-l] Wikitech maintenance and CI downtime tomorrow, 2015-02-12 16:00 UTC

[Andrew Bogott]

Alex and I have spent much of the day hunting down bugs, and the obvious 
ones are now fixed.  The only issue that I'm still seeing is that if 
you're a member of every single project, logins are very slow... 
fortunately I am probably the only user bit by this :)


So, now:  please open a phab task if you encounter bad behavior on 
wikitech, and assign it to me or direct my attention to it via email or IRC.


Many thanks to Alex Monk for all his code review, and thanks to everyone 
for your patience with the downtime.


-Andrew


On 2/12/16 12:11 PM, Andrew Bogott wrote:

Update:

This migration is complete.  There are a bunch of bugs with labs 
features in wikitech which I'm now hunting down one by one.  In the 
meantime, CI should be working again, and most common wikitech actions 
should mostly work.


I'll send another 'all clear' once I've stamped out some of the more 
obvious bugs.  In the meantime maybe hold off on filing phab tickets 
since most likely we already know :)


-Andrew


On 2/11/16 9:54 AM, Andrew Bogott wrote:

Executive summary:

CI tests won't run for a while tomorrow.  Also you won't be able 
to log into Wikitech.  This will begin at 16:00 UTC (8:00 AM in 
California) and may take a couple of hours.


Full story:

As part of a long-overdue tech-debt payment[0], I'll be migrating 
all of the Labs project, membership and role data from ldap into 
mysql[1].  Keystone (the Labs Openstack authentication layer) will be 
shut off briefly, and subsequently will have incomplete data while 
things are gradually copied over from Ldap.
At the same time, Alex will be rolling out a bunch of 
OpenStackManager patches[2] to cope with the new reality.  I've 
tested these quite a lot, but no doubt there will be unexpected 
issues with such a big refactor.


Existing, running labs and tools sessions should not be 
affected.  New instance creation will be disabled for part of the 
window, which will prevent CI from starting new tests. Wikitech login 
will be disabled, and users will have to login afresh after logins 
are restored.


There is, unfortunately, no user-facing improvement associated 
with this update.  If all goes perfectly, Wikitech will be restored 
to its original state but be slightly slower. If, after the update, 
anyone encounters new Wikitech issues (and I emphasize the NEW), 
please open a phab ticket and inform me immediately.


As always, in the worst case scenario you can refer to our doc 
site of last resort, https://wikitech-static.wikimedia.org



-Andrew


[0] https://phabricator.wikimedia.org/T115029
[1] https://wikitech.wikimedia.org/wiki/Labs_keystone_roles
[2] The patchset begins with https://gerrit.wikimedia.org/r/#/c/252615/





___
Labs-l mailing list
Labs-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-l

[Labs-l] [Labs-announce] Wikitech maintenance and CI downtime tomorrow, 2015-02-12 16:00 UTC

[Andrew Bogott]

Executive summary:

CI tests won't run for a while tomorrow.  Also you won't be able to 
log into Wikitech.  This will begin at 16:00 UTC (8:00 AM in California) 
and may take a couple of hours.


Full story:

As part of a long-overdue tech-debt payment[0], I'll be migrating 
all of the Labs project, membership and role data from ldap into 
mysql[1].  Keystone (the Labs Openstack authentication layer) will be 
shut off briefly, and subsequently will have incomplete data while 
things are gradually copied over from Ldap.
At the same time, Alex will be rolling out a bunch of 
OpenStackManager patches[2] to cope with the new reality.  I've tested 
these quite a lot, but no doubt there will be unexpected issues with 
such a big refactor.


Existing, running labs and tools sessions should not be affected.  
New instance creation will be disabled for part of the window, which 
will prevent CI from starting new tests.  Wikitech login will be 
disabled, and users will have to login afresh after logins are restored.


There is, unfortunately, no user-facing improvement associated with 
this update.  If all goes perfectly, Wikitech will be restored to its 
original state but be slightly slower.  If, after the update, anyone 
encounters new Wikitech issues (and I emphasize the NEW), please open a 
phab ticket and inform me immediately.


As always, in the worst case scenario you can refer to our doc site 
of last resort, https://wikitech-static.wikimedia.org



-Andrew


[0] https://phabricator.wikimedia.org/T115029
[1] https://wikitech.wikimedia.org/wiki/Labs_keystone_roles
[2] The patchset begins with https://gerrit.wikimedia.org/r/#/c/252615/

___
Labs-announce mailing list
labs-annou...@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-announce
___
Labs-l mailing list
Labs-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-l

[Labs-l] [Labs-announce] Wikitech outage Friday 2015-02-05 at 15:00 UTC

[Andrew Bogott]

wikitech.wikimedia.org will be unavailable for a short time on 
Friday, beginning around 7AM San Francisco time while we apply kernel 
updates and reboot[1].  Ideally the downtime will only last 5-10 minutes 
but I've scheduled an hour window on the deploy calendar in case any of 
the boxes have trouble reviving.


Labs instances and Toollabs services should not be affected. Some 
alerts may fire during the window as we're planning to reboot one of the 
monitoring hosts and the effects of that are unpredictable.


In case of emergency, the content of wikitech (minus some images) 
is always available on the external backup: 
https://wikitech-static.wikimedia.org/wiki/Main_Page


-Andrew


[1] For the curious (and my future reference), we're rebooting silver, 
holmium, labcontrol1002, labnet1001, labmon1001


___
Labs-announce mailing list
labs-annou...@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-announce
___
Labs-l mailing list
Labs-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-l

Re: [Labs-l] [Labs-announce] IMPORTANT: rolling reboots today (DONE)

[Andrew Bogott]

All reboots are now complete.  Let us know if you see any new, unusual 
behavior.


Tools users:  non-continuous jobs were most likely killed by the 
reboots.  You will need to re-run things by hand if such services are 
not managed by cron.


-Andrew


On 1/20/16 5:31 PM, Andrew Bogott wrote:

On 1/20/16 12:06 PM, Andrew Bogott wrote:
In response to a new kernel exploit revealed today, I will be 
upgrading and rebooting all labs VMs today.  We'll be running a 
dist-upgrade on all reachable labs instances, then rebooting the virt 
hosts (which will, in turn, reboot the associated labs vms.) Total 
downtime for any given instance should only be 5-10 minutes, but you 
will need to restart services and such after the restart.
Update -- getting staged for reboot is taking much longer than 
expected.  The reboots will most likely happen tomorrow 2016-12-21, at 
around 17:00 UTC.




We'll do our best to ensure that tools survive the reboots -- more on 
that as the day develops.


-Andrew







___
Labs-announce mailing list
labs-annou...@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-announce
___
Labs-l mailing list
Labs-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-l

Re: [Labs-l] [Labs-announce] IMPORTANT: rolling reboots today

[Andrew Bogott]

On 1/20/16 12:06 PM, Andrew Bogott wrote:
In response to a new kernel exploit revealed today, I will be 
upgrading and rebooting all labs VMs today.  We'll be running a 
dist-upgrade on all reachable labs instances, then rebooting the virt 
hosts (which will, in turn, reboot the associated labs vms.) Total 
downtime for any given instance should only be 5-10 minutes, but you 
will need to restart services and such after the restart.
Update -- getting staged for reboot is taking much longer than 
expected.  The reboots will most likely happen tomorrow 2016-12-21, at 
around 17:00 UTC.




We'll do our best to ensure that tools survive the reboots -- more on 
that as the day develops.


-Andrew





___
Labs-announce mailing list
labs-annou...@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-announce
___
Labs-l mailing list
Labs-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-l

[Labs-l] [Labs-announce] IMPORTANT: rolling reboots today

[Andrew Bogott]

In response to a new kernel exploit revealed today, I will be upgrading 
and rebooting all labs VMs today.  We'll be running a dist-upgrade on 
all reachable labs instances, then rebooting the virt hosts (which will, 
in turn, reboot the associated labs vms.) Total downtime for any given 
instance should only be 5-10 minutes, but you will need to restart 
services and such after the restart.


We'll do our best to ensure that tools survive the reboots -- more on 
that as the day develops.


-Andrew



___
Labs-announce mailing list
labs-annou...@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-announce
___
Labs-l mailing list
Labs-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-l

Re: [Labs-l] Query results are inconsistent between DBs

[Andrew Bogott]

On 1/17/16 4:04 PM, Huji Lee wrote:

How can I ask for that replica to be synchronized? Phabricator task?


Yes, please open a phabricator case. Also be warned that our one and 
only DBA is on holiday so it may be a few days before we're able to 
address it adequately.


-Andrew


On Fri, Jan 15, 2016 at 8:29 PM, John > wrote:


Just ran the query across all three DB servers and
labsdb1003.eqiad.wmnet is the culprit

On Fri, Jan 15, 2016 at 8:21 PM, Huji Lee > wrote:

When I run this query on Quarry:

https://quarry.wmflabs.org/query/6855

I get 17 results, all of which are unused images. When my bot
runs the exact same query, it gets 18 results, as shown on
this FA WP page

.

The one extra is
Image:Taylor_Swift_-_Wildest_Dreams_(Official_Single_Cover).png 

which is NOT an unused image (it is used in one page on FA
WP). It seems the DB instances that the bot hits is
inconsistent. The bot is called HujiBot and is hosted on Labs
servers.

If you need me to run additional comments to pinpoint the root
cause please let me know.

Best,

Huji

___
Labs-l mailing list
Labs-l@lists.wikimedia.org 
https://lists.wikimedia.org/mailman/listinfo/labs-l



___
Labs-l mailing list
Labs-l@lists.wikimedia.org 
https://lists.wikimedia.org/mailman/listinfo/labs-l




___
Labs-l mailing list
Labs-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-l


___
Labs-l mailing list
Labs-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-l