Re: [OpenStack-Infra] Wiki.o.o sustaining spam attack

[Marton Kiss]

Hi,

I created the following patch, infra cores must approve that:
https://review.openstack.org/285641 Add ssh key of JP Maxwell to wiki.o.o

Marton

On Sat, Feb 27, 2016 at 6:41 AM JP Maxwell  wrote:

> Marton has SSH access and applied a patch earlier today.  It appears the
> spam continues to flow:
>
>
> https://wiki.openstack.org/wiki/40_Thoughts_Of_Using_Open_Shelves_On_A_Kitchen
>
> Marton let me know if you can look at it some more or Infra if you want to
> give me SSH I'll do so as well in the morning (public key attached).
>
>
>
> ssh-rsa
> B3NzaC1yc2EBIwAAAQEA2b5I7Yff9FCrtRmSjpILUePi54Vbc8zqJTbzrIAQZGFLBi3xd2MLlhV5QVgpDBC9H3lGjbdnc81D3aFd3HwHT4dvvvyedT12PR3VDEpftdW84vw3jzdtALcayOQznjbGnScwvX5SgnRhNxuX9Rkh8qNvOsjYPUafRr9azkQoomJFkdNVI4Vb5DbLhTpt18FPeOf0UuqDt/J2tHI4SjZ3kjzr7Nbwpg8xGgANPNE0+2pJbwCA8YDt4g3bzfzvVafQs5o9Gfc9tudkR9ugQG1M+EWCgu42CleOwMTd/rYEB2fgNNPsZAWqwQfdPajVuk70EBKUEQSyoA09eEZX+xJN9Q==
> jpmax...@tipit.net
>
>
>
>
> J.P. Maxwell / tipit.net 
>
>
> On Fri, Feb 26, 2016 at 12:09 PM, Jimmy McArthur 
> wrote:
>
>> Super thankful for all the folks that have jumped in over the last couple
>> of days to help with the puppetization, etc... I just feel like we're
>> taking a very wrong approach here.
>>
>> Paul Belanger wrote:
>>
>> Right, and I don't have an issue with that approach.  Based on the work we 
>> did
>> yesterday, anybody can do that via our workflow. Please submit a patch to
>> puppet-mediawiki[1] and ping an infra-root in #openstack-infra IRC.
>>
>> What I'm proposing is the workflow is really meant for software, not for
>> web applications. It's tedious and time consuming when what's needed here
>> is a set of tests on the server. Submitting a patch, waiting for a +1, then
>> getting on IRC to find someone with access (and time) to paste the logs is
>> a pretty time consuming process for what should be a series of rapid-fire
>> changes/fixes on the server. Especially when we're dealign with an active
>> attack.
>>
>>
>> We can then have somebody look at the logs.  I think it is more about 
>> scheduling
>> the task since more infra-root as travling back from the mid-cycle last night
>> and today.
>>
>> Right, this is my point. This has been going on for 3 weeks (or more).
>> Tom Fifeldt was asking for help without response. And here we are through
>> another week and no closer to stemming the flow.
>>
>> I'm fully aware what I'm proposing goes against what Infra and the
>> OpenStack workflow is all about, but I'd ask you all to look at this from a
>> web development perspective instead of a software development perspective.
>>
>> Jimmy
>>
>>
>> Last email from me, just on a plane.  Will follow up when I land.
>>
>> [1] https://git.openstack.org/cgit/openstack-infra/puppet-mediawiki
>>
>>  J.P. Maxwell | tipit.net [http://tipit.net] | fibercove.com
>> [http://www.fibercove.com]
>> On Fri, Feb 26, 2016 at 11:25 AM, Paul Belanger  
>> 
>> wrote:
>> On Fri, Feb 26, 2016 at 11:08:18AM -0600, Jimmy McArthur wrote:
>>
>> Given the state of the wiki a the moment, I think taking the quickest path
>> to get it fixed would be prudent. Is there a way we can get JP root access
>> to this server, even temporarily? We get 25% of our website traffic (2
>> million visitors) to the wiki. I realize we're all after the same thing,
>>
>> but
>>
>> spammers are not going to hit the dev environment, so there's really no
>>
>> way
>>
>> to tell if teh problem is fixed without actually working directly on the
>> production machine. This should be a 30 minute fix.
>>
>>
>> I am still unclear what the 30min fix is. If really 30mins, then it
>> shouldn't be
>> hard to get the fix into our workflow. Could somebody please elaborate.
>>
>> If we are talking about deploying new versions of php or mediawiki manually,
>> I
>> not be in-favor of this. To me, while the attack sucks, we should be working
>> on
>> 2 fronts. Getting the help needed to mitigate the attack, then adding the
>> changes into -infra workflow in parallel.
>>
>>
>> I realize there is a lot of risk in giving ssh access to infra machines,
>>
>> but
>>
>> I think it's worth taking a look at either putting this machine in a place
>> where a different level of admin could access it without giving away the
>> keys to the entire OpenStack infrastructure or figuring out a way to set
>>
>> up
>>
>> credentials with varying levels of access.
>>
>>
>> As a note, all the work I've been doing to help with the attack hasn't
>> require
>> SSH access for me to wiki.o.o. I did need infra-root help to expose our
>> configuration safely. I'd rather take some time to see what the fixes are,
>> having infra-root apply changes, then move them into puppet.
>>
>> It also has been discussed to simply disable write access to the wiki if we
>> really want spamming to stop, obviously that will affect normal usage.
>>
>>
>> Jimmy
>>
>> Paul Belanger wrote:

Re: [OpenStack-Infra] Wiki.o.o sustaining spam attack

[JP Maxwell]

Marton has SSH access and applied a patch earlier today.  It appears the
spam continues to flow:

https://wiki.openstack.org/wiki/40_Thoughts_Of_Using_Open_Shelves_On_A_Kitchen

Marton let me know if you can look at it some more or Infra if you want to
give me SSH I'll do so as well in the morning (public key attached).



ssh-rsa
B3NzaC1yc2EBIwAAAQEA2b5I7Yff9FCrtRmSjpILUePi54Vbc8zqJTbzrIAQZGFLBi3xd2MLlhV5QVgpDBC9H3lGjbdnc81D3aFd3HwHT4dvvvyedT12PR3VDEpftdW84vw3jzdtALcayOQznjbGnScwvX5SgnRhNxuX9Rkh8qNvOsjYPUafRr9azkQoomJFkdNVI4Vb5DbLhTpt18FPeOf0UuqDt/J2tHI4SjZ3kjzr7Nbwpg8xGgANPNE0+2pJbwCA8YDt4g3bzfzvVafQs5o9Gfc9tudkR9ugQG1M+EWCgu42CleOwMTd/rYEB2fgNNPsZAWqwQfdPajVuk70EBKUEQSyoA09eEZX+xJN9Q==
jpmax...@tipit.net




J.P. Maxwell / tipit.net 


On Fri, Feb 26, 2016 at 12:09 PM, Jimmy McArthur 
wrote:

> Super thankful for all the folks that have jumped in over the last couple
> of days to help with the puppetization, etc... I just feel like we're
> taking a very wrong approach here.
>
> Paul Belanger wrote:
>
> Right, and I don't have an issue with that approach.  Based on the work we did
> yesterday, anybody can do that via our workflow. Please submit a patch to
> puppet-mediawiki[1] and ping an infra-root in #openstack-infra IRC.
>
> What I'm proposing is the workflow is really meant for software, not for
> web applications. It's tedious and time consuming when what's needed here
> is a set of tests on the server. Submitting a patch, waiting for a +1, then
> getting on IRC to find someone with access (and time) to paste the logs is
> a pretty time consuming process for what should be a series of rapid-fire
> changes/fixes on the server. Especially when we're dealign with an active
> attack.
>
>
> We can then have somebody look at the logs.  I think it is more about 
> scheduling
> the task since more infra-root as travling back from the mid-cycle last night
> and today.
>
> Right, this is my point. This has been going on for 3 weeks (or more). Tom
> Fifeldt was asking for help without response. And here we are through
> another week and no closer to stemming the flow.
>
> I'm fully aware what I'm proposing goes against what Infra and the
> OpenStack workflow is all about, but I'd ask you all to look at this from a
> web development perspective instead of a software development perspective.
>
> Jimmy
>
>
> Last email from me, just on a plane.  Will follow up when I land.
>
> [1] https://git.openstack.org/cgit/openstack-infra/puppet-mediawiki
>
>  J.P. Maxwell | tipit.net [http://tipit.net] | fibercove.com
> [http://www.fibercove.com]
> On Fri, Feb 26, 2016 at 11:25 AM, Paul Belanger  
> 
> wrote:
> On Fri, Feb 26, 2016 at 11:08:18AM -0600, Jimmy McArthur wrote:
>
> Given the state of the wiki a the moment, I think taking the quickest path
> to get it fixed would be prudent. Is there a way we can get JP root access
> to this server, even temporarily? We get 25% of our website traffic (2
> million visitors) to the wiki. I realize we're all after the same thing,
>
> but
>
> spammers are not going to hit the dev environment, so there's really no
>
> way
>
> to tell if teh problem is fixed without actually working directly on the
> production machine. This should be a 30 minute fix.
>
>
> I am still unclear what the 30min fix is. If really 30mins, then it
> shouldn't be
> hard to get the fix into our workflow. Could somebody please elaborate.
>
> If we are talking about deploying new versions of php or mediawiki manually,
> I
> not be in-favor of this. To me, while the attack sucks, we should be working
> on
> 2 fronts. Getting the help needed to mitigate the attack, then adding the
> changes into -infra workflow in parallel.
>
>
> I realize there is a lot of risk in giving ssh access to infra machines,
>
> but
>
> I think it's worth taking a look at either putting this machine in a place
> where a different level of admin could access it without giving away the
> keys to the entire OpenStack infrastructure or figuring out a way to set
>
> up
>
> credentials with varying levels of access.
>
>
> As a note, all the work I've been doing to help with the attack hasn't
> require
> SSH access for me to wiki.o.o. I did need infra-root help to expose our
> configuration safely. I'd rather take some time to see what the fixes are,
> having infra-root apply changes, then move them into puppet.
>
> It also has been discussed to simply disable write access to the wiki if we
> really want spamming to stop, obviously that will affect normal usage.
>
>
> Jimmy
>
> Paul Belanger wrote:
>
> On Fri, Feb 26, 2016 at 10:12:12AM -0600, JP Maxwell wrote:
>
> But if you wanted to upgrade everything, remove the mobile view
>
> extension,
>
> test in a dev/staging environment then deploy to production fingers
> crossed, I think that would be a valid approach as well.
>
>
> Current review up[1]. I'll launch a node tonight / tomorrow locally to
>

Re: [OpenStack-Infra] Wiki.o.o sustaining spam attack

[Jimmy McArthur]

Super thankful for all the folks that have jumped in over the last 
couple of days to help with the puppetization, etc... I just feel like 
we're taking a very wrong approach here.


Paul Belanger wrote:

Right, and I don't have an issue with that approach.  Based on the work we did
yesterday, anybody can do that via our workflow. Please submit a patch to
puppet-mediawiki[1] and ping an infra-root in #openstack-infra IRC.
What I'm proposing is the workflow is really meant for software, not for 
web applications. It's tedious and time consuming when what's needed 
here is a set of tests on the server. Submitting a patch, waiting for a 
+1, then getting on IRC to find someone with access (and time) to paste 
the logs is a pretty time consuming process for what should be a series 
of rapid-fire changes/fixes on the server. Especially when we're dealign 
with an active attack.


We can then have somebody look at the logs.  I think it is more about scheduling
the task since more infra-root as travling back from the mid-cycle last night
and today.
Right, this is my point. This has been going on for 3 weeks (or more). 
Tom Fifeldt was asking for help without response. And here we are 
through another week and no closer to stemming the flow.


I'm fully aware what I'm proposing goes against what Infra and the 
OpenStack workflow is all about, but I'd ask you all to look at this 
from a web development perspective instead of a software development 
perspective.


Jimmy


Last email from me, just on a plane.  Will follow up when I land.

[1] https://git.openstack.org/cgit/openstack-infra/puppet-mediawiki



J.P. Maxwell | tipit.net [http://tipit.net] | fibercove.com
[http://www.fibercove.com]
On Fri, Feb 26, 2016 at 11:25 AM, Paul Belanger
wrote:
On Fri, Feb 26, 2016 at 11:08:18AM -0600, Jimmy McArthur wrote:

Given the state of the wiki a the moment, I think taking the quickest path
to get it fixed would be prudent. Is there a way we can get JP root access
to this server, even temporarily? We get 25% of our website traffic (2
million visitors) to the wiki. I realize we're all after the same thing,

but

spammers are not going to hit the dev environment, so there's really no

way

to tell if teh problem is fixed without actually working directly on the
production machine. This should be a 30 minute fix.


I am still unclear what the 30min fix is. If really 30mins, then it
shouldn't be
hard to get the fix into our workflow. Could somebody please elaborate.

If we are talking about deploying new versions of php or mediawiki manually,
I
not be in-favor of this. To me, while the attack sucks, we should be working
on
2 fronts. Getting the help needed to mitigate the attack, then adding the
changes into -infra workflow in parallel.


I realize there is a lot of risk in giving ssh access to infra machines,

but

I think it's worth taking a look at either putting this machine in a place
where a different level of admin could access it without giving away the
keys to the entire OpenStack infrastructure or figuring out a way to set

up

credentials with varying levels of access.


As a note, all the work I've been doing to help with the attack hasn't
require
SSH access for me to wiki.o.o. I did need infra-root help to expose our
configuration safely. I'd rather take some time to see what the fixes are,
having infra-root apply changes, then move them into puppet.

It also has been discussed to simply disable write access to the wiki if we
really want spamming to stop, obviously that will affect normal usage.


Jimmy

Paul Belanger wrote:

On Fri, Feb 26, 2016 at 10:12:12AM -0600, JP Maxwell wrote:

But if you wanted to upgrade everything, remove the mobile view

extension,

test in a dev/staging environment then deploy to production fingers
crossed, I think that would be a valid approach as well.


Current review up[1]. I'll launch a node tonight / tomorrow locally to

see
how

puppet reacts. I suspect there will be some issues.

If infra-roots are fine with this approach, we can use that box to test

against.

[1] https://review.openstack.org/#/c/285405/


J.P. Maxwell | tipit.net | fibercove.com
On Feb 26, 2016 10:08 AM, "JP Maxwell"  wrote:


Plus one except in this case it is much easier to know if our efforts

are

working on production because the spam either stops or not.

J.P. Maxwell | tipit.net | fibercove.com
On Feb 26, 2016 9:48 AM, "Paul Belanger"  wrote:


On Fri, Feb 26, 2016 at 09:18:00AM -0600, JP Maxwell wrote:

I really think you might consider the option that there is a

vulnerability

in one of the extensions. If that is the case black listing IPs will

be

an

ongoing wild goose chase.

I think this would be easily proven or disproven by making the questy
question impossible and see if the spam continues.


We'll have to let an infra-root make that call. Since nobody would be
able to
use the wiki. Honestly, I'd rather spend the time standing 

Re: [OpenStack-Infra] Wiki.o.o sustaining spam attack

[JP Maxwell]

A quick google indicates this may be an unrelated issue that should be fixed,
but I don’t *think* it is related to the spam.

J.P. Maxwell | tipit.net [http://tipit.net] | fibercove.com 
[http://www.fibercove.com]
On Fri, Feb 26, 2016 at 11:56 AM, Marton Kiss  wrote:
I'm going to get a dinner, but I'll be on irc after, so if I can help somehow, I
will be here. #openstack-infra mrmartin
M.
On Fri, Feb 26, 2016 at 6:51 PM Paul Belanger < pbela...@redhat.com 
[pbela...@redhat.com] > wrote:
On phone but patch puppet-mediawiki and enable captcha for all pages. We only
did edit and create

On Feb 26, 2016 10:38 AM, Marton Kiss < marton.k...@gmail.com 
[marton.k...@gmail.com] > wrote:

I see a ton of incoming post requests:
POST
/w/index.php?title=Special%3ARunJobs=jobs=1=1456508270=571cfb216f944b15d2eee1c0253d08b77003328e

M.
On Fri, Feb 26, 2016 at 6:35 PM Marton Kiss < marton.k...@gmail.com 
[marton.k...@gmail.com] > wrote:
Oh, I can login. So what we need?
M.
On Fri, Feb 26, 2016 at 6:33 PM JP Maxwell < j...@tipit.net [j...@tipit.net] > 
wrote:
I think what Jimmy is referring to is what I was suggesting by removing the
extensions / making the question impossible to answer. Basically a series of
rapid fire changes while tailing the logs and seeing what stops the spam. Once
you know what worked then you can submit as an official patch. But being able to
quickly try these things on a server actually under attack is the fastest path
toward identifying the fix.

J.P. Maxwell | tipit.net [http://tipit.net] | fibercove.com 
[http://www.fibercove.com]
On Fri, Feb 26, 2016 at 11:25 AM, Paul Belanger < pabelan...@redhat.com 
[pabelan...@redhat.com] > wrote:
On Fri, Feb 26, 2016 at 11:08:18AM -0600, Jimmy McArthur wrote:
> Given the state of the wiki a the moment, I think taking the quickest path
> to get it fixed would be prudent. Is there a way we can get JP root access
> to this server, even temporarily? We get 25% of our website traffic (2
> million visitors) to the wiki. I realize we're all after the same thing, but
> spammers are not going to hit the dev environment, so there's really no way
> to tell if teh problem is fixed without actually working directly on the
> production machine. This should be a 30 minute fix.
>
I am still unclear what the 30min fix is. If really 30mins, then it shouldn't be
hard to get the fix into our workflow. Could somebody please elaborate.

If we are talking about deploying new versions of php or mediawiki manually, I
not be in-favor of this. To me, while the attack sucks, we should be working on
2 fronts. Getting the help needed to mitigate the attack, then adding the
changes into -infra workflow in parallel.

> I realize there is a lot of risk in giving ssh access to infra machines, but
> I think it's worth taking a look at either putting this machine in a place
> where a different level of admin could access it without giving away the
> keys to the entire OpenStack infrastructure or figuring out a way to set up
> credentials with varying levels of access.
>
As a note, all the work I've been doing to help with the attack hasn't require
SSH access for me to wiki.o.o. I did need infra-root help to expose our
configuration safely. I'd rather take some time to see what the fixes are,
having infra-root apply changes, then move them into puppet.

It also has been discussed to simply disable write access to the wiki if we
really want spamming to stop, obviously that will affect normal usage.

> Jimmy
>
> Paul Belanger wrote:
> >On Fri, Feb 26, 2016 at 10:12:12AM -0600, JP Maxwell wrote:
> >>But if you wanted to upgrade everything, remove the mobile view extension,
> >>test in a dev/staging environment then deploy to production fingers
> >>crossed, I think that would be a valid approach as well.
> >>
> >Current review up[1]. I'll launch a node tonight / tomorrow locally to see
how
> >puppet reacts. I suspect there will be some issues.
> >
> >If infra-roots are fine with this approach, we can use that box to test
against.
> >
> >[1] https://review.openstack.org/#/c/285405/
[https://review.openstack.org/#/c/285405/]
> >
> >>J.P. Maxwell | tipit.net [http://tipit.net] | fibercove.com 
> >>[http://fibercove.com]
> >>On Feb 26, 2016 10:08 AM, "JP Maxwell"< j...@tipit.net [j...@tipit.net] > 
> >>wrote:
> >>
> >>>Plus one except in this case it is much easier to know if our efforts are
> >>>working on production because the spam either stops or not.
> >>>
> >>>J.P. Maxwell | tipit.net [http://tipit.net] | fibercove.com 
> >>>[http://fibercove.com]
> >>>On Feb 26, 2016 9:48 AM, "Paul Belanger"< pabelan...@redhat.com 
> >>>[pabelan...@redhat.com] > wrote:
> >>>
> On Fri, Feb 26, 2016 at 09:18:00AM -0600, JP Maxwell wrote:
> >I really think you might consider the option that there is a
> vulnerability
> >in one of the extensions. If that is the case black listing IPs will be
> an
> >ongoing wild goose chase.
> >
> >I think this would be easily 

Re: [OpenStack-Infra] Wiki.o.o sustaining spam attack

[Marton Kiss]

Yeah, I checked it and it is internal job runner:
https://www.mediawiki.org/wiki/Manual:Job_queue

M.

On Fri, Feb 26, 2016 at 7:00 PM JP Maxwell  wrote:

> A quick google indicates this may be an unrelated issue that should be
> fixed, but I don’t *think* it is related to the spam.
>
> *J.P. Maxwell* | tipit.net | fibercove.com 
>
> On Fri, Feb 26, 2016 at 11:56 AM, Marton Kiss 
> wrote:
>
> I'm going to get a dinner, but I'll be on irc after, so if I can help
> somehow, I will be here. #openstack-infra mrmartin
>
> M.
>
> On Fri, Feb 26, 2016 at 6:51 PM Paul Belanger  wrote:
>
>> On phone but patch puppet-mediawiki and enable captcha for all pages. We
>> only did edit and create
>> On Feb 26, 2016 10:38 AM, Marton Kiss  wrote:
>>
>> I see a ton of incoming post requests:
>>
>> POST
>> /w/index.php?title=Special%3ARunJobs=jobs=1=1456508270=571cfb216f944b15d2eee1c0253d08b77003328e
>>
>> M.
>>
>> On Fri, Feb 26, 2016 at 6:35 PM Marton Kiss 
>> wrote:
>>
>>> Oh, I can login. So what we need?
>>>
>>> M.
>>>
>>> On Fri, Feb 26, 2016 at 6:33 PM JP Maxwell  wrote:
>>>
 I think what Jimmy is referring to is what I was suggesting by removing
 the extensions / making the question impossible to answer. Basically a
 series of rapid fire changes while tailing the logs and seeing what stops
 the spam. Once you know what worked then you can submit as an official
 patch. But being able to quickly try these things on a server actually
 under attack is the fastest path toward identifying the fix.

 *J.P. Maxwell* | tipit.net | fibercove.com 

 On Fri, Feb 26, 2016 at 11:25 AM, Paul Belanger 
 wrote:

 On Fri, Feb 26, 2016 at 11:08:18AM -0600, Jimmy McArthur wrote:
 > Given the state of the wiki a the moment, I think taking the quickest
 path
 > to get it fixed would be prudent. Is there a way we can get JP root
 access
 > to this server, even temporarily? We get 25% of our website traffic (2
 > million visitors) to the wiki. I realize we're all after the same
 thing, but
 > spammers are not going to hit the dev environment, so there's really
 no way
 > to tell if teh problem is fixed without actually working directly on
 the
 > production machine. This should be a 30 minute fix.
 >
 I am still unclear what the 30min fix is. If really 30mins, then it
 shouldn't be
 hard to get the fix into our workflow. Could somebody please elaborate.

 If we are talking about deploying new versions of php or mediawiki
 manually, I
 not be in-favor of this. To me, while the attack sucks, we should be
 working on
 2 fronts. Getting the help needed to mitigate the attack, then adding
 the
 changes into -infra workflow in parallel.

 > I realize there is a lot of risk in giving ssh access to infra
 machines, but
 > I think it's worth taking a look at either putting this machine in a
 place
 > where a different level of admin could access it without giving away
 the
 > keys to the entire OpenStack infrastructure or figuring out a way to
 set up
 > credentials with varying levels of access.
 >
 As a note, all the work I've been doing to help with the attack hasn't
 require
 SSH access for me to wiki.o.o. I did need infra-root help to expose our
 configuration safely. I'd rather take some time to see what the fixes
 are,
 having infra-root apply changes, then move them into puppet.

 It also has been discussed to simply disable write access to the wiki
 if we
 really want spamming to stop, obviously that will affect normal usage.

 > Jimmy
 >
 > Paul Belanger wrote:
 > >On Fri, Feb 26, 2016 at 10:12:12AM -0600, JP Maxwell wrote:
 > >>But if you wanted to upgrade everything, remove the mobile view
 extension,
 > >>test in a dev/staging environment then deploy to production fingers
 > >>crossed, I think that would be a valid approach as well.
 > >>
 > >Current review up[1]. I'll launch a node tonight / tomorrow locally
 to see how
 > >puppet reacts. I suspect there will be some issues.
 > >
 > >If infra-roots are fine with this approach, we can use that box to
 test against.
 > >
 > >[1] https://review.openstack.org/#/c/285405/
 > >
 > >>J.P. Maxwell | tipit.net | fibercove.com
 > >>On Feb 26, 2016 10:08 AM, "JP Maxwell" wrote:
 > >>
 > >>>Plus one except in this case it is much easier to know if our
 efforts are
 > >>>working on production because the spam either stops or not.
 > >>>
 > >>>J.P. Maxwell | tipit.net | fibercove.com
 > >>>On Feb 26, 2016 9:48 AM, "Paul Belanger"

Re: [OpenStack-Infra] Wiki.o.o sustaining spam attack

[Marton Kiss]

I'm going to get a dinner, but I'll be on irc after, so if I can help
somehow, I will be here. #openstack-infra mrmartin

M.

On Fri, Feb 26, 2016 at 6:51 PM Paul Belanger  wrote:

> On phone but patch puppet-mediawiki and enable captcha for all pages. We
> only did edit and create
> On Feb 26, 2016 10:38 AM, Marton Kiss  wrote:
>
> I see a ton of incoming post requests:
>
> POST
> /w/index.php?title=Special%3ARunJobs=jobs=1=1456508270=571cfb216f944b15d2eee1c0253d08b77003328e
>
> M.
>
> On Fri, Feb 26, 2016 at 6:35 PM Marton Kiss  wrote:
>
>> Oh, I can login. So what we need?
>>
>> M.
>>
>> On Fri, Feb 26, 2016 at 6:33 PM JP Maxwell  wrote:
>>
>>> I think what Jimmy is referring to is what I was suggesting by removing
>>> the extensions / making the question impossible to answer.  Basically a
>>> series of rapid fire changes while tailing the logs and seeing what stops
>>> the spam.  Once you know what worked then you can submit as an official
>>> patch.  But being able to quickly try these things on a server actually
>>> under attack is the fastest path toward identifying the fix.
>>>
>>> *J.P. Maxwell* | tipit.net | fibercove.com 
>>>
>>> On Fri, Feb 26, 2016 at 11:25 AM, Paul Belanger 
>>> wrote:
>>>
>>> On Fri, Feb 26, 2016 at 11:08:18AM -0600, Jimmy McArthur wrote:
>>> > Given the state of the wiki a the moment, I think taking the quickest
>>> path
>>> > to get it fixed would be prudent. Is there a way we can get JP root
>>> access
>>> > to this server, even temporarily? We get 25% of our website traffic (2
>>> > million visitors) to the wiki. I realize we're all after the same
>>> thing, but
>>> > spammers are not going to hit the dev environment, so there's really
>>> no way
>>> > to tell if teh problem is fixed without actually working directly on
>>> the
>>> > production machine. This should be a 30 minute fix.
>>> >
>>> I am still unclear what the 30min fix is. If really 30mins, then it
>>> shouldn't be
>>> hard to get the fix into our workflow. Could somebody please elaborate.
>>>
>>> If we are talking about deploying new versions of php or mediawiki
>>> manually, I
>>> not be in-favor of this. To me, while the attack sucks, we should be
>>> working on
>>> 2 fronts. Getting the help needed to mitigate the attack, then adding the
>>> changes into -infra workflow in parallel.
>>>
>>> > I realize there is a lot of risk in giving ssh access to infra
>>> machines, but
>>> > I think it's worth taking a look at either putting this machine in a
>>> place
>>> > where a different level of admin could access it without giving away
>>> the
>>> > keys to the entire OpenStack infrastructure or figuring out a way to
>>> set up
>>> > credentials with varying levels of access.
>>> >
>>> As a note, all the work I've been doing to help with the attack hasn't
>>> require
>>> SSH access for me to wiki.o.o. I did need infra-root help to expose our
>>> configuration safely. I'd rather take some time to see what the fixes
>>> are,
>>> having infra-root apply changes, then move them into puppet.
>>>
>>> It also has been discussed to simply disable write access to the wiki if
>>> we
>>> really want spamming to stop, obviously that will affect normal usage.
>>>
>>> > Jimmy
>>> >
>>> > Paul Belanger wrote:
>>> > >On Fri, Feb 26, 2016 at 10:12:12AM -0600, JP Maxwell wrote:
>>> > >>But if you wanted to upgrade everything, remove the mobile view
>>> extension,
>>> > >>test in a dev/staging environment then deploy to production fingers
>>> > >>crossed, I think that would be a valid approach as well.
>>> > >>
>>> > >Current review up[1]. I'll launch a node tonight / tomorrow locally
>>> to see how
>>> > >puppet reacts. I suspect there will be some issues.
>>> > >
>>> > >If infra-roots are fine with this approach, we can use that box to
>>> test against.
>>> > >
>>> > >[1] https://review.openstack.org/#/c/285405/
>>> > >
>>> > >>J.P. Maxwell | tipit.net | fibercove.com
>>> > >>On Feb 26, 2016 10:08 AM, "JP Maxwell" wrote:
>>> > >>
>>> > >>>Plus one except in this case it is much easier to know if our
>>> efforts are
>>> > >>>working on production because the spam either stops or not.
>>> > >>>
>>> > >>>J.P. Maxwell | tipit.net | fibercove.com
>>> > >>>On Feb 26, 2016 9:48 AM, "Paul Belanger"
>>> wrote:
>>> > >>>
>>> > On Fri, Feb 26, 2016 at 09:18:00AM -0600, JP Maxwell wrote:
>>> > >I really think you might consider the option that there is a
>>> > vulnerability
>>> > >in one of the extensions. If that is the case black listing IPs
>>> will be
>>> > an
>>> > >ongoing wild goose chase.
>>> > >
>>> > >I think this would be easily proven or disproven by making the
>>> questy
>>> > >question impossible and see if the spam continues.
>>> > >
>>> > We'll have to let an infra-root make that call. Since nobody would
>>> be

Re: [OpenStack-Infra] Wiki.o.o sustaining spam attack

[Marton Kiss]

On the wiki instance, my ssh access is working now. What I see in the logs
are the continuous POST requests.

M.

On Fri, Feb 26, 2016 at 6:42 PM JP Maxwell  wrote:

> Marton
>
> Where are you seeing the logs?
>
> Paul
>
> The point is that to comment out a line in VI and watch the logs in
> another window takes about a minute or two.  To submit a patch, get
> approval, push, ask someone to share the logs takes a lot longer and relies
> on other people.
>
> I understand the need for openness and I appreciate the process and the
> automation.  I think Jimmy’s point was if we want a quick fix….
>
> *J.P. Maxwell* | tipit.net | fibercove.com 
>
> On Fri, Feb 26, 2016 at 11:38 AM, Marton Kiss 
> wrote:
>
> I see a ton of incoming post requests:
>
> POST
> /w/index.php?title=Special%3ARunJobs=jobs=1=1456508270=571cfb216f944b15d2eee1c0253d08b77003328e
>
> M.
>
> On Fri, Feb 26, 2016 at 6:35 PM Marton Kiss  wrote:
>
>> Oh, I can login. So what we need?
>>
>> M.
>>
>> On Fri, Feb 26, 2016 at 6:33 PM JP Maxwell  wrote:
>>
>>> I think what Jimmy is referring to is what I was suggesting by removing
>>> the extensions / making the question impossible to answer. Basically a
>>> series of rapid fire changes while tailing the logs and seeing what stops
>>> the spam. Once you know what worked then you can submit as an official
>>> patch. But being able to quickly try these things on a server actually
>>> under attack is the fastest path toward identifying the fix.
>>>
>>> *J.P. Maxwell* | tipit.net | fibercove.com 
>>>
>>> On Fri, Feb 26, 2016 at 11:25 AM, Paul Belanger 
>>> wrote:
>>>
>>> On Fri, Feb 26, 2016 at 11:08:18AM -0600, Jimmy McArthur wrote:
>>> > Given the state of the wiki a the moment, I think taking the quickest
>>> path
>>> > to get it fixed would be prudent. Is there a way we can get JP root
>>> access
>>> > to this server, even temporarily? We get 25% of our website traffic (2
>>> > million visitors) to the wiki. I realize we're all after the same
>>> thing, but
>>> > spammers are not going to hit the dev environment, so there's really
>>> no way
>>> > to tell if teh problem is fixed without actually working directly on
>>> the
>>> > production machine. This should be a 30 minute fix.
>>> >
>>> I am still unclear what the 30min fix is. If really 30mins, then it
>>> shouldn't be
>>> hard to get the fix into our workflow. Could somebody please elaborate.
>>>
>>> If we are talking about deploying new versions of php or mediawiki
>>> manually, I
>>> not be in-favor of this. To me, while the attack sucks, we should be
>>> working on
>>> 2 fronts. Getting the help needed to mitigate the attack, then adding the
>>> changes into -infra workflow in parallel.
>>>
>>> > I realize there is a lot of risk in giving ssh access to infra
>>> machines, but
>>> > I think it's worth taking a look at either putting this machine in a
>>> place
>>> > where a different level of admin could access it without giving away
>>> the
>>> > keys to the entire OpenStack infrastructure or figuring out a way to
>>> set up
>>> > credentials with varying levels of access.
>>> >
>>> As a note, all the work I've been doing to help with the attack hasn't
>>> require
>>> SSH access for me to wiki.o.o. I did need infra-root help to expose our
>>> configuration safely. I'd rather take some time to see what the fixes
>>> are,
>>> having infra-root apply changes, then move them into puppet.
>>>
>>> It also has been discussed to simply disable write access to the wiki if
>>> we
>>> really want spamming to stop, obviously that will affect normal usage.
>>>
>>> > Jimmy
>>> >
>>> > Paul Belanger wrote:
>>> > >On Fri, Feb 26, 2016 at 10:12:12AM -0600, JP Maxwell wrote:
>>> > >>But if you wanted to upgrade everything, remove the mobile view
>>> extension,
>>> > >>test in a dev/staging environment then deploy to production fingers
>>> > >>crossed, I think that would be a valid approach as well.
>>> > >>
>>> > >Current review up[1]. I'll launch a node tonight / tomorrow locally
>>> to see how
>>> > >puppet reacts. I suspect there will be some issues.
>>> > >
>>> > >If infra-roots are fine with this approach, we can use that box to
>>> test against.
>>> > >
>>> > >[1] https://review.openstack.org/#/c/285405/
>>> > >
>>> > >>J.P. Maxwell | tipit.net | fibercove.com
>>> > >>On Feb 26, 2016 10:08 AM, "JP Maxwell" wrote:
>>> > >>
>>> > >>>Plus one except in this case it is much easier to know if our
>>> efforts are
>>> > >>>working on production because the spam either stops or not.
>>> > >>>
>>> > >>>J.P. Maxwell | tipit.net | fibercove.com
>>> > >>>On Feb 26, 2016 9:48 AM, "Paul Belanger"
>>> wrote:
>>> > >>>
>>> > On Fri, Feb 26, 2016 at 09:18:00AM -0600, JP Maxwell wrote:
>>> > >I really think you might consider the option that there is a
>>> > 

Re: [OpenStack-Infra] Wiki.o.o sustaining spam attack

[Marton Kiss]

I see a ton of incoming post requests:

POST
/w/index.php?title=Special%3ARunJobs=jobs=1=1456508270=571cfb216f944b15d2eee1c0253d08b77003328e

M.

On Fri, Feb 26, 2016 at 6:35 PM Marton Kiss  wrote:

> Oh, I can login. So what we need?
>
> M.
>
> On Fri, Feb 26, 2016 at 6:33 PM JP Maxwell  wrote:
>
>> I think what Jimmy is referring to is what I was suggesting by removing
>> the extensions / making the question impossible to answer.  Basically a
>> series of rapid fire changes while tailing the logs and seeing what stops
>> the spam.  Once you know what worked then you can submit as an official
>> patch.  But being able to quickly try these things on a server actually
>> under attack is the fastest path toward identifying the fix.
>>
>> *J.P. Maxwell* | tipit.net | fibercove.com 
>>
>> On Fri, Feb 26, 2016 at 11:25 AM, Paul Belanger 
>> wrote:
>>
>> On Fri, Feb 26, 2016 at 11:08:18AM -0600, Jimmy McArthur wrote:
>> > Given the state of the wiki a the moment, I think taking the quickest
>> path
>> > to get it fixed would be prudent. Is there a way we can get JP root
>> access
>> > to this server, even temporarily? We get 25% of our website traffic (2
>> > million visitors) to the wiki. I realize we're all after the same
>> thing, but
>> > spammers are not going to hit the dev environment, so there's really no
>> way
>> > to tell if teh problem is fixed without actually working directly on the
>> > production machine. This should be a 30 minute fix.
>> >
>> I am still unclear what the 30min fix is. If really 30mins, then it
>> shouldn't be
>> hard to get the fix into our workflow. Could somebody please elaborate.
>>
>> If we are talking about deploying new versions of php or mediawiki
>> manually, I
>> not be in-favor of this. To me, while the attack sucks, we should be
>> working on
>> 2 fronts. Getting the help needed to mitigate the attack, then adding the
>> changes into -infra workflow in parallel.
>>
>> > I realize there is a lot of risk in giving ssh access to infra
>> machines, but
>> > I think it's worth taking a look at either putting this machine in a
>> place
>> > where a different level of admin could access it without giving away the
>> > keys to the entire OpenStack infrastructure or figuring out a way to
>> set up
>> > credentials with varying levels of access.
>> >
>> As a note, all the work I've been doing to help with the attack hasn't
>> require
>> SSH access for me to wiki.o.o. I did need infra-root help to expose our
>> configuration safely. I'd rather take some time to see what the fixes are,
>> having infra-root apply changes, then move them into puppet.
>>
>> It also has been discussed to simply disable write access to the wiki if
>> we
>> really want spamming to stop, obviously that will affect normal usage.
>>
>> > Jimmy
>> >
>> > Paul Belanger wrote:
>> > >On Fri, Feb 26, 2016 at 10:12:12AM -0600, JP Maxwell wrote:
>> > >>But if you wanted to upgrade everything, remove the mobile view
>> extension,
>> > >>test in a dev/staging environment then deploy to production fingers
>> > >>crossed, I think that would be a valid approach as well.
>> > >>
>> > >Current review up[1]. I'll launch a node tonight / tomorrow locally to
>> see how
>> > >puppet reacts. I suspect there will be some issues.
>> > >
>> > >If infra-roots are fine with this approach, we can use that box to
>> test against.
>> > >
>> > >[1] https://review.openstack.org/#/c/285405/
>> > >
>> > >>J.P. Maxwell | tipit.net | fibercove.com
>> > >>On Feb 26, 2016 10:08 AM, "JP Maxwell" wrote:
>> > >>
>> > >>>Plus one except in this case it is much easier to know if our
>> efforts are
>> > >>>working on production because the spam either stops or not.
>> > >>>
>> > >>>J.P. Maxwell | tipit.net | fibercove.com
>> > >>>On Feb 26, 2016 9:48 AM, "Paul Belanger"
>> wrote:
>> > >>>
>> > On Fri, Feb 26, 2016 at 09:18:00AM -0600, JP Maxwell wrote:
>> > >I really think you might consider the option that there is a
>> > vulnerability
>> > >in one of the extensions. If that is the case black listing IPs
>> will be
>> > an
>> > >ongoing wild goose chase.
>> > >
>> > >I think this would be easily proven or disproven by making the
>> questy
>> > >question impossible and see if the spam continues.
>> > >
>> > We'll have to let an infra-root make that call. Since nobody would
>> be
>> > able to
>> > use the wiki. Honestly, I'd rather spend the time standing up a
>> mirror dev
>> > instance for us to work on, rather then production.
>> > 
>> > >J.P. Maxwell | tipit.net | fibercove.com
>> > >On Feb 26, 2016 9:12 AM, "Paul Belanger"
>> wrote:
>> > >
>> > >>On Thu, Feb 25, 2016 at 08:10:34PM -0800, Elizabeth K. Joseph
>> wrote:
>> > >>>On Thu, Feb 25, 2016 at 6:35 AM, Jeremy Stanley<
>> fu...@yuggoth.org>
>> > >>wrote:
>> 

Re: [OpenStack-Infra] Wiki.o.o sustaining spam attack

[Paul Belanger]

On Fri, Feb 26, 2016 at 11:29:31AM -0600, JP Maxwell wrote:
> I think what Jimmy is referring to is what I was suggesting by removing the
> extensions / making the question impossible to answer. Basically a series of
> rapid fire changes while tailing the logs and seeing what stops the spam.
> Once
> you know what worked then you can submit as an official patch. But being
> able to
> quickly try these things on a server actually under attack is the fastest
> path
> toward identifying the fix.
> 
Right, and I don't have an issue with that approach.  Based on the work we did
yesterday, anybody can do that via our workflow. Please submit a patch to
puppet-mediawiki[1] and ping an infra-root in #openstack-infra IRC.

We can then have somebody look at the logs.  I think it is more about scheduling
the task since more infra-root as travling back from the mid-cycle last night
and today.

Last email from me, just on a plane.  Will follow up when I land.

[1] https://git.openstack.org/cgit/openstack-infra/puppet-mediawiki


> J.P. Maxwell | tipit.net [http://tipit.net] | fibercove.com
> [http://www.fibercove.com]
> On Fri, Feb 26, 2016 at 11:25 AM, Paul Belanger 
> wrote:
> On Fri, Feb 26, 2016 at 11:08:18AM -0600, Jimmy McArthur wrote:
> > Given the state of the wiki a the moment, I think taking the quickest path
> > to get it fixed would be prudent. Is there a way we can get JP root access
> > to this server, even temporarily? We get 25% of our website traffic (2
> > million visitors) to the wiki. I realize we're all after the same thing,
> but
> > spammers are not going to hit the dev environment, so there's really no
> way
> > to tell if teh problem is fixed without actually working directly on the
> > production machine. This should be a 30 minute fix.
> >
> I am still unclear what the 30min fix is. If really 30mins, then it
> shouldn't be
> hard to get the fix into our workflow. Could somebody please elaborate.
> 
> If we are talking about deploying new versions of php or mediawiki manually,
> I
> not be in-favor of this. To me, while the attack sucks, we should be working
> on
> 2 fronts. Getting the help needed to mitigate the attack, then adding the
> changes into -infra workflow in parallel.
> 
> > I realize there is a lot of risk in giving ssh access to infra machines,
> but
> > I think it's worth taking a look at either putting this machine in a place
> > where a different level of admin could access it without giving away the
> > keys to the entire OpenStack infrastructure or figuring out a way to set
> up
> > credentials with varying levels of access.
> >
> As a note, all the work I've been doing to help with the attack hasn't
> require
> SSH access for me to wiki.o.o. I did need infra-root help to expose our
> configuration safely. I'd rather take some time to see what the fixes are,
> having infra-root apply changes, then move them into puppet.
> 
> It also has been discussed to simply disable write access to the wiki if we
> really want spamming to stop, obviously that will affect normal usage.
> 
> > Jimmy
> >
> > Paul Belanger wrote:
> > >On Fri, Feb 26, 2016 at 10:12:12AM -0600, JP Maxwell wrote:
> > >>But if you wanted to upgrade everything, remove the mobile view
> extension,
> > >>test in a dev/staging environment then deploy to production fingers
> > >>crossed, I think that would be a valid approach as well.
> > >>
> > >Current review up[1]. I'll launch a node tonight / tomorrow locally to
> see
> how
> > >puppet reacts. I suspect there will be some issues.
> > >
> > >If infra-roots are fine with this approach, we can use that box to test
> against.
> > >
> > >[1] https://review.openstack.org/#/c/285405/
> > >
> > >>J.P. Maxwell | tipit.net | fibercove.com
> > >>On Feb 26, 2016 10:08 AM, "JP Maxwell" wrote:
> > >>
> > >>>Plus one except in this case it is much easier to know if our efforts
> are
> > >>>working on production because the spam either stops or not.
> > >>>
> > >>>J.P. Maxwell | tipit.net | fibercove.com
> > >>>On Feb 26, 2016 9:48 AM, "Paul Belanger" wrote:
> > >>>
> > On Fri, Feb 26, 2016 at 09:18:00AM -0600, JP Maxwell wrote:
> > >I really think you might consider the option that there is a
> > vulnerability
> > >in one of the extensions. If that is the case black listing IPs will
> be
> > an
> > >ongoing wild goose chase.
> > >
> > >I think this would be easily proven or disproven by making the questy
> > >question impossible and see if the spam continues.
> > >
> > We'll have to let an infra-root make that call. Since nobody would be
> > able to
> > use the wiki. Honestly, I'd rather spend the time standing up a mirror
> dev
> > instance for us to work on, rather then production.
> > 
> > >J.P. Maxwell | tipit.net | fibercove.com
> > >On Feb 26, 2016 9:12 AM, "Paul Belanger"
> wrote:
> > >
> > >>On Thu, Feb 25, 2016 

Re: [OpenStack-Infra] Wiki.o.o sustaining spam attack

[Marton Kiss]

Oh, I can login. So what we need?

M.

On Fri, Feb 26, 2016 at 6:33 PM JP Maxwell  wrote:

> I think what Jimmy is referring to is what I was suggesting by removing
> the extensions / making the question impossible to answer.  Basically a
> series of rapid fire changes while tailing the logs and seeing what stops
> the spam.  Once you know what worked then you can submit as an official
> patch.  But being able to quickly try these things on a server actually
> under attack is the fastest path toward identifying the fix.
>
> *J.P. Maxwell* | tipit.net | fibercove.com 
>
> On Fri, Feb 26, 2016 at 11:25 AM, Paul Belanger 
> wrote:
>
> On Fri, Feb 26, 2016 at 11:08:18AM -0600, Jimmy McArthur wrote:
> > Given the state of the wiki a the moment, I think taking the quickest
> path
> > to get it fixed would be prudent. Is there a way we can get JP root
> access
> > to this server, even temporarily? We get 25% of our website traffic (2
> > million visitors) to the wiki. I realize we're all after the same thing,
> but
> > spammers are not going to hit the dev environment, so there's really no
> way
> > to tell if teh problem is fixed without actually working directly on the
> > production machine. This should be a 30 minute fix.
> >
> I am still unclear what the 30min fix is. If really 30mins, then it
> shouldn't be
> hard to get the fix into our workflow. Could somebody please elaborate.
>
> If we are talking about deploying new versions of php or mediawiki
> manually, I
> not be in-favor of this. To me, while the attack sucks, we should be
> working on
> 2 fronts. Getting the help needed to mitigate the attack, then adding the
> changes into -infra workflow in parallel.
>
> > I realize there is a lot of risk in giving ssh access to infra machines,
> but
> > I think it's worth taking a look at either putting this machine in a
> place
> > where a different level of admin could access it without giving away the
> > keys to the entire OpenStack infrastructure or figuring out a way to set
> up
> > credentials with varying levels of access.
> >
> As a note, all the work I've been doing to help with the attack hasn't
> require
> SSH access for me to wiki.o.o. I did need infra-root help to expose our
> configuration safely. I'd rather take some time to see what the fixes are,
> having infra-root apply changes, then move them into puppet.
>
> It also has been discussed to simply disable write access to the wiki if we
> really want spamming to stop, obviously that will affect normal usage.
>
> > Jimmy
> >
> > Paul Belanger wrote:
> > >On Fri, Feb 26, 2016 at 10:12:12AM -0600, JP Maxwell wrote:
> > >>But if you wanted to upgrade everything, remove the mobile view
> extension,
> > >>test in a dev/staging environment then deploy to production fingers
> > >>crossed, I think that would be a valid approach as well.
> > >>
> > >Current review up[1]. I'll launch a node tonight / tomorrow locally to
> see how
> > >puppet reacts. I suspect there will be some issues.
> > >
> > >If infra-roots are fine with this approach, we can use that box to test
> against.
> > >
> > >[1] https://review.openstack.org/#/c/285405/
> > >
> > >>J.P. Maxwell | tipit.net | fibercove.com
> > >>On Feb 26, 2016 10:08 AM, "JP Maxwell" wrote:
> > >>
> > >>>Plus one except in this case it is much easier to know if our efforts
> are
> > >>>working on production because the spam either stops or not.
> > >>>
> > >>>J.P. Maxwell | tipit.net | fibercove.com
> > >>>On Feb 26, 2016 9:48 AM, "Paul Belanger"
> wrote:
> > >>>
> > On Fri, Feb 26, 2016 at 09:18:00AM -0600, JP Maxwell wrote:
> > >I really think you might consider the option that there is a
> > vulnerability
> > >in one of the extensions. If that is the case black listing IPs
> will be
> > an
> > >ongoing wild goose chase.
> > >
> > >I think this would be easily proven or disproven by making the
> questy
> > >question impossible and see if the spam continues.
> > >
> > We'll have to let an infra-root make that call. Since nobody would be
> > able to
> > use the wiki. Honestly, I'd rather spend the time standing up a
> mirror dev
> > instance for us to work on, rather then production.
> > 
> > >J.P. Maxwell | tipit.net | fibercove.com
> > >On Feb 26, 2016 9:12 AM, "Paul Belanger"
> wrote:
> > >
> > >>On Thu, Feb 25, 2016 at 08:10:34PM -0800, Elizabeth K. Joseph
> wrote:
> > >>>On Thu, Feb 25, 2016 at 6:35 AM, Jeremy Stanley >
> > >>wrote:
> > On 2016-02-25 02:46:13 -0600 (-0600), JP Maxwell wrote:
> > >Please be aware that you can now create accounts under the
> mobile
> > >view in the wiki native user table. I just created an account
> for
> > >JpMaxMan. Not sure if this matters but wanted to make sure you
> > >were aware.
> > Oh, 

Re: [OpenStack-Infra] Wiki.o.o sustaining spam attack

[JP Maxwell]

I think what Jimmy is referring to is what I was suggesting by removing the
extensions / making the question impossible to answer. Basically a series 
of
rapid fire changes while tailing the logs and seeing what stops the spam. 
Once
you know what worked then you can submit as an official patch. But being 
able to
quickly try these things on a server actually under attack is the fastest 
path

toward identifying the fix.

J.P. Maxwell | tipit.net [http://tipit.net] | fibercove.com 
[http://www.fibercove.com]
On Fri, Feb 26, 2016 at 11:25 AM, Paul Belanger  
wrote:

On Fri, Feb 26, 2016 at 11:08:18AM -0600, Jimmy McArthur wrote:
> Given the state of the wiki a the moment, I think taking the quickest 
path
> to get it fixed would be prudent. Is there a way we can get JP root 
access

> to this server, even temporarily? We get 25% of our website traffic (2
> million visitors) to the wiki. I realize we're all after the same thing, 
but
> spammers are not going to hit the dev environment, so there's really no 
way

> to tell if teh problem is fixed without actually working directly on the
> production machine. This should be a 30 minute fix.
>
I am still unclear what the 30min fix is. If really 30mins, then it 
shouldn't be

hard to get the fix into our workflow. Could somebody please elaborate.

If we are talking about deploying new versions of php or mediawiki 
manually, I
not be in-favor of this. To me, while the attack sucks, we should be 
working on

2 fronts. Getting the help needed to mitigate the attack, then adding the
changes into -infra workflow in parallel.

> I realize there is a lot of risk in giving ssh access to infra machines, 
but
> I think it's worth taking a look at either putting this machine in a 
place

> where a different level of admin could access it without giving away the
> keys to the entire OpenStack infrastructure or figuring out a way to set 
up

> credentials with varying levels of access.
>
As a note, all the work I've been doing to help with the attack hasn't 
require

SSH access for me to wiki.o.o. I did need infra-root help to expose our
configuration safely. I'd rather take some time to see what the fixes are,
having infra-root apply changes, then move them into puppet.

It also has been discussed to simply disable write access to the wiki if we
really want spamming to stop, obviously that will affect normal usage.

> Jimmy
>
> Paul Belanger wrote:
> >On Fri, Feb 26, 2016 at 10:12:12AM -0600, JP Maxwell wrote:
> >>But if you wanted to upgrade everything, remove the mobile view 
extension,

> >>test in a dev/staging environment then deploy to production fingers
> >>crossed, I think that would be a valid approach as well.
> >>
> >Current review up[1]. I'll launch a node tonight / tomorrow locally to 
see

how
> >puppet reacts. I suspect there will be some issues.
> >
> >If infra-roots are fine with this approach, we can use that box to test
against.
> >
> >[1] https://review.openstack.org/#/c/285405/
> >
> >>J.P. Maxwell | tipit.net | fibercove.com
> >>On Feb 26, 2016 10:08 AM, "JP Maxwell" wrote:
> >>
> >>>Plus one except in this case it is much easier to know if our efforts 
are

> >>>working on production because the spam either stops or not.
> >>>
> >>>J.P. Maxwell | tipit.net | fibercove.com
> >>>On Feb 26, 2016 9:48 AM, "Paul Belanger" 
wrote:

> >>>
> On Fri, Feb 26, 2016 at 09:18:00AM -0600, JP Maxwell wrote:
> >I really think you might consider the option that there is a
> vulnerability
> >in one of the extensions. If that is the case black listing IPs 
will be

> an
> >ongoing wild goose chase.
> >
> >I think this would be easily proven or disproven by making the 
questy

> >question impossible and see if the spam continues.
> >
> We'll have to let an infra-root make that call. Since nobody would 
be

> able to
> use the wiki. Honestly, I'd rather spend the time standing up a 
mirror dev

> instance for us to work on, rather then production.
> 
> >J.P. Maxwell | tipit.net | fibercove.com
> >On Feb 26, 2016 9:12 AM, "Paul Belanger" 
wrote:

> >
> >>On Thu, Feb 25, 2016 at 08:10:34PM -0800, Elizabeth K. Joseph 
wrote:
> >>>On Thu, Feb 25, 2016 at 6:35 AM, Jeremy 
Stanley

> >>wrote:
> On 2016-02-25 02:46:13 -0600 (-0600), JP Maxwell wrote:
> >Please be aware that you can now create accounts under the 
mobile
> >view in the wiki native user table. I just created an account 
for

> >JpMaxMan. Not sure if this matters but wanted to make sure you
> >were aware.
> Oh, yes I think having a random garbage question/answer was in
> fact
> previously preventing account creation under the mobile view. We
> probably need a way to disable mobile view account creation as 
it

> bypasses OpenID authentication entirely.
> 

Re: [OpenStack-Infra] Wiki.o.o sustaining spam attack

[Paul Belanger]

On Fri, Feb 26, 2016 at 11:08:18AM -0600, Jimmy McArthur wrote:
> Given the state of the wiki a the moment, I think taking the quickest path
> to get it fixed would be prudent. Is there a way we can get JP root access
> to this server, even temporarily? We get 25% of our website traffic (2
> million visitors) to the wiki. I realize we're all after the same thing, but
> spammers are not going to hit the dev environment, so there's really no way
> to tell if teh problem is fixed without actually working directly on the
> production machine. This should be a 30 minute fix.
> 
I am still unclear what the 30min fix is. If really 30mins, then it shouldn't be
hard to get the fix into our workflow. Could somebody please elaborate.

If we are talking about deploying new versions of php or mediawiki manually, I
not be in-favor of this.  To me, while the attack sucks, we should be working on
2 fronts.  Getting the help needed to mitigate the attack, then adding the
changes into -infra workflow in parallel.

> I realize there is a lot of risk in giving ssh access to infra machines, but
> I think it's worth taking a look at either putting this machine in a place
> where a different level of admin could access it without giving away the
> keys to the entire OpenStack infrastructure or figuring out a way to set up
> credentials with varying levels of access.
> 
As a note, all the work I've been doing to help with the attack hasn't require
SSH access for me to wiki.o.o.  I did need infra-root help to expose our
configuration safely.  I'd rather take some time to see what the fixes are,
having infra-root apply changes, then move them into puppet.

It also has been discussed to simply disable write access to the wiki if we
really want spamming to stop, obviously that will affect normal usage.

> Jimmy
> 
> Paul Belanger wrote:
> >On Fri, Feb 26, 2016 at 10:12:12AM -0600, JP Maxwell wrote:
> >>But if you wanted to upgrade everything, remove the mobile view extension,
> >>test in a dev/staging environment then deploy to production fingers
> >>crossed, I think that would be a valid approach as well.
> >>
> >Current review up[1]. I'll launch a node tonight / tomorrow locally to see 
> >how
> >puppet reacts.  I suspect there will be some issues.
> >
> >If infra-roots are fine with this approach, we can use that box to test 
> >against.
> >
> >[1] https://review.openstack.org/#/c/285405/
> >
> >>J.P. Maxwell | tipit.net | fibercove.com
> >>On Feb 26, 2016 10:08 AM, "JP Maxwell"  wrote:
> >>
> >>>Plus one except in this case it is much easier to know if our efforts are
> >>>working on production because the spam either stops or not.
> >>>
> >>>J.P. Maxwell | tipit.net | fibercove.com
> >>>On Feb 26, 2016 9:48 AM, "Paul Belanger"  wrote:
> >>>
> On Fri, Feb 26, 2016 at 09:18:00AM -0600, JP Maxwell wrote:
> >I really think you might consider the option that there is a
> vulnerability
> >in one of the extensions. If that is the case black listing IPs will be
> an
> >ongoing wild goose chase.
> >
> >I think this would be easily proven or disproven by making the questy
> >question impossible and see if the spam continues.
> >
> We'll have to let an infra-root make that call. Since nobody would be
> able to
> use the wiki. Honestly, I'd rather spend the time standing up a mirror dev
> instance for us to work on, rather then production.
> 
> >J.P. Maxwell | tipit.net | fibercove.com
> >On Feb 26, 2016 9:12 AM, "Paul Belanger"  wrote:
> >
> >>On Thu, Feb 25, 2016 at 08:10:34PM -0800, Elizabeth K. Joseph wrote:
> >>>On Thu, Feb 25, 2016 at 6:35 AM, Jeremy Stanley
> >>wrote:
> On 2016-02-25 02:46:13 -0600 (-0600), JP Maxwell wrote:
> >Please be aware that you can now create accounts under the mobile
> >view in the wiki native user table. I just created an account for
> >JpMaxMan.  Not sure if this matters but wanted to make sure you
> >were aware.
> Oh, yes I think having a random garbage question/answer was in
> fact
> previously preventing account creation under the mobile view. We
> probably need a way to disable mobile view account creation as it
> bypasses OpenID authentication entirely.
> >>>So that's what it was doing! We'll have to tackle the mobile view
> issue.
> >>>Otherwise, quick update here:
> >>>
> >>>The captcha didn't appear to help stem the spam tide. We'll want to
> >>>explore and start implementing some of the other solutions.
> >>>
> >>>I did some database poking around today and it does seem like all
> the
> >>>users do have launchpad accounts and email addresses.
> >>>
> >>So, I have a few hours before jumping on my plane and checked into
> this.
> >>We are
> >>using QuestyCaptcha which according to docs, should almost be

Re: [OpenStack-Infra] Wiki.o.o sustaining spam attack

[Jimmy McArthur]

Given the state of the wiki a the moment, I think taking the quickest 
path to get it fixed would be prudent. Is there a way we can get JP root 
access to this server, even temporarily? We get 25% of our website 
traffic (2 million visitors) to the wiki. I realize we're all after the 
same thing, but spammers are not going to hit the dev environment, so 
there's really no way to tell if teh problem is fixed without actually 
working directly on the production machine. This should be a 30 minute fix.


I realize there is a lot of risk in giving ssh access to infra machines, 
but I think it's worth taking a look at either putting this machine in a 
place where a different level of admin could access it without giving 
away the keys to the entire OpenStack infrastructure or figuring out a 
way to set up credentials with varying levels of access.


Jimmy

Paul Belanger wrote:

On Fri, Feb 26, 2016 at 10:12:12AM -0600, JP Maxwell wrote:

But if you wanted to upgrade everything, remove the mobile view extension,
test in a dev/staging environment then deploy to production fingers
crossed, I think that would be a valid approach as well.


Current review up[1]. I'll launch a node tonight / tomorrow locally to see how
puppet reacts.  I suspect there will be some issues.

If infra-roots are fine with this approach, we can use that box to test against.

[1] https://review.openstack.org/#/c/285405/


J.P. Maxwell | tipit.net | fibercove.com
On Feb 26, 2016 10:08 AM, "JP Maxwell"  wrote:


Plus one except in this case it is much easier to know if our efforts are
working on production because the spam either stops or not.

J.P. Maxwell | tipit.net | fibercove.com
On Feb 26, 2016 9:48 AM, "Paul Belanger"  wrote:


On Fri, Feb 26, 2016 at 09:18:00AM -0600, JP Maxwell wrote:

I really think you might consider the option that there is a

vulnerability

in one of the extensions. If that is the case black listing IPs will be

an

ongoing wild goose chase.

I think this would be easily proven or disproven by making the questy
question impossible and see if the spam continues.


We'll have to let an infra-root make that call. Since nobody would be
able to
use the wiki. Honestly, I'd rather spend the time standing up a mirror dev
instance for us to work on, rather then production.


J.P. Maxwell | tipit.net | fibercove.com
On Feb 26, 2016 9:12 AM, "Paul Belanger"  wrote:


On Thu, Feb 25, 2016 at 08:10:34PM -0800, Elizabeth K. Joseph wrote:

On Thu, Feb 25, 2016 at 6:35 AM, Jeremy Stanley

wrote:

On 2016-02-25 02:46:13 -0600 (-0600), JP Maxwell wrote:

Please be aware that you can now create accounts under the mobile
view in the wiki native user table. I just created an account for
JpMaxMan.  Not sure if this matters but wanted to make sure you
were aware.

Oh, yes I think having a random garbage question/answer was in

fact

previously preventing account creation under the mobile view. We
probably need a way to disable mobile view account creation as it
bypasses OpenID authentication entirely.

So that's what it was doing! We'll have to tackle the mobile view

issue.

Otherwise, quick update here:

The captcha didn't appear to help stem the spam tide. We'll want to
explore and start implementing some of the other solutions.

I did some database poking around today and it does seem like all

the

users do have launchpad accounts and email addresses.


So, I have a few hours before jumping on my plane and checked into

this.

We are
using QuestyCaptcha which according to docs, should almost be

impossible

for
spammers to by pass in an automated fashion.  So, either our captcha

is too

easy, or we didn't set it up properly.  I don't have SSH on wiki.o.o

so

others
will have to check logs.  I did test new pages and edits, and was

promoted

by
captcha.

As a next step, we might need to add additional apache2 configuration

to

blacklist IPs.  I am reading up on that now.


--
Elizabeth Krumbach Joseph || Lyz || pleia2

___
OpenStack-Infra mailing list
OpenStack-Infra@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra

___
OpenStack-Infra mailing list
OpenStack-Infra@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra



___
OpenStack-Infra mailing list
OpenStack-Infra@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra


___
OpenStack-Infra mailing list
OpenStack-Infra@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra

Re: [OpenStack-Infra] Wiki.o.o sustaining spam attack

[Paul Belanger]

On Fri, Feb 26, 2016 at 10:12:12AM -0600, JP Maxwell wrote:
> But if you wanted to upgrade everything, remove the mobile view extension,
> test in a dev/staging environment then deploy to production fingers
> crossed, I think that would be a valid approach as well.
> 
Current review up[1]. I'll launch a node tonight / tomorrow locally to see how
puppet reacts.  I suspect there will be some issues.

If infra-roots are fine with this approach, we can use that box to test against.

[1] https://review.openstack.org/#/c/285405/

> J.P. Maxwell | tipit.net | fibercove.com
> On Feb 26, 2016 10:08 AM, "JP Maxwell"  wrote:
> 
> > Plus one except in this case it is much easier to know if our efforts are
> > working on production because the spam either stops or not.
> >
> > J.P. Maxwell | tipit.net | fibercove.com
> > On Feb 26, 2016 9:48 AM, "Paul Belanger"  wrote:
> >
> >> On Fri, Feb 26, 2016 at 09:18:00AM -0600, JP Maxwell wrote:
> >> > I really think you might consider the option that there is a
> >> vulnerability
> >> > in one of the extensions. If that is the case black listing IPs will be
> >> an
> >> > ongoing wild goose chase.
> >> >
> >> > I think this would be easily proven or disproven by making the questy
> >> > question impossible and see if the spam continues.
> >> >
> >> We'll have to let an infra-root make that call. Since nobody would be
> >> able to
> >> use the wiki. Honestly, I'd rather spend the time standing up a mirror dev
> >> instance for us to work on, rather then production.
> >>
> >> > J.P. Maxwell | tipit.net | fibercove.com
> >> > On Feb 26, 2016 9:12 AM, "Paul Belanger"  wrote:
> >> >
> >> > > On Thu, Feb 25, 2016 at 08:10:34PM -0800, Elizabeth K. Joseph wrote:
> >> > > > On Thu, Feb 25, 2016 at 6:35 AM, Jeremy Stanley 
> >> > > wrote:
> >> > > > > On 2016-02-25 02:46:13 -0600 (-0600), JP Maxwell wrote:
> >> > > > >> Please be aware that you can now create accounts under the mobile
> >> > > > >> view in the wiki native user table. I just created an account for
> >> > > > >> JpMaxMan.  Not sure if this matters but wanted to make sure you
> >> > > > >> were aware.
> >> > > > >
> >> > > > > Oh, yes I think having a random garbage question/answer was in
> >> fact
> >> > > > > previously preventing account creation under the mobile view. We
> >> > > > > probably need a way to disable mobile view account creation as it
> >> > > > > bypasses OpenID authentication entirely.
> >> > > >
> >> > > > So that's what it was doing! We'll have to tackle the mobile view
> >> issue.
> >> > > >
> >> > > > Otherwise, quick update here:
> >> > > >
> >> > > > The captcha didn't appear to help stem the spam tide. We'll want to
> >> > > > explore and start implementing some of the other solutions.
> >> > > >
> >> > > > I did some database poking around today and it does seem like all
> >> the
> >> > > > users do have launchpad accounts and email addresses.
> >> > > >
> >> > > So, I have a few hours before jumping on my plane and checked into
> >> this.
> >> > > We are
> >> > > using QuestyCaptcha which according to docs, should almost be
> >> impossible
> >> > > for
> >> > > spammers to by pass in an automated fashion.  So, either our captcha
> >> is too
> >> > > easy, or we didn't set it up properly.  I don't have SSH on wiki.o.o
> >> so
> >> > > others
> >> > > will have to check logs.  I did test new pages and edits, and was
> >> promoted
> >> > > by
> >> > > captcha.
> >> > >
> >> > > As a next step, we might need to add additional apache2 configuration
> >> to
> >> > > blacklist IPs.  I am reading up on that now.
> >> > >
> >> > > > --
> >> > > > Elizabeth Krumbach Joseph || Lyz || pleia2
> >> > > >
> >> > > > ___
> >> > > > OpenStack-Infra mailing list
> >> > > > OpenStack-Infra@lists.openstack.org
> >> > > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
> >> > >
> >> > > ___
> >> > > OpenStack-Infra mailing list
> >> > > OpenStack-Infra@lists.openstack.org
> >> > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
> >> > >
> >>
> >

___
OpenStack-Infra mailing list
OpenStack-Infra@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra

Re: [OpenStack-Infra] Wiki.o.o sustaining spam attack

[JP Maxwell]

But if you wanted to upgrade everything, remove the mobile view extension,
test in a dev/staging environment then deploy to production fingers
crossed, I think that would be a valid approach as well.

J.P. Maxwell | tipit.net | fibercove.com
On Feb 26, 2016 10:08 AM, "JP Maxwell"  wrote:

> Plus one except in this case it is much easier to know if our efforts are
> working on production because the spam either stops or not.
>
> J.P. Maxwell | tipit.net | fibercove.com
> On Feb 26, 2016 9:48 AM, "Paul Belanger"  wrote:
>
>> On Fri, Feb 26, 2016 at 09:18:00AM -0600, JP Maxwell wrote:
>> > I really think you might consider the option that there is a
>> vulnerability
>> > in one of the extensions. If that is the case black listing IPs will be
>> an
>> > ongoing wild goose chase.
>> >
>> > I think this would be easily proven or disproven by making the questy
>> > question impossible and see if the spam continues.
>> >
>> We'll have to let an infra-root make that call. Since nobody would be
>> able to
>> use the wiki. Honestly, I'd rather spend the time standing up a mirror dev
>> instance for us to work on, rather then production.
>>
>> > J.P. Maxwell | tipit.net | fibercove.com
>> > On Feb 26, 2016 9:12 AM, "Paul Belanger"  wrote:
>> >
>> > > On Thu, Feb 25, 2016 at 08:10:34PM -0800, Elizabeth K. Joseph wrote:
>> > > > On Thu, Feb 25, 2016 at 6:35 AM, Jeremy Stanley 
>> > > wrote:
>> > > > > On 2016-02-25 02:46:13 -0600 (-0600), JP Maxwell wrote:
>> > > > >> Please be aware that you can now create accounts under the mobile
>> > > > >> view in the wiki native user table. I just created an account for
>> > > > >> JpMaxMan.  Not sure if this matters but wanted to make sure you
>> > > > >> were aware.
>> > > > >
>> > > > > Oh, yes I think having a random garbage question/answer was in
>> fact
>> > > > > previously preventing account creation under the mobile view. We
>> > > > > probably need a way to disable mobile view account creation as it
>> > > > > bypasses OpenID authentication entirely.
>> > > >
>> > > > So that's what it was doing! We'll have to tackle the mobile view
>> issue.
>> > > >
>> > > > Otherwise, quick update here:
>> > > >
>> > > > The captcha didn't appear to help stem the spam tide. We'll want to
>> > > > explore and start implementing some of the other solutions.
>> > > >
>> > > > I did some database poking around today and it does seem like all
>> the
>> > > > users do have launchpad accounts and email addresses.
>> > > >
>> > > So, I have a few hours before jumping on my plane and checked into
>> this.
>> > > We are
>> > > using QuestyCaptcha which according to docs, should almost be
>> impossible
>> > > for
>> > > spammers to by pass in an automated fashion.  So, either our captcha
>> is too
>> > > easy, or we didn't set it up properly.  I don't have SSH on wiki.o.o
>> so
>> > > others
>> > > will have to check logs.  I did test new pages and edits, and was
>> promoted
>> > > by
>> > > captcha.
>> > >
>> > > As a next step, we might need to add additional apache2 configuration
>> to
>> > > blacklist IPs.  I am reading up on that now.
>> > >
>> > > > --
>> > > > Elizabeth Krumbach Joseph || Lyz || pleia2
>> > > >
>> > > > ___
>> > > > OpenStack-Infra mailing list
>> > > > OpenStack-Infra@lists.openstack.org
>> > > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
>> > >
>> > > ___
>> > > OpenStack-Infra mailing list
>> > > OpenStack-Infra@lists.openstack.org
>> > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
>> > >
>>
>
___
OpenStack-Infra mailing list
OpenStack-Infra@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra

Re: [OpenStack-Infra] Wiki.o.o sustaining spam attack

[JP Maxwell]

Plus one except in this case it is much easier to know if our efforts are
working on production because the spam either stops or not.

J.P. Maxwell | tipit.net | fibercove.com
On Feb 26, 2016 9:48 AM, "Paul Belanger"  wrote:

> On Fri, Feb 26, 2016 at 09:18:00AM -0600, JP Maxwell wrote:
> > I really think you might consider the option that there is a
> vulnerability
> > in one of the extensions. If that is the case black listing IPs will be
> an
> > ongoing wild goose chase.
> >
> > I think this would be easily proven or disproven by making the questy
> > question impossible and see if the spam continues.
> >
> We'll have to let an infra-root make that call. Since nobody would be able
> to
> use the wiki. Honestly, I'd rather spend the time standing up a mirror dev
> instance for us to work on, rather then production.
>
> > J.P. Maxwell | tipit.net | fibercove.com
> > On Feb 26, 2016 9:12 AM, "Paul Belanger"  wrote:
> >
> > > On Thu, Feb 25, 2016 at 08:10:34PM -0800, Elizabeth K. Joseph wrote:
> > > > On Thu, Feb 25, 2016 at 6:35 AM, Jeremy Stanley 
> > > wrote:
> > > > > On 2016-02-25 02:46:13 -0600 (-0600), JP Maxwell wrote:
> > > > >> Please be aware that you can now create accounts under the mobile
> > > > >> view in the wiki native user table. I just created an account for
> > > > >> JpMaxMan.  Not sure if this matters but wanted to make sure you
> > > > >> were aware.
> > > > >
> > > > > Oh, yes I think having a random garbage question/answer was in fact
> > > > > previously preventing account creation under the mobile view. We
> > > > > probably need a way to disable mobile view account creation as it
> > > > > bypasses OpenID authentication entirely.
> > > >
> > > > So that's what it was doing! We'll have to tackle the mobile view
> issue.
> > > >
> > > > Otherwise, quick update here:
> > > >
> > > > The captcha didn't appear to help stem the spam tide. We'll want to
> > > > explore and start implementing some of the other solutions.
> > > >
> > > > I did some database poking around today and it does seem like all the
> > > > users do have launchpad accounts and email addresses.
> > > >
> > > So, I have a few hours before jumping on my plane and checked into
> this.
> > > We are
> > > using QuestyCaptcha which according to docs, should almost be
> impossible
> > > for
> > > spammers to by pass in an automated fashion.  So, either our captcha
> is too
> > > easy, or we didn't set it up properly.  I don't have SSH on wiki.o.o so
> > > others
> > > will have to check logs.  I did test new pages and edits, and was
> promoted
> > > by
> > > captcha.
> > >
> > > As a next step, we might need to add additional apache2 configuration
> to
> > > blacklist IPs.  I am reading up on that now.
> > >
> > > > --
> > > > Elizabeth Krumbach Joseph || Lyz || pleia2
> > > >
> > > > ___
> > > > OpenStack-Infra mailing list
> > > > OpenStack-Infra@lists.openstack.org
> > > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
> > >
> > > ___
> > > OpenStack-Infra mailing list
> > > OpenStack-Infra@lists.openstack.org
> > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
> > >
>
___
OpenStack-Infra mailing list
OpenStack-Infra@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra

Re: [OpenStack-Infra] Wiki.o.o sustaining spam attack

[Paul Belanger]

On Fri, Feb 26, 2016 at 09:18:00AM -0600, JP Maxwell wrote:
> I really think you might consider the option that there is a vulnerability
> in one of the extensions. If that is the case black listing IPs will be an
> ongoing wild goose chase.
> 
> I think this would be easily proven or disproven by making the questy
> question impossible and see if the spam continues.
> 
We'll have to let an infra-root make that call. Since nobody would be able to
use the wiki. Honestly, I'd rather spend the time standing up a mirror dev
instance for us to work on, rather then production.

> J.P. Maxwell | tipit.net | fibercove.com
> On Feb 26, 2016 9:12 AM, "Paul Belanger"  wrote:
> 
> > On Thu, Feb 25, 2016 at 08:10:34PM -0800, Elizabeth K. Joseph wrote:
> > > On Thu, Feb 25, 2016 at 6:35 AM, Jeremy Stanley 
> > wrote:
> > > > On 2016-02-25 02:46:13 -0600 (-0600), JP Maxwell wrote:
> > > >> Please be aware that you can now create accounts under the mobile
> > > >> view in the wiki native user table. I just created an account for
> > > >> JpMaxMan.  Not sure if this matters but wanted to make sure you
> > > >> were aware.
> > > >
> > > > Oh, yes I think having a random garbage question/answer was in fact
> > > > previously preventing account creation under the mobile view. We
> > > > probably need a way to disable mobile view account creation as it
> > > > bypasses OpenID authentication entirely.
> > >
> > > So that's what it was doing! We'll have to tackle the mobile view issue.
> > >
> > > Otherwise, quick update here:
> > >
> > > The captcha didn't appear to help stem the spam tide. We'll want to
> > > explore and start implementing some of the other solutions.
> > >
> > > I did some database poking around today and it does seem like all the
> > > users do have launchpad accounts and email addresses.
> > >
> > So, I have a few hours before jumping on my plane and checked into this.
> > We are
> > using QuestyCaptcha which according to docs, should almost be impossible
> > for
> > spammers to by pass in an automated fashion.  So, either our captcha is too
> > easy, or we didn't set it up properly.  I don't have SSH on wiki.o.o so
> > others
> > will have to check logs.  I did test new pages and edits, and was promoted
> > by
> > captcha.
> >
> > As a next step, we might need to add additional apache2 configuration to
> > blacklist IPs.  I am reading up on that now.
> >
> > > --
> > > Elizabeth Krumbach Joseph || Lyz || pleia2
> > >
> > > ___
> > > OpenStack-Infra mailing list
> > > OpenStack-Infra@lists.openstack.org
> > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
> >
> > ___
> > OpenStack-Infra mailing list
> > OpenStack-Infra@lists.openstack.org
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
> >

___
OpenStack-Infra mailing list
OpenStack-Infra@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra

Re: [OpenStack-Infra] Wiki.o.o sustaining spam attack

[JP Maxwell]

I really think you might consider the option that there is a vulnerability
in one of the extensions. If that is the case black listing IPs will be an
ongoing wild goose chase.

I think this would be easily proven or disproven by making the questy
question impossible and see if the spam continues.

J.P. Maxwell | tipit.net | fibercove.com
On Feb 26, 2016 9:12 AM, "Paul Belanger"  wrote:

> On Thu, Feb 25, 2016 at 08:10:34PM -0800, Elizabeth K. Joseph wrote:
> > On Thu, Feb 25, 2016 at 6:35 AM, Jeremy Stanley 
> wrote:
> > > On 2016-02-25 02:46:13 -0600 (-0600), JP Maxwell wrote:
> > >> Please be aware that you can now create accounts under the mobile
> > >> view in the wiki native user table. I just created an account for
> > >> JpMaxMan.  Not sure if this matters but wanted to make sure you
> > >> were aware.
> > >
> > > Oh, yes I think having a random garbage question/answer was in fact
> > > previously preventing account creation under the mobile view. We
> > > probably need a way to disable mobile view account creation as it
> > > bypasses OpenID authentication entirely.
> >
> > So that's what it was doing! We'll have to tackle the mobile view issue.
> >
> > Otherwise, quick update here:
> >
> > The captcha didn't appear to help stem the spam tide. We'll want to
> > explore and start implementing some of the other solutions.
> >
> > I did some database poking around today and it does seem like all the
> > users do have launchpad accounts and email addresses.
> >
> So, I have a few hours before jumping on my plane and checked into this.
> We are
> using QuestyCaptcha which according to docs, should almost be impossible
> for
> spammers to by pass in an automated fashion.  So, either our captcha is too
> easy, or we didn't set it up properly.  I don't have SSH on wiki.o.o so
> others
> will have to check logs.  I did test new pages and edits, and was promoted
> by
> captcha.
>
> As a next step, we might need to add additional apache2 configuration to
> blacklist IPs.  I am reading up on that now.
>
> > --
> > Elizabeth Krumbach Joseph || Lyz || pleia2
> >
> > ___
> > OpenStack-Infra mailing list
> > OpenStack-Infra@lists.openstack.org
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
>
> ___
> OpenStack-Infra mailing list
> OpenStack-Infra@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
>
___
OpenStack-Infra mailing list
OpenStack-Infra@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra

Re: [OpenStack-Infra] Wiki.o.o sustaining spam attack

[Paul Belanger]

On Thu, Feb 25, 2016 at 08:10:34PM -0800, Elizabeth K. Joseph wrote:
> On Thu, Feb 25, 2016 at 6:35 AM, Jeremy Stanley  wrote:
> > On 2016-02-25 02:46:13 -0600 (-0600), JP Maxwell wrote:
> >> Please be aware that you can now create accounts under the mobile
> >> view in the wiki native user table. I just created an account for
> >> JpMaxMan.  Not sure if this matters but wanted to make sure you
> >> were aware.
> >
> > Oh, yes I think having a random garbage question/answer was in fact
> > previously preventing account creation under the mobile view. We
> > probably need a way to disable mobile view account creation as it
> > bypasses OpenID authentication entirely.
> 
> So that's what it was doing! We'll have to tackle the mobile view issue.
> 
> Otherwise, quick update here:
> 
> The captcha didn't appear to help stem the spam tide. We'll want to
> explore and start implementing some of the other solutions.
> 
> I did some database poking around today and it does seem like all the
> users do have launchpad accounts and email addresses.
> 
So, I have a few hours before jumping on my plane and checked into this.  We are
using QuestyCaptcha which according to docs, should almost be impossible for
spammers to by pass in an automated fashion.  So, either our captcha is too
easy, or we didn't set it up properly.  I don't have SSH on wiki.o.o so others
will have to check logs.  I did test new pages and edits, and was promoted by
captcha.

As a next step, we might need to add additional apache2 configuration to
blacklist IPs.  I am reading up on that now.

> -- 
> Elizabeth Krumbach Joseph || Lyz || pleia2
> 
> ___
> OpenStack-Infra mailing list
> OpenStack-Infra@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra

___
OpenStack-Infra mailing list
OpenStack-Infra@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra

Re: [OpenStack-Infra] Wiki.o.o sustaining spam attack

[JP Maxwell]

Marton

Make sure you are using the right upstream repository. They are in version
1.25. Check out: https://wiki.openstack.org/wiki/Special:Version

Not that it shouldn't all be upgraded ;) be aware there seem to be config
file formatting differences in the latest version vs 1.25 as well.

J.P. Maxwell | tipit.net | fibercove.com
On Feb 26, 2016 4:35 AM, "Marton Kiss"  wrote:

> I've deployed the mediawiki using our puppet modules to my dev machine,
> and we have more problems here:
> [image: The MediaWiki logo] MediaWiki 1.27 internal error
>
> MediaWiki 1.27 requires at least PHP version 5.5.9, you are using PHP
> 5.3.10-1ubuntu3.21.
> Supported PHP versions
>
> Please consider upgrading your copy of PHP
> . PHP versions less than 5.5.0 are no
> longer supported by the PHP Group and will not receive security or bugfix
> updates.
>
> If for some reason you are unable to upgrade your PHP version, you will
> need to download  an older
> version of MediaWiki from our website. See our compatibility page
>  for details of which
> versions are compatible with prior versions of PHP.
>
> The wiki.o.o seems to be running on precise, meanwhile the git consumed
> repo simply not supporting the PHP version provided there.
>
> M.
>
> On Fri, Feb 26, 2016 at 5:19 AM JP Maxwell  wrote:
>
>> Is it an option to put the question back to an impossible answer for even
>> a little while? I think it would be very telling if the spam continues then
>> there may be an exploit possibly tied to the launchpad SSO.  It would at
>> least give a clue where to focus.
>>
>> J.P. Maxwell | tipit.net | fibercove.com
>> On Feb 25, 2016 10:10 PM, "Elizabeth K. Joseph" 
>> wrote:
>>
>>> On Thu, Feb 25, 2016 at 6:35 AM, Jeremy Stanley 
>>> wrote:
>>> > On 2016-02-25 02:46:13 -0600 (-0600), JP Maxwell wrote:
>>> >> Please be aware that you can now create accounts under the mobile
>>> >> view in the wiki native user table. I just created an account for
>>> >> JpMaxMan.  Not sure if this matters but wanted to make sure you
>>> >> were aware.
>>> >
>>> > Oh, yes I think having a random garbage question/answer was in fact
>>> > previously preventing account creation under the mobile view. We
>>> > probably need a way to disable mobile view account creation as it
>>> > bypasses OpenID authentication entirely.
>>>
>>> So that's what it was doing! We'll have to tackle the mobile view issue.
>>>
>>> Otherwise, quick update here:
>>>
>>> The captcha didn't appear to help stem the spam tide. We'll want to
>>> explore and start implementing some of the other solutions.
>>>
>>> I did some database poking around today and it does seem like all the
>>> users do have launchpad accounts and email addresses.
>>>
>>> --
>>> Elizabeth Krumbach Joseph || Lyz || pleia2
>>>
>> ___
>> OpenStack-Infra mailing list
>> OpenStack-Infra@lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
>>
>
___
OpenStack-Infra mailing list
OpenStack-Infra@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra

Re: [OpenStack-Infra] Feedback after the infra midcycle

[Jeremy Stanley]

On 2016-02-26 10:51:12 +0100 (+0100), Thierry Carrez wrote:
> Spencer Krum wrote:
> [...]
> > * This is the first infra sprint that wasn't focused on 'infra
> > 101' training for brand new contributors. Several attendees said
> > they felt they got a lot more work done at this sprint than when
> > bootstrapping was a topic.
> 
> Well, we actually had a "storyboard"-focused sprint some time ago
> that was also not about training, and was also very productive :)

More precisely, the feedback on this point was about how we
succeeded in not doing any "Infra 101" at all and actively
discouraged anyone from showing up unless they were going to be able
to come focus on the sprint topic. That said, I skipped the
Storyboard sprint (I think?) so don't recall if any happened there.
Of the Infra mid-cycles I've attended, only our first one was
entirely 101-ish, but others had at least a day set aside for Infra
overview stuff.
-- 
Jeremy Stanley

___
OpenStack-Infra mailing list
OpenStack-Infra@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra

Re: [OpenStack-Infra] Feedback after the infra midcycle

[Thierry Carrez]

Spencer Krum wrote:

Things that went well:

* Many people expressed that focusing on a single topic made the event
great.


I think that's a very good point. I think we had three types of midcycle 
events. In the first one you meet to get together to know each other 
better and converge on a common goal. I call them "team building" 
midcycles. In the second one you have a specific objective and leverage 
high-bandwidth and increased focus to get there a lot faster than you 
would online. I call those "sprints". The third one was a bit specific 
to the infra team, where you would train people on the art of infra... I 
would call that "onboarding" or "training".


My hope is that with the event split we won't need as much "team 
building" ones, but I can certainly see "sprints" (online or f2f) and 
onboarding events to persist. The key benefit is that it's easier to 
decide if you should come when you precisely know what will be covered 
and decided there. For example I opted to skip the infracloud sprint, 
while I may have felt compelled to attend a more generic "Infra 
midcycle" for fear of missing out on critical decisions.



[...]
* This is the first infra sprint that wasn't focused on 'infra 101'
training for brand new contributors. Several attendees said they felt
they got a lot more work done at this sprint than when bootstrapping was
a topic.


Well, we actually had a "storyboard"-focused sprint some time ago that 
was also not about training, and was also very productive :)


--
Thierry Carrez (ttx)

___
OpenStack-Infra mailing list
OpenStack-Infra@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra