Yeah, I checked it and it is internal job runner: https://www.mediawiki.org/wiki/Manual:Job_queue
M. On Fri, Feb 26, 2016 at 7:00 PM JP Maxwell <[email protected]> wrote: > A quick google indicates this may be an unrelated issue that should be > fixed, but I don’t *think* it is related to the spam. > > *J.P. Maxwell* | tipit.net | fibercove.com <http://www.fibercove.com> > > On Fri, Feb 26, 2016 at 11:56 AM, Marton Kiss <[email protected]> > wrote: > > I'm going to get a dinner, but I'll be on irc after, so if I can help > somehow, I will be here. #openstack-infra mrmartin > > M. > > On Fri, Feb 26, 2016 at 6:51 PM Paul Belanger <[email protected]> wrote: > >> On phone but patch puppet-mediawiki and enable captcha for all pages. We >> only did edit and create >> On Feb 26, 2016 10:38 AM, Marton Kiss <[email protected]> wrote: >> >> I see a ton of incoming post requests: >> >> POST >> /w/index.php?title=Special%3ARunJobs&tasks=jobs&maxjobs=1&sigexpiry=1456508270&signature=571cfb216f944b15d2eee1c0253d08b77003328e >> >> M. >> >> On Fri, Feb 26, 2016 at 6:35 PM Marton Kiss <[email protected]> >> wrote: >> >>> Oh, I can login. So what we need? >>> >>> M. >>> >>> On Fri, Feb 26, 2016 at 6:33 PM JP Maxwell <[email protected]> wrote: >>> >>>> I think what Jimmy is referring to is what I was suggesting by removing >>>> the extensions / making the question impossible to answer. Basically a >>>> series of rapid fire changes while tailing the logs and seeing what stops >>>> the spam. Once you know what worked then you can submit as an official >>>> patch. But being able to quickly try these things on a server actually >>>> under attack is the fastest path toward identifying the fix. >>>> >>>> *J.P. Maxwell* | tipit.net | fibercove.com <http://www.fibercove.com> >>>> >>>> On Fri, Feb 26, 2016 at 11:25 AM, Paul Belanger <[email protected]> >>>> wrote: >>>> >>>> On Fri, Feb 26, 2016 at 11:08:18AM -0600, Jimmy McArthur wrote: >>>> > Given the state of the wiki a the moment, I think taking the quickest >>>> path >>>> > to get it fixed would be prudent. Is there a way we can get JP root >>>> access >>>> > to this server, even temporarily? We get 25% of our website traffic (2 >>>> > million visitors) to the wiki. I realize we're all after the same >>>> thing, but >>>> > spammers are not going to hit the dev environment, so there's really >>>> no way >>>> > to tell if teh problem is fixed without actually working directly on >>>> the >>>> > production machine. This should be a 30 minute fix. >>>> > >>>> I am still unclear what the 30min fix is. If really 30mins, then it >>>> shouldn't be >>>> hard to get the fix into our workflow. Could somebody please elaborate. >>>> >>>> If we are talking about deploying new versions of php or mediawiki >>>> manually, I >>>> not be in-favor of this. To me, while the attack sucks, we should be >>>> working on >>>> 2 fronts. Getting the help needed to mitigate the attack, then adding >>>> the >>>> changes into -infra workflow in parallel. >>>> >>>> > I realize there is a lot of risk in giving ssh access to infra >>>> machines, but >>>> > I think it's worth taking a look at either putting this machine in a >>>> place >>>> > where a different level of admin could access it without giving away >>>> the >>>> > keys to the entire OpenStack infrastructure or figuring out a way to >>>> set up >>>> > credentials with varying levels of access. >>>> > >>>> As a note, all the work I've been doing to help with the attack hasn't >>>> require >>>> SSH access for me to wiki.o.o. I did need infra-root help to expose our >>>> configuration safely. I'd rather take some time to see what the fixes >>>> are, >>>> having infra-root apply changes, then move them into puppet. >>>> >>>> It also has been discussed to simply disable write access to the wiki >>>> if we >>>> really want spamming to stop, obviously that will affect normal usage. >>>> >>>> > Jimmy >>>> > >>>> > Paul Belanger wrote: >>>> > >On Fri, Feb 26, 2016 at 10:12:12AM -0600, JP Maxwell wrote: >>>> > >>But if you wanted to upgrade everything, remove the mobile view >>>> extension, >>>> > >>test in a dev/staging environment then deploy to production fingers >>>> > >>crossed, I think that would be a valid approach as well. >>>> > >> >>>> > >Current review up[1]. I'll launch a node tonight / tomorrow locally >>>> to see how >>>> > >puppet reacts. I suspect there will be some issues. >>>> > > >>>> > >If infra-roots are fine with this approach, we can use that box to >>>> test against. >>>> > > >>>> > >[1] https://review.openstack.org/#/c/285405/ >>>> > > >>>> > >>J.P. Maxwell | tipit.net | fibercove.com >>>> > >>On Feb 26, 2016 10:08 AM, "JP Maxwell"<[email protected]> wrote: >>>> > >> >>>> > >>>Plus one except in this case it is much easier to know if our >>>> efforts are >>>> > >>>working on production because the spam either stops or not. >>>> > >>> >>>> > >>>J.P. Maxwell | tipit.net | fibercove.com >>>> > >>>On Feb 26, 2016 9:48 AM, "Paul Belanger"<[email protected]> >>>> wrote: >>>> > >>> >>>> > >>>>On Fri, Feb 26, 2016 at 09:18:00AM -0600, JP Maxwell wrote: >>>> > >>>>>I really think you might consider the option that there is a >>>> > >>>>vulnerability >>>> > >>>>>in one of the extensions. If that is the case black listing IPs >>>> will be >>>> > >>>>an >>>> > >>>>>ongoing wild goose chase. >>>> > >>>>> >>>> > >>>>>I think this would be easily proven or disproven by making the >>>> questy >>>> > >>>>>question impossible and see if the spam continues. >>>> > >>>>> >>>> > >>>>We'll have to let an infra-root make that call. Since nobody >>>> would be >>>> > >>>>able to >>>> > >>>>use the wiki. Honestly, I'd rather spend the time standing up a >>>> mirror dev >>>> > >>>>instance for us to work on, rather then production. >>>> > >>>> >>>> > >>>>>J.P. Maxwell | tipit.net | fibercove.com >>>> > >>>>>On Feb 26, 2016 9:12 AM, "Paul Belanger"<[email protected]> >>>> wrote: >>>> > >>>>> >>>> > >>>>>>On Thu, Feb 25, 2016 at 08:10:34PM -0800, Elizabeth K. Joseph >>>> wrote: >>>> > >>>>>>>On Thu, Feb 25, 2016 at 6:35 AM, Jeremy Stanley< >>>> [email protected]> >>>> > >>>>>>wrote: >>>> > >>>>>>>>On 2016-02-25 02:46:13 -0600 (-0600), JP Maxwell wrote: >>>> > >>>>>>>>>Please be aware that you can now create accounts under the >>>> mobile >>>> > >>>>>>>>>view in the wiki native user table. I just created an >>>> account for >>>> > >>>>>>>>>JpMaxMan. Not sure if this matters but wanted to make sure >>>> you >>>> > >>>>>>>>>were aware. >>>> > >>>>>>>>Oh, yes I think having a random garbage question/answer was in >>>> > >>>>fact >>>> > >>>>>>>>previously preventing account creation under the mobile view. >>>> We >>>> > >>>>>>>>probably need a way to disable mobile view account creation >>>> as it >>>> > >>>>>>>>bypasses OpenID authentication entirely. >>>> > >>>>>>>So that's what it was doing! We'll have to tackle the mobile >>>> view >>>> > >>>>issue. >>>> > >>>>>>>Otherwise, quick update here: >>>> > >>>>>>> >>>> > >>>>>>>The captcha didn't appear to help stem the spam tide. We'll >>>> want to >>>> > >>>>>>>explore and start implementing some of the other solutions. >>>> > >>>>>>> >>>> > >>>>>>>I did some database poking around today and it does seem like >>>> all >>>> > >>>>the >>>> > >>>>>>>users do have launchpad accounts and email addresses. >>>> > >>>>>>> >>>> > >>>>>>So, I have a few hours before jumping on my plane and checked >>>> into >>>> > >>>>this. >>>> > >>>>>>We are >>>> > >>>>>>using QuestyCaptcha which according to docs, should almost be >>>> > >>>>impossible >>>> > >>>>>>for >>>> > >>>>>>spammers to by pass in an automated fashion. So, either our >>>> captcha >>>> > >>>>is too >>>> > >>>>>>easy, or we didn't set it up properly. I don't have SSH on >>>> wiki.o.o >>>> > >>>>so >>>> > >>>>>>others >>>> > >>>>>>will have to check logs. I did test new pages and edits, and was >>>> > >>>>promoted >>>> > >>>>>>by >>>> > >>>>>>captcha. >>>> > >>>>>> >>>> > >>>>>>As a next step, we might need to add additional apache2 >>>> configuration >>>> > >>>>to >>>> > >>>>>>blacklist IPs. I am reading up on that now. >>>> > >>>>>> >>>> > >>>>>>>-- >>>> > >>>>>>>Elizabeth Krumbach Joseph || Lyz || pleia2 >>>> > >>>>>>> >>>> > >>>>>>>_______________________________________________ >>>> > >>>>>>>OpenStack-Infra mailing list >>>> > >>>>>>>[email protected] >>>> > >>>>>>> >>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra >>>> > >>>>>>_______________________________________________ >>>> > >>>>>>OpenStack-Infra mailing list >>>> > >>>>>>[email protected] >>>> > >>>>>> >>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra >>>> > >>>>>> >>>> > > >>>> > >_______________________________________________ >>>> > >OpenStack-Infra mailing list >>>> > >[email protected] >>>> > >http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra >>>> > >>>> >>>> > _______________________________________________ >>>> > OpenStack-Infra mailing list >>>> > [email protected] >>>> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra >>>> >>>> >>>> _______________________________________________ >>>> OpenStack-Infra mailing list >>>> [email protected] >>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra >>>> >>>> _______________________________________________ >>>> OpenStack-Infra mailing list >>>> [email protected] >>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra >>>> >>>
_______________________________________________ OpenStack-Infra mailing list [email protected] http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
