Ok, I'll be there. M.
On Sat, Feb 27, 2016 at 5:15 PM Elizabeth K. Joseph <l...@princessleia.com> wrote: > We'll be getting together on Monday around 1700 UTC to work through this > together in a debug session in #openstack-infra (I'm too sick this weekend, > plus we need a time when more infra-root folks with the institutional > knowledge are around). > On Feb 27, 2016 05:37, "Marton Kiss" <marton.k...@gmail.com> wrote: > >> Yeah, the Settings.php was overriden by the latest puppet run. We need to >> wait for some infra guys to approve my patches and make it permanent: >> https://review.openstack.org/285669 Disable standard password based auth >> https://review.openstack.org/285672 Disable mobile frontend >> >> M. >> >> On Sat, Feb 27, 2016 at 2:27 PM JP Maxwell <j...@tipit.net> wrote: >> >>> FYI. Still seeing the mobile view... >>> >>> J.P. Maxwell | tipit.net | fibercove.com >>> On Feb 27, 2016 6:53 AM, "Marton Kiss" <marton.k...@gmail.com> wrote: >>> >>>> Yes, applied them manually. Let's wait a few hours, and check for new >>>> spam content / user accounts. >>>> >>>> M. >>>> JP Maxwell <j...@tipit.net> (időpont: 2016. febr. 27., Szo, 13:50) ezt >>>> írta: >>>> >>>>> Cool. Are these applied? Any indication it has stopped the spam? >>>>> Should we clear out these non launchpad accounts from the DB? >>>>> >>>>> J.P. Maxwell | tipit.net | fibercove.com >>>>> On Feb 27, 2016 6:47 AM, "Marton Kiss" <marton.k...@gmail.com> wrote: >>>>> >>>>>> And the mobile frontend will be disabled permanently with this patch: >>>>>> https://review.openstack.org/285672 Disable mobile frontend >>>>>> >>>>>> M. >>>>>> >>>>>> On Sat, Feb 27, 2016 at 1:39 PM Marton Kiss <marton.k...@gmail.com> >>>>>> wrote: >>>>>> >>>>>>> I made some investigation, and it seems to be that the spam pages >>>>>>> are created by accounts registered with password accounts, and the >>>>>>> launchpad openid auth is not affected at all. >>>>>>> >>>>>>> So the spam script is creating accounts like this: >>>>>>> mysql> select * from user where user_name = 'CedricJamieson'\G; >>>>>>> *************************** 1. row *************************** >>>>>>> user_id: 7494 >>>>>>> user_name: CedricJamieson >>>>>>> user_real_name: Cedric Jamieson >>>>>>> user_password: >>>>>>> :pbkdf2:sha256:10000:128:Mlo9tdaP+38niZrrEka7Ow==:jEVnrTclkwIpE1RzJywDlrSvkY5G3idYwOwYRkv5O0J/MSHjY+gdhtKmArQ53v6/w7o8E1wXb2QOR6HdL5TPfOI1bswS/fYXVVYjPjkEEdxqZ8q9L5p2f3N6rEYpMfT5tk+wDiy+j5aimrHrGSga44hndAHgX9/SnqUyxlutDVY= >>>>>>> user_newpassword: >>>>>>> user_newpass_time: NULL >>>>>>> user_email: balashkina.evdok...@mail.ru >>>>>>> user_touched: 20160227052454 >>>>>>> user_token: 7c39e44e849fb0e2bfae8790d6cc1379 >>>>>>> user_email_authenticated: NULL >>>>>>> user_email_token: be963ac3bd43e70ff2f323063c61e320 >>>>>>> user_email_token_expires: 20160305052441 >>>>>>> user_registration: 20160227052441 >>>>>>> user_editcount: 2 >>>>>>> user_password_expires: NULL >>>>>>> >>>>>>> The user_password field is always filled with a value, meanwhile >>>>>>> this field of non-infected user accounts with openid logins is empty. >>>>>>> We have 423 total accounts with passwords: >>>>>>> mysql> select count(*) from user where user_password != ''; >>>>>>> +----------+ >>>>>>> | count(*) | >>>>>>> +----------+ >>>>>>> | 423 | >>>>>>> +----------+ >>>>>>> 1 row in set (0.00 sec) >>>>>>> >>>>>>> Mediawiki logs-in the newly created users without any preliminary >>>>>>> email confirmation, right after the registration. I disabled the >>>>>>> standard >>>>>>> user login page, as described here: >>>>>>> https://www.mediawiki.org/wiki/Manual:Special_pages#Disabling_Special:UserLogin_and_Special:UserLogout_pages >>>>>>> >>>>>>> And I made this patch to make it permanent: >>>>>>> https://review.openstack.org/285669 Disable standard password based >>>>>>> auth >>>>>>> >>>>>>> Just for the record, the last spam user account: >>>>>>> 7536 | EarthaChester22 >>>>>>> >>>>>>> Marton >>>>>>> >>>>>>> >>>>>>> On Sat, Feb 27, 2016 at 8:31 AM Marton Kiss <marton.k...@gmail.com> >>>>>>> wrote: >>>>>>> >>>>>>>> Hi, >>>>>>>> >>>>>>>> I created the following patch, infra cores must approve that: >>>>>>>> https://review.openstack.org/285641 Add ssh key of JP Maxwell to >>>>>>>> wiki.o.o >>>>>>>> >>>>>>>> Marton >>>>>>>> >>>>>>>> On Sat, Feb 27, 2016 at 6:41 AM JP Maxwell <j...@tipit.net> wrote: >>>>>>>> >>>>>>>>> Marton has SSH access and applied a patch earlier today. It >>>>>>>>> appears the spam continues to flow: >>>>>>>>> >>>>>>>>> >>>>>>>>> https://wiki.openstack.org/wiki/40_Thoughts_Of_Using_Open_Shelves_On_A_Kitchen >>>>>>>>> >>>>>>>>> Marton let me know if you can look at it some more or Infra if you >>>>>>>>> want to give me SSH I'll do so as well in the morning (public key >>>>>>>>> attached). >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> ssh-rsa >>>>>>>>> AAAAB3NzaC1yc2EAAAABIwAAAQEA2b5I7Yff9FCrtRmSjpILUePi54Vbc8zqJTbzrIAQZGFLBi3xd2MLlhV5QVgpDBC9H3lGjbdnc81D3aFd3HwHT4dvvvyedT12PR3VDEpftdW84vw3jzdtALcayOQznjbGnScwvX5SgnRhNxuX9Rkh8qNvOsjYPUafRr9azkQoomJFkdNVI4Vb5DbLhTpt18FPeOf0UuqDt/J2tHI4SjZ3kjzr7Nbwpg8xGgANPNE0+2pJbwCA8YDt4g3bzfzvVafQs5o9Gfc9tudkR9ugQG1M+EWCgu42CleOwMTd/rYEB2fgNNPsZAWqwQfdPajVuk70EBKUEQSyoA09eEZX+xJN9Q== >>>>>>>>> jpmax...@tipit.net >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> J.P. Maxwell / tipit.net <http://www.tipit.net> >>>>>>>>> >>>>>>>>> >>>>>>>>> On Fri, Feb 26, 2016 at 12:09 PM, Jimmy McArthur < >>>>>>>>> ji...@openstack.org> wrote: >>>>>>>>> >>>>>>>>>> Super thankful for all the folks that have jumped in over the >>>>>>>>>> last couple of days to help with the puppetization, etc... I just >>>>>>>>>> feel like >>>>>>>>>> we're taking a very wrong approach here. >>>>>>>>>> >>>>>>>>>> Paul Belanger wrote: >>>>>>>>>> >>>>>>>>>> Right, and I don't have an issue with that approach. Based on the >>>>>>>>>> work we did >>>>>>>>>> yesterday, anybody can do that via our workflow. Please submit a >>>>>>>>>> patch to >>>>>>>>>> puppet-mediawiki[1] and ping an infra-root in #openstack-infra IRC. >>>>>>>>>> >>>>>>>>>> What I'm proposing is the workflow is really meant for software, >>>>>>>>>> not for web applications. It's tedious and time consuming when what's >>>>>>>>>> needed here is a set of tests on the server. Submitting a patch, >>>>>>>>>> waiting >>>>>>>>>> for a +1, then getting on IRC to find someone with access (and time) >>>>>>>>>> to >>>>>>>>>> paste the logs is a pretty time consuming process for what should be >>>>>>>>>> a >>>>>>>>>> series of rapid-fire changes/fixes on the server. Especially when >>>>>>>>>> we're >>>>>>>>>> dealign with an active attack. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> We can then have somebody look at the logs. I think it is more >>>>>>>>>> about scheduling >>>>>>>>>> the task since more infra-root as travling back from the mid-cycle >>>>>>>>>> last night >>>>>>>>>> and today. >>>>>>>>>> >>>>>>>>>> Right, this is my point. This has been going on for 3 weeks (or >>>>>>>>>> more). Tom Fifeldt was asking for help without response. And here we >>>>>>>>>> are >>>>>>>>>> through another week and no closer to stemming the flow. >>>>>>>>>> >>>>>>>>>> I'm fully aware what I'm proposing goes against what Infra and >>>>>>>>>> the OpenStack workflow is all about, but I'd ask you all to look at >>>>>>>>>> this >>>>>>>>>> from a web development perspective instead of a software development >>>>>>>>>> perspective. >>>>>>>>>> >>>>>>>>>> Jimmy >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Last email from me, just on a plane. Will follow up when I land. >>>>>>>>>> >>>>>>>>>> [1] https://git.openstack.org/cgit/openstack-infra/puppet-mediawiki >>>>>>>>>> >>>>>>>>>> J.P. Maxwell | tipit.net [http://tipit.net] | fibercove.com >>>>>>>>>> [http://www.fibercove.com] >>>>>>>>>> On Fri, Feb 26, 2016 at 11:25 AM, Paul Belanger >>>>>>>>>> <pabelan...@redhat.com> <pabelan...@redhat.com> >>>>>>>>>> wrote: >>>>>>>>>> On Fri, Feb 26, 2016 at 11:08:18AM -0600, Jimmy McArthur wrote: >>>>>>>>>> >>>>>>>>>> Given the state of the wiki a the moment, I think taking the >>>>>>>>>> quickest path >>>>>>>>>> to get it fixed would be prudent. Is there a way we can get JP root >>>>>>>>>> access >>>>>>>>>> to this server, even temporarily? We get 25% of our website traffic >>>>>>>>>> (2 >>>>>>>>>> million visitors) to the wiki. I realize we're all after the same >>>>>>>>>> thing, >>>>>>>>>> >>>>>>>>>> but >>>>>>>>>> >>>>>>>>>> spammers are not going to hit the dev environment, so there's really >>>>>>>>>> no >>>>>>>>>> >>>>>>>>>> way >>>>>>>>>> >>>>>>>>>> to tell if teh problem is fixed without actually working directly on >>>>>>>>>> the >>>>>>>>>> production machine. This should be a 30 minute fix. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> I am still unclear what the 30min fix is. If really 30mins, then it >>>>>>>>>> shouldn't be >>>>>>>>>> hard to get the fix into our workflow. Could somebody please >>>>>>>>>> elaborate. >>>>>>>>>> >>>>>>>>>> If we are talking about deploying new versions of php or mediawiki >>>>>>>>>> manually, >>>>>>>>>> I >>>>>>>>>> not be in-favor of this. To me, while the attack sucks, we should be >>>>>>>>>> working >>>>>>>>>> on >>>>>>>>>> 2 fronts. Getting the help needed to mitigate the attack, then >>>>>>>>>> adding the >>>>>>>>>> changes into -infra workflow in parallel. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> I realize there is a lot of risk in giving ssh access to infra >>>>>>>>>> machines, >>>>>>>>>> >>>>>>>>>> but >>>>>>>>>> >>>>>>>>>> I think it's worth taking a look at either putting this machine in a >>>>>>>>>> place >>>>>>>>>> where a different level of admin could access it without giving away >>>>>>>>>> the >>>>>>>>>> keys to the entire OpenStack infrastructure or figuring out a way to >>>>>>>>>> set >>>>>>>>>> >>>>>>>>>> up >>>>>>>>>> >>>>>>>>>> credentials with varying levels of access. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> As a note, all the work I've been doing to help with the attack >>>>>>>>>> hasn't >>>>>>>>>> require >>>>>>>>>> SSH access for me to wiki.o.o. I did need infra-root help to expose >>>>>>>>>> our >>>>>>>>>> configuration safely. I'd rather take some time to see what the >>>>>>>>>> fixes are, >>>>>>>>>> having infra-root apply changes, then move them into puppet. >>>>>>>>>> >>>>>>>>>> It also has been discussed to simply disable write access to the >>>>>>>>>> wiki if we >>>>>>>>>> really want spamming to stop, obviously that will affect normal >>>>>>>>>> usage. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Jimmy >>>>>>>>>> >>>>>>>>>> Paul Belanger wrote: >>>>>>>>>> >>>>>>>>>> On Fri, Feb 26, 2016 at 10:12:12AM -0600, JP Maxwell wrote: >>>>>>>>>> >>>>>>>>>> But if you wanted to upgrade everything, remove the mobile view >>>>>>>>>> >>>>>>>>>> extension, >>>>>>>>>> >>>>>>>>>> test in a dev/staging environment then deploy to production fingers >>>>>>>>>> crossed, I think that would be a valid approach as well. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Current review up[1]. I'll launch a node tonight / tomorrow locally >>>>>>>>>> to >>>>>>>>>> >>>>>>>>>> see >>>>>>>>>> how >>>>>>>>>> >>>>>>>>>> puppet reacts. I suspect there will be some issues. >>>>>>>>>> >>>>>>>>>> If infra-roots are fine with this approach, we can use that box to >>>>>>>>>> test >>>>>>>>>> >>>>>>>>>> against. >>>>>>>>>> >>>>>>>>>> [1] https://review.openstack.org/#/c/285405/ >>>>>>>>>> >>>>>>>>>> J.P. Maxwell | tipit.net | fibercove.com >>>>>>>>>> On Feb 26, 2016 10:08 AM, "JP Maxwell"<j...@tipit.net> >>>>>>>>>> <j...@tipit.net> wrote: >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Plus one except in this case it is much easier to know if our efforts >>>>>>>>>> >>>>>>>>>> are >>>>>>>>>> >>>>>>>>>> working on production because the spam either stops or not. >>>>>>>>>> >>>>>>>>>> J.P. Maxwell | tipit.net | fibercove.com >>>>>>>>>> On Feb 26, 2016 9:48 AM, "Paul Belanger"<pabelan...@redhat.com> >>>>>>>>>> <pabelan...@redhat.com> wrote: >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Fri, Feb 26, 2016 at 09:18:00AM -0600, JP Maxwell wrote: >>>>>>>>>> >>>>>>>>>> I really think you might consider the option that there is a >>>>>>>>>> >>>>>>>>>> vulnerability >>>>>>>>>> >>>>>>>>>> in one of the extensions. If that is the case black listing IPs will >>>>>>>>>> >>>>>>>>>> be >>>>>>>>>> >>>>>>>>>> an >>>>>>>>>> >>>>>>>>>> ongoing wild goose chase. >>>>>>>>>> >>>>>>>>>> I think this would be easily proven or disproven by making the questy >>>>>>>>>> question impossible and see if the spam continues. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> We'll have to let an infra-root make that call. Since nobody would be >>>>>>>>>> able to >>>>>>>>>> use the wiki. Honestly, I'd rather spend the time standing up a >>>>>>>>>> mirror >>>>>>>>>> >>>>>>>>>> dev >>>>>>>>>> >>>>>>>>>> instance for us to work on, rather then production. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> J.P. Maxwell | tipit.net | fibercove.com >>>>>>>>>> On Feb 26, 2016 9:12 AM, "Paul Belanger"<pabelan...@redhat.com> >>>>>>>>>> <pabelan...@redhat.com> >>>>>>>>>> >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>> On Thu, Feb 25, 2016 at 08:10:34PM -0800, Elizabeth K. Joseph wrote: >>>>>>>>>> >>>>>>>>>> On Thu, Feb 25, 2016 at 6:35 AM, Jeremy Stanley<fu...@yuggoth.org> >>>>>>>>>> <fu...@yuggoth.org> >>>>>>>>>> >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>> On 2016-02-25 02:46:13 -0600 (-0600), JP Maxwell wrote: >>>>>>>>>> >>>>>>>>>> Please be aware that you can now create accounts under the mobile >>>>>>>>>> view in the wiki native user table. I just created an account for >>>>>>>>>> JpMaxMan. Not sure if this matters but wanted to make sure you >>>>>>>>>> were aware. >>>>>>>>>> >>>>>>>>>> Oh, yes I think having a random garbage question/answer was in >>>>>>>>>> >>>>>>>>>> fact >>>>>>>>>> >>>>>>>>>> previously preventing account creation under the mobile view. We >>>>>>>>>> probably need a way to disable mobile view account creation as it >>>>>>>>>> bypasses OpenID authentication entirely. >>>>>>>>>> >>>>>>>>>> So that's what it was doing! We'll have to tackle the mobile view >>>>>>>>>> >>>>>>>>>> issue. >>>>>>>>>> >>>>>>>>>> Otherwise, quick update here: >>>>>>>>>> >>>>>>>>>> The captcha didn't appear to help stem the spam tide. We'll want to >>>>>>>>>> explore and start implementing some of the other solutions. >>>>>>>>>> >>>>>>>>>> I did some database poking around today and it does seem like all >>>>>>>>>> >>>>>>>>>> the >>>>>>>>>> >>>>>>>>>> users do have launchpad accounts and email addresses. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> So, I have a few hours before jumping on my plane and checked into >>>>>>>>>> >>>>>>>>>> this. >>>>>>>>>> >>>>>>>>>> We are >>>>>>>>>> using QuestyCaptcha which according to docs, should almost be >>>>>>>>>> >>>>>>>>>> impossible >>>>>>>>>> >>>>>>>>>> for >>>>>>>>>> spammers to by pass in an automated fashion. So, either our captcha >>>>>>>>>> >>>>>>>>>> is too >>>>>>>>>> >>>>>>>>>> easy, or we didn't set it up properly. I don't have SSH on wiki.o.o >>>>>>>>>> >>>>>>>>>> so >>>>>>>>>> >>>>>>>>>> others >>>>>>>>>> will have to check logs. I did test new pages and edits, and was >>>>>>>>>> >>>>>>>>>> promoted >>>>>>>>>> >>>>>>>>>> by >>>>>>>>>> captcha. >>>>>>>>>> >>>>>>>>>> As a next step, we might need to add additional apache2 >>>>>>>>>> >>>>>>>>>> configuration >>>>>>>>>> >>>>>>>>>> to >>>>>>>>>> >>>>>>>>>> blacklist IPs. I am reading up on that now. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> Elizabeth Krumbach Joseph || Lyz || pleia2 >>>>>>>>>> >>>>>>>>>> _______________________________________________ >>>>>>>>>> OpenStack-Infra mailing listopenstack-in...@lists.openstack.org >>>>>>>>>> >>>>>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra >>>>>>>>>> _______________________________________________ >>>>>>>>>> OpenStack-Infra mailing listopenstack-in...@lists.openstack.org >>>>>>>>>> >>>>>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra >>>>>>>>>> >>>>>>>>>> _______________________________________________ >>>>>>>>>> OpenStack-Infra mailing >>>>>>>>>> listOpenStack-Infra@lists.openstack.orghttp://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra >>>>>>>>>> >>>>>>>>>> _______________________________________________ >>>>>>>>>> OpenStack-Infra mailing >>>>>>>>>> listOpenStack-Infra@lists.openstack.orghttp://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra >>>>>>>>>> >>>>>>>>>> _______________________________________________ >>>>>>>>>> OpenStack-Infra mailing >>>>>>>>>> listOpenStack-Infra@lists.openstack.orghttp://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> OpenStack-Infra mailing list >>>>>>>>> OpenStack-Infra@lists.openstack.org >>>>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra >>>>>>>>> >>>>>>>> >> _______________________________________________ >> OpenStack-Infra mailing list >> OpenStack-Infra@lists.openstack.org >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra >> >>
_______________________________________________ OpenStack-Infra mailing list OpenStack-Infra@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra