Re: [OpenStack-Infra] Logs for Ask.O.o - chasing false positive spam labeling

[Christopher Bartz]

Hello,

I am the test user, mentioned by Tom.

Since some months, and unfortunately  still now after your mentioned 
upgrade, I get the error "Spam is detected on your post - sorry if it 
was a mistake." if I want to comment a post or if I want to create a new 
post. It is independent of the content I am producing, it seems that my 
account is marked as "spam" in a certain way.


Thank you for your help.

Best Regards,

Christopher Bartz


Am 05.01.2017 um 17:12 schrieb Tom Fifield:

On 05/01/17 23:26, Jeremy Stanley wrote:

On 2017-01-02 23:03:07 + (+), Jeremy Stanley wrote:
[...]

We're basically ready to retry the previous upgrade now that some
issues have been identified/corrected by Marton. Change
https://review.openstack.org/408657 to trigger it is waiting on
https://review.openstack.org/416072 to grant him a shell account on
the production server for improved reactivity if it fails again, so
that he can collaborate with us more directly troubleshooting before
we have to make the call to fix or revert.

[...]

With much credit to Marton's efforts, we upgraded Askbot on
ask.openstack.org yesterday to a much more recent version (and it's
still up now a day later with no complaints AFAIK). It's worth
rechecking for further issues similar to what you experienced
previously, to see whether they're resolved now.



Wohoo! Many thanks - I just asked my test user to try it out

___
OpenStack-Infra mailing list
OpenStack-Infra@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra


--
Christopher Bartz
Entwickler / Administrator von Cloud-Diensten
Abteilung Systeme

Deutsches Klimarechenzentrum GmbH (DKRZ)
Bundesstraße 45 a • D-20146 Hamburg • Germany

Phone:+49 40 460094-406
FAX:  +49 40 460094-270
Email:ba...@dkrz.de
URL:  www.dkrz.de

Geschäftsführer: Prof. Dr. Thomas Ludwig
Sitz der Gesellschaft: Hamburg
Amtsgericht Hamburg HRB 39784




smime.p7s
Description: S/MIME Cryptographic Signature
___
OpenStack-Infra mailing list
OpenStack-Infra@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra

Re: [OpenStack-Infra] Logs for Ask.O.o - chasing false positive spam labeling

[Tom Fifield]

On 05/01/17 23:26, Jeremy Stanley wrote:

On 2017-01-02 23:03:07 + (+), Jeremy Stanley wrote:
[...]

We're basically ready to retry the previous upgrade now that some
issues have been identified/corrected by Marton. Change
https://review.openstack.org/408657 to trigger it is waiting on
https://review.openstack.org/416072 to grant him a shell account on
the production server for improved reactivity if it fails again, so
that he can collaborate with us more directly troubleshooting before
we have to make the call to fix or revert.

[...]

With much credit to Marton's efforts, we upgraded Askbot on
ask.openstack.org yesterday to a much more recent version (and it's
still up now a day later with no complaints AFAIK). It's worth
rechecking for further issues similar to what you experienced
previously, to see whether they're resolved now.



Wohoo! Many thanks - I just asked my test user to try it out

___
OpenStack-Infra mailing list
OpenStack-Infra@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra

Re: [OpenStack-Infra] Logs for Ask.O.o - chasing false positive spam labeling

[Jeremy Stanley]

On 2017-01-02 23:03:07 + (+), Jeremy Stanley wrote:
[...]
> We're basically ready to retry the previous upgrade now that some
> issues have been identified/corrected by Marton. Change
> https://review.openstack.org/408657 to trigger it is waiting on
> https://review.openstack.org/416072 to grant him a shell account on
> the production server for improved reactivity if it fails again, so
> that he can collaborate with us more directly troubleshooting before
> we have to make the call to fix or revert.
[...]

With much credit to Marton's efforts, we upgraded Askbot on
ask.openstack.org yesterday to a much more recent version (and it's
still up now a day later with no complaints AFAIK). It's worth
rechecking for further issues similar to what you experienced
previously, to see whether they're resolved now.
-- 
Jeremy Stanley

___
OpenStack-Infra mailing list
OpenStack-Infra@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra

Re: [OpenStack-Infra] Logs for Ask.O.o - chasing false positive spam labeling

[Jeremy Stanley]

On 2017-01-03 10:54:48 +0800 (+0800), Tom Fifield wrote:
[...]
> If you check for a POST after a GET of:
> 
> /admin/auth/user/2253/delete/
> 
> Around
> 
> 2017-02-03 02:52
> 
> from
> 
> 1.169.254.207
> 
> that should be it.

Assuming you mean 2017-01-03 then yes, I see one from that IP
address around the aforementioned time:

1.169.254.207 - - [03/Jan/2017:02:51:47 +] "GET
/admin/auth/user/2253/delete/ HTTP/1.1" 200 13733
"https://ask.openstack.org/admin/auth/user/2253/; "Mozilla/5.0
(X11; Ubuntu; Linux x86_64; rv:50.0) Gecko/20100101
Firefox/50.0"

1.169.254.207 - - [03/Jan/2017:02:52:02 +] "POST
/admin/auth/user/2253/delete/ HTTP/1.1" 500 20828
"https://ask.openstack.org/admin/auth/user/2253/delete/;
"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:50.0) Gecko/20100101
Firefox/50.0"

> I believe the OOM error comes from trying to view any post in the django
> admin interface (/admin/askbot/post/%d/ ) -- those things essentially never
> stop loading for some reason.

Makes sense. The 500 above does not coincide with any OOM event (nor
can I find anything to correlate it to in syslog, Apache error logs,
Askbot application logs, Solr/Jetty request or stderrout logs, et
cetera).
-- 
Jeremy Stanley

___
OpenStack-Infra mailing list
OpenStack-Infra@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra

Re: [OpenStack-Infra] Logs for Ask.O.o - chasing false positive spam labeling

[Tom Fifield]

On 03/01/17 07:03, Jeremy Stanley wrote:

On 2016-12-30 12:38:02 +0800 (+0800), Tom Fifield wrote:

This problem (legitimate users having all posts flatly rejected as
spam) is still happening. Any progress?


We're basically ready to retry the previous upgrade now that some
issues have been identified/corrected by Marton. Change
https://review.openstack.org/408657 to trigger it is waiting on
https://review.openstack.org/416072 to grant him a shell account on
the production server for improved reactivity if it fails again, so
that he can collaborate with us more directly troubleshooting before
we have to make the call to fix or revert.


Excellent!


I've been doing what I can without access to the server, but my
latest attempt - completely deleting an account so it could be
re-created by the affected user - was thwarted by a 500 error. Did
that appear in the logs?


Given that I don't know the exact time you tried nor your IP
address, and am unsure what that failure would look like in the logs
aside from coming with a 500 error code this Apache access log entry
18 minutes before your E-mail's timestamp stands out:

[30/Dec/2016:04:20:40 +] "GET /admin/askbot/post/25842/
HTTP/1.1" 500 917
"https://ask.openstack.org/admin/auth/user/2253/delete/;
"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:50.0) Gecko/20100101
Firefox/50.0"

I don't see anything related around that timeframe in the Apache
error log files, the Askbot application log, the Solr Jetty log...
though there's this in dmesg which is suspiciously close (timestamps
in dmesg aren't entirely reliable, so this could have happened 5
minutes earlier):

[Fri Dec 30 04:25:25 2016] apache2 invoked oom-killer:
gfp_mask=0x200da, order=0, oom_score_adj=0

And indeed, there's a spike in swap utilization around that time
which, given the five-minute granularity could certainly support the
notion that a runaway process ate all available virtual memory on
the system:

http://cacti.openstack.org/cacti/graph.php?action=zoom_graph_id=2546_id=2_type=_start=1483029900_end=1483156284_height=120_width=500_font_size=10
 >

The RAM usage graph suggests we were doing okay with a target
utilization of ~50% so something eating an additional 6GiB of memory
in a matter of a few minutes would definitely count as anomalous:

http://cacti.openstack.org/cacti/graph.php?action=zoom_graph_id=2544_id=2_type=_start=1483029900_end=1483156284_height=120_width=500_font_size=10
 >

Was the memory spike caused by that deletion? Or was the deletion
error coincidental and caused by a memory event which just happened
to be going on at the same time? I have insufficient knowledge of
the system to be able to say either way. It's possible there are
additional logs I don't know to look at which could tell us.



Ah, how silly of me to fail to specify information - apologies!


If you check for a POST after a GET of:

/admin/auth/user/2253/delete/

Around

2017-02-03 02:52

from

1.169.254.207

that should be it.


I believe the OOM error comes from trying to view any post in the django 
admin interface (/admin/askbot/post/%d/ ) -- those things essentially 
never stop loading for some reason.







___
OpenStack-Infra mailing list
OpenStack-Infra@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra

Re: [OpenStack-Infra] Logs for Ask.O.o - chasing false positive spam labeling

[Jeremy Stanley]

On 2016-12-30 12:38:02 +0800 (+0800), Tom Fifield wrote:
> This problem (legitimate users having all posts flatly rejected as
> spam) is still happening. Any progress?

We're basically ready to retry the previous upgrade now that some
issues have been identified/corrected by Marton. Change
https://review.openstack.org/408657 to trigger it is waiting on
https://review.openstack.org/416072 to grant him a shell account on
the production server for improved reactivity if it fails again, so
that he can collaborate with us more directly troubleshooting before
we have to make the call to fix or revert.

> I've been doing what I can without access to the server, but my
> latest attempt - completely deleting an account so it could be
> re-created by the affected user - was thwarted by a 500 error. Did
> that appear in the logs?

Given that I don't know the exact time you tried nor your IP
address, and am unsure what that failure would look like in the logs
aside from coming with a 500 error code this Apache access log entry
18 minutes before your E-mail's timestamp stands out:

[30/Dec/2016:04:20:40 +] "GET /admin/askbot/post/25842/
HTTP/1.1" 500 917
"https://ask.openstack.org/admin/auth/user/2253/delete/;
"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:50.0) Gecko/20100101
Firefox/50.0"

I don't see anything related around that timeframe in the Apache
error log files, the Askbot application log, the Solr Jetty log...
though there's this in dmesg which is suspiciously close (timestamps
in dmesg aren't entirely reliable, so this could have happened 5
minutes earlier):

[Fri Dec 30 04:25:25 2016] apache2 invoked oom-killer:
gfp_mask=0x200da, order=0, oom_score_adj=0

And indeed, there's a spike in swap utilization around that time
which, given the five-minute granularity could certainly support the
notion that a runaway process ate all available virtual memory on
the system:

http://cacti.openstack.org/cacti/graph.php?action=zoom_graph_id=2546_id=2_type=_start=1483029900_end=1483156284_height=120_width=500_font_size=10
 >

The RAM usage graph suggests we were doing okay with a target
utilization of ~50% so something eating an additional 6GiB of memory
in a matter of a few minutes would definitely count as anomalous:

http://cacti.openstack.org/cacti/graph.php?action=zoom_graph_id=2544_id=2_type=_start=1483029900_end=1483156284_height=120_width=500_font_size=10
 >

Was the memory spike caused by that deletion? Or was the deletion
error coincidental and caused by a memory event which just happened
to be going on at the same time? I have insufficient knowledge of
the system to be able to say either way. It's possible there are
additional logs I don't know to look at which could tell us.
-- 
Jeremy Stanley

___
OpenStack-Infra mailing list
OpenStack-Infra@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra

Re: [OpenStack-Infra] Logs for Ask.O.o - chasing false positive spam labeling

[Tom Fifield]

On 10/11/16 22:23, Jeremy Stanley wrote:

On 2016-11-10 09:19:32 + (+), Marton Kiss wrote:

Jeremy, I can apply and test this patch in a current test environment, it
was sitting there for a while. Usually the config changes of askbot broke
the site.

[...]

If you get a chance, that would be a big help. I have logs from the
failed upgrade, but the gist is that the git resource provider
didn't update /srv/dist/askbot (Puppet's log never even mentions it
trying to do so) and then then migrate command threw:

AttributeError: 'Settings' object has no attribute 'ASKBOT_MULTILINGUAL'

Which their upgrade FAQ says is an indication that the urls.py
template needs to be updated (and that makes sense given that the
git repo never moved to the newer commit we specified). I mulled
over possibilities with others in #openstack-infra, and Spencer
suggested that latest=>true may be causing calls into the provider
to short-circuit since it always returns true if a commit or tag is
passed. The next round, I was going to try dropping that from the
commit and tag cases in puppet-askbot and seeing if it helps.



Hi,

This problem (legitimate users having all posts flatly rejected as spam) 
is still happening. Any progress?


I've been doing what I can without access to the server, but my latest 
attempt - completely deleting an account so it could be re-created by 
the affected user - was thwarted by a 500 error. Did that appear in the 
logs?




Regards,


Tom

___
OpenStack-Infra mailing list
OpenStack-Infra@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra

Re: [OpenStack-Infra] Logs for Ask.O.o - chasing false positive spam labeling

[Marton Kiss]

Jeremy, I can apply and test this patch in a current test environment, it
was sitting there for a while. Usually the config changes of askbot broke
the site.

Marton

On Wed, Nov 9, 2016 at 10:26 PM Jeremy Stanley  wrote:

> On 2016-11-09 19:53:27 + (+), Jeremy Stanley wrote:
> > On 2016-11-09 18:11:39 + (+), Jeremy Stanley wrote:
> > > On 2016-11-08 19:12:32 +0800 (+0800), Tom Fifield wrote:
> > > [...]
> > > > Upstream apparently revamped the spam system in the version marked
> > > > to upgrade to in:
> > > > https://review.openstack.org/#/c/274032/
> > >
> > > I've gone ahead and approved this just now... I was unaware it was
> > > out there waiting for approval. Sorry about that! I'll try to
> > > double-check that ask.o.o is still working correctly once it gets
> > > applied.
> > [...]
> >
> > Just to follow up, it looks like the git_resource for
> > /srv/dist/askbot did not get updated to the newly specified commit
> > in 274032 so we're digging into why that is. The site still seems to
> > be up and working for now.
>
> After a while, the site began to throw internal server errors, so
> I'm reverting with https://review.openstack.org/395797 for now until
> we can more thoroughly troubleshoot.
> --
> Jeremy Stanley
>
> ___
> OpenStack-Infra mailing list
> OpenStack-Infra@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
>
___
OpenStack-Infra mailing list
OpenStack-Infra@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra

Re: [OpenStack-Infra] Logs for Ask.O.o - chasing false positive spam labeling

[Jeremy Stanley]

On 2016-11-09 19:53:27 + (+), Jeremy Stanley wrote:
> On 2016-11-09 18:11:39 + (+), Jeremy Stanley wrote:
> > On 2016-11-08 19:12:32 +0800 (+0800), Tom Fifield wrote:
> > [...]
> > > Upstream apparently revamped the spam system in the version marked
> > > to upgrade to in:
> > > https://review.openstack.org/#/c/274032/
> > 
> > I've gone ahead and approved this just now... I was unaware it was
> > out there waiting for approval. Sorry about that! I'll try to
> > double-check that ask.o.o is still working correctly once it gets
> > applied.
> [...]
> 
> Just to follow up, it looks like the git_resource for
> /srv/dist/askbot did not get updated to the newly specified commit
> in 274032 so we're digging into why that is. The site still seems to
> be up and working for now.

After a while, the site began to throw internal server errors, so
I'm reverting with https://review.openstack.org/395797 for now until
we can more thoroughly troubleshoot.
-- 
Jeremy Stanley

___
OpenStack-Infra mailing list
OpenStack-Infra@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra

Re: [OpenStack-Infra] Logs for Ask.O.o - chasing false positive spam labeling

[Jeremy Stanley]

On 2016-11-08 19:12:32 +0800 (+0800), Tom Fifield wrote:
[...]
> Upstream apparently revamped the spam system in the version marked
> to upgrade to in:
> https://review.openstack.org/#/c/274032/

I've gone ahead and approved this just now... I was unaware it was
out there waiting for approval. Sorry about that! I'll try to
double-check that ask.o.o is still working correctly once it gets
applied.

> However, in order to make sure we're not losing up to 60% (akismet stat for
> October) of our potential legitimate posts while we wait for that, it would
> be great if there were some logs to try and find out what's going on.
> 
> Anyone able to dig and send me something?

I'm happy to. The Askbot application logs seem to contain nothing of
relevance, so I'm assuming you want the Apache logs in this case.
What timeframe are you interested in? Our retention right now is
on the order of several gigabytes compressed so don't want to
inundate you with the entirety if a subset will suffice.

> (If you got curious and want something to grep for, try
> 2001:638:70e:11:2ad2:44ff:*:* || 136.172.17.* )

I see a bunch of current hits for that v6 prefix, though not finding
any for the v4 one.
-- 
Jeremy Stanley

___
OpenStack-Infra mailing list
OpenStack-Infra@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra