Thanks, this is excellent work. On Thu, Apr 23, 2015 at 12:10:21AM +0200, Walter Hop wrote: > In our short IRC discussion I promised to give a better reproduction of this > issue. > > To recap, with Apache + mod_apparmor + Keepalive enabled, some (iOS) clients > are triggering an AppArmor violation like the following, causing the HTTP > connection to be reset: > > type=1400 audit(1429734615.495:288): apparmor="DENIED" operation="file_perm" > profile="/usr/sbin/apache2//HANDLING_UNTRUSTED_INPUT" > > name="/sites/keepalive/www/wp-content/plugins/wptouch/resources/icons/elegant/Paper.png" > pid=17735 comm="apache2" requested_mask="r" denied_mask="r" fsuid=33 > ouid=1000 > > I made packet dumps when triggering the problem with an iOS device, and from > those, distilled a sequence of GET requests and the time difference between > them: http://lf.ms/requests.json > > I made a small script to replay those requests with exact timings to a local > VM. This reproduces the problem nearly 100% for me. The script is here: > http://lf.ms/replay.phps
I also am unable to see this script, as a mod_security firewall(?) seems to block it. > # php replay.php > Sleeping 0 usec... Getting / > Sleeping 31012 usec... Getting > /wp-content/plugins/wptouch/themes/foundation/modules/wptouch-icons/css/wptouch-icons.css?ver=2.3.3 > Sleeping 354 usec... Getting /wp-content/plugins/quicktime-embed/qtobject.js > Sleeping 478 usec... Getting > /wp-content/wptouch-data/cache/wptouch-567c11e69e00b40b5814ded9a6320ea84048adc2.js > Sleeping 6 usec... Getting > /wp-content/plugins/wp-disable-comments/javascript/wp-disable-comments.js?ver=0.4 > Sleeping 1078 usec... Getting > /wp-content/plugins/wptouch/resources/icons/elegant/Paper.png > Sleeping 9755 usec... Getting > /wp-includes/js/jquery/jquery-migrate.min.js?ver=1.2.1 > Sleeping 3798 usec... Getting > /wp-content/plugins/wptouch/themes/foundation/default/style.css?ver=3.7.5.3 > Sleeping 25 usec... Getting > /wp-content/plugins/wptouch/themes/bauhaus/default/style.css?ver=3.7.5.3 > Sleeping 9679 usec... Getting > /wp-content/plugins/wptouch/themes/foundation/modules/pushit/pushit.css?ver=4.1.1 > Sleeping 6477 usec... Getting /wp-includes/js/jquery/jquery.js?ver=1.11.1 > Sleeping 166401 usec... Getting > /wp-content/plugins/wptouch/themes/foundation/modules/wptouch-icons/font/wptouch-icons.woff?64777116 > Notice: fwrite(): send of 180 bytes failed with errno=104 Connection reset by > peer in /home/walter/replay.php on line 18 > Yay! Connection was broken! > > As you can see, the server closes the connection prematurely. At > this moment the ^HANDLING_UNTRUSTED_INPUT audit log for "Paper.png" > has happened. > > Funny enough, with this example, I always get the error in > "Paper.png"! That file is very small by the way (336 bytes). Any GET > requests coming after "Paper.png" are not found in Apache's access_log > (although doing the extra requests does seem necessary for the problem > to happen!) Nothing in Apache’s error_logs. > > When I omit stuff from requests.json or shuffle it around, I don't get > the problem that easily. The problem likely depends on very specific > timing. But, the good thing is that the test case triggers the problem > on multiple Ubuntu 14.04 LTS machines. > > Various observations: 1) "EnableSendfile Off" reduces the problem > 75%. My test case stops reproducing, but production still has some > failures for iOS users. 2) "EnableMMAP Off” affects timing, > sometimes making the error switch between other requested files. 3) > "KeepAliveTimeout" is unrelated, the problem happens instantly into the > connection, so the problem is not related to normal connection teardown. > > What is a good next step? > > The packet capture is probably not generally useful, but there might be > a chance to reproduce the problem on a different machine; I can pack > up the essential files from my web root to go with my requests.json > if needed. > > Since I've got an easy way to reproduce on a test VM, I could also > experiment with debug mod_apparmor builds or different kernels, > although a bit of guidance in getting these running would be helpful > (my experience is mostly with FreeBSD). I run Ubuntu 14.04 LTS, > apache24+php56 from ondrej's PPA, and compiled mod_apparmor from > source. Trying the apparmor userland backports screwed up my system > and didn’t seem to help, but I could give it another try on the VM. Running mod_apparmor compiled from source is fine. One thing you can try to do is to reproduce the issue with debug logging enabled for mod_apparmor. You would add the following like so in your apache config: LogLevel apparmor:debug http://httpd.apache.org/docs/current/logs.html#permodule documents the feature. That said, the additional logging load may mess with the timing involved if the issue is the result of a race. And obviously, you wouldn't want to enable this in a production environment. Thanks again for digging into this. -- Steve Beattie <sbeat...@ubuntu.com> http://NxNW.org/~steve/
signature.asc
Description: Digital signature
-- AppArmor mailing list AppArmor@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/apparmor
