Dr0ptp4kt has uploaded a new change for review.
https://gerrit.wikimedia.org/r/133158
Change subject: More PhantomJS updates and documentation.
......................................................................
More PhantomJS updates and documentation.
Change-Id: I4009e64f0a31a6c1509a34eaeb1ec4025fd0ef3a
---
M maintenance/phantom/README
M maintenance/phantom/job.sh
M maintenance/phantom/zero_automated_tests.js
3 files changed, 20 insertions(+), 92 deletions(-)
git pull
ssh://gerrit.wikimedia.org:29418/mediawiki/extensions/ZeroRatedMobileAccess
refs/changes/58/133158/1
diff --git a/maintenance/phantom/README b/maintenance/phantom/README
index 597f9db..37624f8 100644
--- a/maintenance/phantom/README
+++ b/maintenance/phantom/README
@@ -3,13 +3,13 @@
The Wikipedia Zero automation test script is housed within the
ZeroRatedMobileAccess Gerrit/Git project under the directory
maintenance/phantom. The script is named zero_automated_tests.js and it has an
accompanying job.sh execution and mailer script and diskusage.sh mailer script.
-The following cron jobs run under user dr0ptp4kt on zero-bdd.wmflabs.org -
minute and hour expressed in UTC.
+The following cron jobs run under user dr0ptp4kt on zbdd in labs - minute and
hour expressed in UTC.
$ crontab -l
PATH=/bin:/usr/bin
-5 20 * * * /home/dr0ptp4kt/job.sh faster
+5 20 * * * /home/dr0ptp4kt/job.sh
50 23 * * * /home/dr0ptp4kt/diskusage.sh
Be sure to establish appropriate permissions:
@@ -17,7 +17,7 @@
$ chmod 740 job.sh
$ chmod 740 diskusage.sh
-See the note below about the /banners/ web directory. You'll need to get that,
and some other dependencies, setup to avoid unnecessary errors when job.sh runs.
+You'll need to get some dependencies setup to avoid unnecessary errors when
job.sh runs. See the following.
Note that security patches are applied to the server regularly and the server
is rebooted upon patch.
@@ -44,90 +44,26 @@
$ phantomjs
phantomjs>
-Now just enter Ctrl+C to get out of the phantomjs prompt.
+If you get an error about libfontconfig.so.1, just run the following:
-job.sh does some zipping of files. Install zip:
+$ sudo apt-get install libfontconfig
+$ phantomjs
+phantomjs>
+
+Now enter Ctrl+C to get out of the phantomjs prompt.
+
+job.sh does some zipping of files. Install zip and unzip:
$ sudo apt-get update
$ sudo apt-get install zip
+
+$ sudo apt-get update
+$ sudo apt-get install unzip
job.sh also does some HTML validation. Install tidy:
$ sudo apt-get update
$ sudo apt-get install tidy
-
-job.sh copies a small set of files on the webserver under the relative web
path /banners/ (real path of /var/www/banners/). So you'll want to set up the
Apache server.
-
-$ sudo apt-get update
-$ sudo apt-get install apache2
-
-Add a little security:
-
-$ sudo vim /etc/apache2/conf.d/security
-
-... Set the settings as follows, commenting out old stuff as appropriate ...
-
- ServerTokens Prod
- ServerSignature Off
- TraceEnable Off
-
-Restart Apache:
-
-$ sudo apachectl graceful
-
-Make the nice crawlers stay away...
-
-$ sudo vim /var/www/robots.txt
-
-... adding the following ...
-
- User-agent: *
- Disallow: /
-
-Now modify the default site's configuration to require login.
-
-$ sudo vim /etc/apache2/sites-enabled/000-default
-
-... add the following entry after the /var/www entry ...
-
- <Directory /var/www/banners>
- Options Indexes FollowSymLinks MultiViews
- AllowOverride AuthConfig
- Order allow,deny
- allow from all
- </Directory>
-
-This says to allow one to add an .htaccess file and, if wanted, .htaccess to
the directory. This isn't the highest performance solution, as Apache
apparently recurses to find the auth files, but it works.
-
-Now restart Apache again:
-
-$ sudo apachectl graceful
-
-Now actually create the banners directory and set up the login settings.
-
-$ cd /var/www
-$ sudo mkdir banners
-$ sudo chmod 750 banners
-$ sudo chown <jobs_id...dr0ptp4kt at the moment> banners
-$ sudo chgrp www-data banners
-$ cd banners
-$ vim .htaccess
-
-... add the following and save ...
-
- AuthUserFile /var/www/banners/.htpasswd
- AuthName standarddeviation
- AuthType Basic
- Require user <whatever web username will be required>
-
-$ chmod 640 .htaccess
-$ sudo chgrp www-data .htaccess
-$ htpasswd -c -s .htpasswd <whatever web username will be required>
-
-... enter the password you want the web user to use when prompted in the
browser ...
-
-$ chmod 640 .htpasswd
-$ sudo chgrp www-data .htpasswd
FINALLY!
@@ -183,9 +119,7 @@
A zip file showing the diffs for bad resources and links for non-cachebusted
pages, as well as improper ResourceLoader inclusion of Wikipedia Zero CSS and
JavaScript, as well as W3C HTML Tidy "gripes" are collected and emailed.
-An HTML file showing production banners for each carrier is produced and
emailed, in addition to being copied to the webserver.
-
-The full set of files is also zipped and copied to the webserver for retrieval
from the banners webpage.
+An HTML file showing production banners for each carrier is produced and
emailed.
WHAT diskusage.sh DOES:
diff --git a/maintenance/phantom/job.sh b/maintenance/phantom/job.sh
index 615a121..8159802 100755
--- a/maintenance/phantom/job.sh
+++ b/maintenance/phantom/job.sh
@@ -119,26 +119,20 @@
ERRORLINECOUNT=`expr $ERRORLINES - 33`
-zip report.zip $ERRORREPORTS fullreport.txt diff-badlinks-scheme-base.txt
diff-badsubresources-scheme-base.txt diff-badlinks-domain-base.txt
diff-badsubresources-domain-base.txt $LATEST/000.html $LATEST/*-topclip.png
tidy_gripes.txt latestreport
-zip banners.zip $LATEST/000.html $LATEST/*-topclip.png
+zip report.zip $ERRORREPORTS fullreport.txt diff-badlinks-scheme-base.txt
diff-badsubresources-scheme-base.txt diff-badlinks-domain-base.txt
diff-badsubresources-domain-base.txt $LATEST/000* $LATEST/*-topclip.png
tidy_gripes.txt latestreport
+zip banners.zip $LATEST/000* $LATEST/*-topclip.png
zip diffs_rl_tidy.zip diff-badlinks-scheme-base.txt
diff-badsubresources-scheme-base.txt diff-badlinks-domain-base.txt
diff-badsubresources-domain-base.txt badlinks-scheme-base.txt
badsubresources-scheme-base.txt errors.txt badlinks-domain-base.txt
badsubresources-domain-base.txt badzeroresourceload.txt tidy_gripes.txt
latestreport
mv $ERRORREPORTS fullreport.txt diff-badlinks-scheme-base.txt
diff-badsubresources-scheme-base.txt diff-badlinks-domain-base.txt
diff-badsubresources-domain-base.txt tidy_gripes.txt $LATEST/
cp latestreport $LATEST/
-
-zip all.zip $LATEST/*
-mv all.zip /var/www/banners/
-cp $LATEST/000.html /var/www/banners/index.html
-cp $LATEST/*-topclip.png /var/www/banners/
-
A=`date --rfc-3339=seconds`
-echo "See attached for diffs on bad links and subresources ($DIFFCOUNT lines
to inspect), plus superfluous ResourceLoader inclusion of Wikipedia Zero
scripts and styles ($RLLINECOUNT lines to inspect), plus Tidy complaints
($TIDYCOUNT lines to inspect) from the latest Wikipedia Zero automation test
run. The diff output: less than symbols refer to the old report, greater than
symbols the new report." | mailx -s "Wikipedia Zero Automated Test Bot Diff,
RL, Tidy for $A (Diff $DIFFCOUNT, RL $RLLINECOUNT, Tidy $TIDYCOUNT)" -a
diffs_rl_tidy.zip -c ab...@wikimedia.org yastrak...@wikimedia.org
+echo "See attached for diffs on bad links and subresources ($DIFFCOUNT lines
to inspect), plus superfluous ResourceLoader inclusion of Wikipedia Zero
scripts and styles ($RLLINECOUNT lines to inspect), plus Tidy complaints
($TIDYCOUNT lines to inspect) from the latest Wikipedia Zero automation test
run. The diff output: less than symbols refer to the old report, greater than
symbols the new report." | mailx -s "Wikipedia Zero Automated Test Bot Diff,
RL, Tidy for $A (Diff $DIFFCOUNT, RL $RLLINECOUNT, Tidy $TIDYCOUNT)" -a
diffs_rl_tidy.zip ab...@wikimedia.org
sleep 62
-echo "See attached for live banners from the latest Wikipedia Zero automation
test run." | mailx -s "Wikipedia Zero Automated Test Bot Banner Results for $A"
-a banners.zip -c ab...@wikimedia.org d...@wikimedia.org
+echo "See attached for live banners from the latest Wikipedia Zero automation
test run." | mailx -s "Wikipedia Zero Automated Test Bot Banner Results for $A"
-a banners.zip ab...@wikimedia.org
sleep 126
diff --git a/maintenance/phantom/zero_automated_tests.js
b/maintenance/phantom/zero_automated_tests.js
index 17de6bb..8b8f2a3 100644
--- a/maintenance/phantom/zero_automated_tests.js
+++ b/maintenance/phantom/zero_automated_tests.js
@@ -65,7 +65,7 @@
isoDateString + '/000.html',
'<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd";><html
xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en"><head><meta
http-equiv="Content-Type" content="text/html; charset=utf-8" /><style
type="text/css">table, th, td { border: 1px solid black;
}</style><title>Wikipedia Zero Automation Test Results for ' +
isoDateString + '</title></head><body><h1>Wikipedia Zero
Automation Test Results for ' +
- isoDateString + '</h1><p><a href="all.zip">Download Full
Report</a></p><div><table>' +
+ isoDateString + '</h1><div><table>' +
'<tr><th>Comment</th>' +
'<th>X-CS</th>' +
'<th>URL</th>' +
--
To view, visit https://gerrit.wikimedia.org/r/133158
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: I4009e64f0a31a6c1509a34eaeb1ec4025fd0ef3a
Gerrit-PatchSet: 1
Gerrit-Project: mediawiki/extensions/ZeroRatedMobileAccess
Gerrit-Branch: master
Gerrit-Owner: Dr0ptp4kt <ab...@wikimedia.org>
_______________________________________________
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
