[MediaWiki-commits] [Gerrit] Output correct Hue tunnel address on console - change (analytics/kraken)
Diederik has uploaded a new change for review.
https://gerrit.wikimedia.org/r/113013
Change subject: Output correct Hue tunnel address on console
..
Output correct Hue tunnel address on console
Change-Id: I3920274a992e9dbe16b594518bc162c2c5d1fff3
---
M bin/ktunnel
1 file changed, 12 insertions(+), 11 deletions(-)
git pull ssh://gerrit.wikimedia.org:29418/analytics/kraken
refs/changes/13/113013/1
diff --git a/bin/ktunnel b/bin/ktunnel
index d33a400..a897bdb 100755
--- a/bin/ktunnel
+++ b/bin/ktunnel
@@ -41,15 +41,15 @@
default_ssh_address = 'bast1001.wikimedia.org'
services = {
-'namenode': {'host': 'analytics1010.eqiad.wmnet', 'port': 50070},
-'datanode': {'host': 'analytics1011.eqiad.wmnet', 'port': 50075},
-'jobs': {'host': 'analytics1010.eqiad.wmnet', 'port': 8088 },
-'history': {'host': 'analytics1010.eqiad.wmnet', 'port': 19888},
-'hue': {'host': 'analytics1027.eqiad.wmnet', 'port': },
-'oozie':{'host': 'analytics1027.eqiad.wmnet', 'port': 11000},
-'logs': {'host': 'analytics1020.eqiad.wmnet', 'port': 8042},
-'j':{'host': 'analytics1010.eqiad.wmnet', 'port': 50030},
-'proxy':{'host': 'analytics1010.eqiad.wmnet', 'port': 8999},
+'namenode': {'host': 'analytics1010.eqiad.wmnet', 'port': 50070, 'https':
False},
+'datanode': {'host': 'analytics1011.eqiad.wmnet', 'port': 50075, 'https':
False},
+'jobs': {'host': 'analytics1010.eqiad.wmnet', 'port': 8088, 'https':
False },
+'history': {'host': 'analytics1010.eqiad.wmnet', 'port': 19888, 'https':
False},
+'hue': {'host': 'analytics1027.eqiad.wmnet', 'port': , 'https':
True },
+'oozie':{'host': 'analytics1027.eqiad.wmnet', 'port': 11000, 'https':
False},
+'logs': {'host': 'analytics1020.eqiad.wmnet', 'port': 8042, 'https':
False},
+'j':{'host': 'analytics1010.eqiad.wmnet', 'port': 50030, 'https':
False},
+'proxy':{'host': 'analytics1010.eqiad.wmnet', 'port': 8999, 'https':
False},
}
def tunnel(ssh_host, bind_port, dest_host, dest_port, background=False,
verbose=True):
@@ -82,6 +82,7 @@
print('Starting SOCKS proxy through %s:%i')
proxy(service['host'], service['port'], arguments['--background'],
arguments['--verbose'])
else:
-print(Tunneling to %s. Tunnel will be available at
http://localhost:%s\n; % (arguments['service'], service['port']))
+protocol = 'https' if service['https'] == True else 'http'
+print(Tunneling to %s. Tunnel will be available at
%s://localhost:%s\n % (arguments['service'], protocol, service['port']))
tunnel(ssh_host, service['port'], service['host'], service['port'],
arguments['--background'], arguments['--verbose'])
-
\ No newline at end of file
+
--
To view, visit https://gerrit.wikimedia.org/r/113013
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: I3920274a992e9dbe16b594518bc162c2c5d1fff3
Gerrit-PatchSet: 1
Gerrit-Project: analytics/kraken
Gerrit-Branch: master
Gerrit-Owner: Diederik dvanli...@wikimedia.org
___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] Enable ResourceManager API in hue.ini to enable JobHistory i... - change (operations...cdh4)
Diederik has uploaded a new change for review. https://gerrit.wikimedia.org/r/113021 Change subject: Enable ResourceManager API in hue.ini to enable JobHistory in Hue. .. Enable ResourceManager API in hue.ini to enable JobHistory in Hue. Change-Id: I3f7d244ec319d5d287c579a69237300a9379d835 --- M templates/hue/hue.ini.erb 1 file changed, 1 insertion(+), 1 deletion(-) git pull ssh://gerrit.wikimedia.org:29418/operations/puppet/cdh4 refs/changes/21/113021/1 diff --git a/templates/hue/hue.ini.erb b/templates/hue/hue.ini.erb index 2762ca7..971753f 100644 --- a/templates/hue/hue.ini.erb +++ b/templates/hue/hue.ini.erb @@ -300,7 +300,7 @@ ## hadoop_conf_dir=/etc/hadoop/conf # URL of the ResourceManager API - ## resourcemanager_api_url=http://%= @namenode_host %:8088 + resourcemanager_api_url=http://%= @namenode_host %:8088 # URL of the ProxyServer API proxy_api_url=http://%= @namenode_host %:8088 -- To view, visit https://gerrit.wikimedia.org/r/113021 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I3f7d244ec319d5d287c579a69237300a9379d835 Gerrit-PatchSet: 1 Gerrit-Project: operations/puppet/cdh4 Gerrit-Branch: master Gerrit-Owner: Diederik dvanli...@wikimedia.org ___ MediaWiki-commits mailing list MediaWiki-commits@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] Updating cdh4 submodule with hue jobbrowser fix - change (operations/puppet)
Diederik has uploaded a new change for review. https://gerrit.wikimedia.org/r/113030 Change subject: Updating cdh4 submodule with hue jobbrowser fix .. Updating cdh4 submodule with hue jobbrowser fix Change-Id: I2966d05d873f79852d2da748588600e1347e518a --- M modules/cdh4 1 file changed, 0 insertions(+), 0 deletions(-) git pull ssh://gerrit.wikimedia.org:29418/operations/puppet refs/changes/30/113030/1 diff --git a/modules/cdh4 b/modules/cdh4 index 7095c8b..d7a2589 16 --- a/modules/cdh4 +++ b/modules/cdh4 -Subproject commit 7095c8b51e16f14f9b88166da8a5f90a5f887063 +Subproject commit d7a258934473f9753cb275c8840f3bbcb8e9a7dc -- To view, visit https://gerrit.wikimedia.org/r/113030 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I2966d05d873f79852d2da748588600e1347e518a Gerrit-PatchSet: 1 Gerrit-Project: operations/puppet Gerrit-Branch: production Gerrit-Owner: Diederik dvanli...@wikimedia.org ___ MediaWiki-commits mailing list MediaWiki-commits@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] Enabling TLSv1.1 and TLSv1.2 on misc Apache services - change (operations/puppet)
Diederik has uploaded a new change for review.
https://gerrit.wikimedia.org/r/111969
Change subject: Enabling TLSv1.1 and TLSv1.2 on misc Apache services
..
Enabling TLSv1.1 and TLSv1.2 on misc Apache services
see also RT 2517
Change-Id: Ia9058d2df5543ae0a6cb59737613127af412ec2e
---
M modules/apache/templates/vhost-proxy.conf.erb
M modules/bugzilla/templates/apache/bugzilla.wikimedia.org.erb
M modules/gitblit/templates/git.wikimedia.org.erb
M modules/ishmael/templates/apache/ishmael.wikimedia.org.erb
M templates/apache/generic_vhost.erb
M templates/apache/sites/etherpad_lite.wikimedia.org.erb
M templates/apache/sites/ganglia.wikimedia.org.erb
M templates/apache/sites/gerrit.wikimedia.org.erb
M templates/apache/sites/icinga.wikimedia.org.erb
M templates/apache/sites/planet-language.erb
M templates/apache/sites/planet.erb
M templates/apache/sites/racktables.wikimedia.org.erb
M templates/apache/sites/smokeping.wikimedia.org.erb
M templates/apache/sites/stats.wikimedia.org.erb
M templates/apache/sites/wikistats.erb
M templates/apache/sites/wikitech.wikimedia.org.erb
16 files changed, 19 insertions(+), 19 deletions(-)
git pull ssh://gerrit.wikimedia.org:29418/operations/puppet
refs/changes/69/111969/1
diff --git a/modules/apache/templates/vhost-proxy.conf.erb
b/modules/apache/templates/vhost-proxy.conf.erb
index 437ebe6..93d3546 100644
--- a/modules/apache/templates/vhost-proxy.conf.erb
+++ b/modules/apache/templates/vhost-proxy.conf.erb
@@ -2,7 +2,7 @@
VirtualHost %= vhost_name %:%= port %
% if ssl == true %
SSLEngine on
- SSLProtocol -ALL +SSLv3 +TLSv1
+ SSLProtocol -ALL +SSLv3 +TLSv1 +TLSv1.1 +TLSv1.2
SSLCipherSuite
AES128-GCM-SHA256:RC4-SHA:RC4-MD5:DES-CBC3-SHA:AES128-SHA:AES256-SHA
SSLHonorCipherOrder on
SSLCertificateFile %= ssl_path %/certs/pl.cert
diff --git a/modules/bugzilla/templates/apache/bugzilla.wikimedia.org.erb
b/modules/bugzilla/templates/apache/bugzilla.wikimedia.org.erb
index 43e43ae..a8d3930 100644
--- a/modules/bugzilla/templates/apache/bugzilla.wikimedia.org.erb
+++ b/modules/bugzilla/templates/apache/bugzilla.wikimedia.org.erb
@@ -49,7 +49,7 @@
ServerName bugs.wikimedia.org
Redirect permanent / https://%=
scope.lookupvar('bugzilla::apache::svc_name') %/
SSLEngine On
- SSLProtocol -ALL +SSLv3 +TLSv1
+ SSLProtocol -ALL +SSLv3 +TLSv1 +TLSv1.1 +TLSv1.2
SSLCipherSuite
AES128-GCM-SHA256:RC4-SHA:RC4-MD5:DES-CBC3-SHA:AES128-SHA:AES256-SHA
SSLHonorCipherOrder on
SSLCertificateFile /etc/ssl/certs/%=
scope.lookupvar('bugzilla::apache::svc_name') %.pem
@@ -65,7 +65,7 @@
ServerName %= scope.lookupvar('bugzilla::apache::svc_name') %
DocumentRoot %= scope.lookupvar('bugzilla::apache::docroot') %
SSLEngine On
- SSLProtocol -ALL +SSLv3 +TLSv1
+ SSLProtocol -ALL +SSLv3 +TLSv1 +TLSv1.1 +TLSv1.2
SSLCipherSuite
AES128-GCM-SHA256:RC4-SHA:RC4-MD5:DES-CBC3-SHA:AES128-SHA:AES256-SHA
SSLHonorCipherOrder on
SSLCertificateFile /etc/ssl/certs/%=
scope.lookupvar('bugzilla::apache::svc_name') %.pem
@@ -178,7 +178,7 @@
ServerName %= scope.lookupvar('bugzilla::apache::attach_svc_name') %
DocumentRoot %= scope.lookupvar('bugzilla::apache::docroot') %
SSLEngine On
- SSLProtocol -ALL +SSLv3 +TLSv1
+ SSLProtocol -ALL +SSLv3 +TLSv1 +TLSv1.1 +TLSv1.2
SSLCipherSuite
AES128-GCM-SHA256:RC4-SHA:RC4-MD5:DES-CBC3-SHA:AES128-SHA:AES256-SHA
SSLHonorCipherOrder on
SSLCertificateFile /etc/ssl/certs/%=
scope.lookupvar('bugzilla::apache::attach_svc_name') %.pem
diff --git a/modules/gitblit/templates/git.wikimedia.org.erb
b/modules/gitblit/templates/git.wikimedia.org.erb
index db76fe5..098677a 100644
--- a/modules/gitblit/templates/git.wikimedia.org.erb
+++ b/modules/gitblit/templates/git.wikimedia.org.erb
@@ -44,7 +44,7 @@
ServerName %= host %
SSLEngine on
- SSLProtocol -ALL +SSLv3 +TLSv1
+ SSLProtocol -ALL +SSLv3 +TLSv1 +TLSv1.1 +TLSv1.2
SSLCipherSuite
AES128-GCM-SHA256:RC4-SHA:RC4-MD5:DES-CBC3-SHA:AES128-SHA:AES256-SHA
SSLHonorCipherOrder on
SSLCertificateFile /etc/ssl/certs/%= ssl_cert %.pem
diff --git a/modules/ishmael/templates/apache/ishmael.wikimedia.org.erb
b/modules/ishmael/templates/apache/ishmael.wikimedia.org.erb
index 795736a..7bdc5d0 100644
--- a/modules/ishmael/templates/apache/ishmael.wikimedia.org.erb
+++ b/modules/ishmael/templates/apache/ishmael.wikimedia.org.erb
@@ -7,7 +7,7 @@
VirtualHost *:443
ServerName %= @site_name %
SSLEngine On
- SSLProtocol -ALL +SSLv3 +TLSv1
+ SSLProtocol -ALL +SSLv3 +TLSv1 +TLSv1.1 +TLSv1.2
SSLCipherSuite
AES128-GCM-SHA256:RC4-SHA:RC4-MD5:DES-CBC3-SHA:AES128-SHA:AES256-SHA
SSLHonorCipherOrder on
SSLCertificateFile /etc/ssl/private/star.wikimedia.org.pem
diff --git
[MediaWiki-commits] [Gerrit] [WIP] MySQL scripts to generate Kanban chart for RT - change (operations/puppet)
Diederik has uploaded a new change for review.
https://gerrit.wikimedia.org/r/52
Change subject: [WIP] MySQL scripts to generate Kanban chart for RT
..
[WIP] MySQL scripts to generate Kanban chart for RT
Do not merge, see also RT 6761
Change-Id: Ie3a5f19c25d2828e64c8c6383525ab3e9dd4bfe7
---
M manifests/misc/statistics.pp
A templates/misc/rt-cumulative-flow-dataset.erb
2 files changed, 143 insertions(+), 0 deletions(-)
git pull ssh://gerrit.wikimedia.org:29418/operations/puppet
refs/changes/52/52/1
diff --git a/manifests/misc/statistics.pp b/manifests/misc/statistics.pp
index 956d136..eda49bc 100644
--- a/manifests/misc/statistics.pp
+++ b/manifests/misc/statistics.pp
@@ -798,6 +798,38 @@
}
}
+# Class: misc::statistics::cron_rt_cdf
+#
+# Sets up daily cron jobs to run a script which
+# generates a cumulative flow dataset used in
+# Kanban to measure work in progress and cycle time.
+# See also https://rt.wikimedia.org/Ticket/Display.html?id=6761
+class misc::statistics::cron_rt_cdf {
+include passwords::mysql::rt_readonly
+
+$script = '/usr/local/bin/rt-cdf.sh'
+
+$db_host = ''
+$db_user = $passwords::mysql::rt_readonly::user
+$db_pass = $passwords::mysql::rt_readonly::pass
+$db_database = 'rt4'
+
+file { $script:
+mode= '0755',
+content = template('misc/rt-cumulative-flow-dataset.erb'),
+}
+
+# Create a daily cron job to run the blog script
+# This requires that the $misc::statistics::user::username
+# user is installed on the source host.
+cron { 'blog_pageviews_email':
+command = $script,
+user= $misc::statistics::user::username,
+hour= 1,
+minute = 0,
+}
+}
+
# Class: misc::statistics::limn::mobile_data_sync
#
diff --git a/templates/misc/rt-cumulative-flow-dataset.erb
b/templates/misc/rt-cumulative-flow-dataset.erb
new file mode 100644
index 000..9d312b7
--- /dev/null
+++ b/templates/misc/rt-cumulative-flow-dataset.erb
@@ -0,0 +1,111 @@
+#!/bin/bash
+
+# Written by Diederik van Liere (dvanli...@wikimedia.org) for WMF
+# These MySQL queries jointly create a cumulative flow dataset
+# that can be used to visualize a chart called the cumulative flow
+# diagram (cdf). A cdf is a key chart in Kanban like operations
+# and it visualizes two key metrics:
+# 1) Work-in-Progress (WIP) -- for any given day, how many tickets
+were worked on?
+# 2) Mean Cycle Time -- how much does it take between an RT ticket
+gets submitted and resolved.
+#
+# RT does not have metrics regarding the 'size' of the task, so each
+# ticket is weighted equally.
+
+DATE=`date --date=yesterday +%Y-%m-%d`;
+
+echo Generating the cumulative flow dataset for $DATE UTC
+
+#RT was taken into production around August 1st, 2010
+
+mysql log -h %= db_host % -p%= db_pass % -u%= db_user % %= db_database
%= -e
+
+-- The drop statement is a bit stupid but MySQL does not have a
+-- CREATE IF NOT EXISTS for procedures
+DROP PROCEDURE IF EXISTS fill_calendar;
+
+CREATE TABLE IF NOT EXISTS calendar (
+id MEDIUMINT NOT NULL AUTO_INCREMENT,
+datefield DATE,
+new MEDIUMINT,
+open MEDIUMINT,
+resolved MEDIUMINT,
+PRIMARY KEY (id)
+);
+
+DELIMITER |
+
+CREATE PROCEDURE fill_calendar()
+BEGIN
+DECLARE start_date DATE;
+DECLARE end_date DATE;
+SELECT COALESCE((SELECT datefield FROM calendar WHERE id = 0), (SELECT
STR_TO_DATE('2010-01-08', '%Y-%m-%d'))) INTO start_date;
+SELECT COALESCE((SELECT datefield FROM calendar ORDER BY id DESC LIMIT 1),
(CURDATE() - INTERVAL 1 DAY)) INTO end_date;
+WHILE start_date end_date DO
+INSERT INTO calendar (datefield) VALUES(start_date);
+SET start_date = ADDDATE(start_date, INTERVAL 1 DAY);
+END WHILE;
+END |
+DELIMITER ;
+
+CALL fill_calendar();
+
+
+-- Days that the ticket was marked new
+UPDATE
+calendar
+JOIN (SELECT calendar.datefield AS date_new,
SUM(IFNULL(IF(DATE(Tickets.Created) '1970-01-01', 1, 0), 0)) AS new
+FROM
+Tickets
+RIGHT JOIN
+calendar ON DATE(calendar.datefield) BETWEEN Tickets.Created AND
IF(DATE(Tickets.Started) = '1970-01-01', CURDATE(), Tickets.Started)
+GROUP BY
+date_new)
+AS
+new_query
+ON
+calendar.datefield = new_query.date_new
+SET
+calendar.new = new_query.new;
+
+-- Days that the ticket was marked open
+UPDATE
+calendar
+JOIN (SELECT calendar.datefield AS date_open,
SUM(IFNULL(IF(DATE(Tickets.Started) '1970-01-01', 1, 0), 0)) AS open
+FROM
+Tickets
+RIGHT JOIN
+calendar
+ON
+DATE(calendar.datefield) BETWEEN Tickets.Started AND
IF(DATE(Tickets.Resolved) = '1970-01-01', CURDATE(), Tickets.Resolved)
+GROUP BY
+date_open)
+AS
+open_query
+ON
+calendar.datefield = open_query.date_open
+SET
+calendar.open = open_query.open;
+
+-- Days that the ticket was marked resolved
+UPDATE
+calendar
[MediaWiki-commits] [Gerrit] Added test to check which version of Ubuntu is running. - change (labs/migration-assistant)
Diederik has uploaded a new change for review.
https://gerrit.wikimedia.org/r/110216
Change subject: Added test to check which version of Ubuntu is running.
..
Added test to check which version of Ubuntu is running.
Change-Id: Idbb231d9954a4278bbac6e6c6532f02f083df37a
---
M labs-migration-assistant/fabfile.py
1 file changed, 21 insertions(+), 1 deletion(-)
git pull ssh://gerrit.wikimedia.org:29418/labs/migration-assistant
refs/changes/16/110216/1
diff --git a/labs-migration-assistant/fabfile.py
b/labs-migration-assistant/fabfile.py
index a0eda72..308f8a8 100644
--- a/labs-migration-assistant/fabfile.py
+++ b/labs-migration-assistant/fabfile.py
@@ -68,7 +68,7 @@
def __init__(self, name, project, datacenter):
self.tasks = ['detect_self_puppetmaster', 'detect_last_puppet_run',
'detect_shared_storage_for_projects',
'detect_shared_storage_for_home',
- 'detect_databases', 'detect_mediawiki']
+ 'detect_databases', 'detect_mediawiki', 'check_ubuntu']
self.name = name
self.project = project
self.datacenter = datacenter
@@ -347,6 +347,25 @@
@task
@check_connection
+def check_ubuntu():
+min_ubuntu_version = 11.04
+result = run('lsb_release -a')
+version = run('lsb_release -r -s')
+try:
+version = float(version)
+if min_ubuntu_version version:
+logging.warning('You are running an outdated version of Ubuntu,
please upgrade. [WARNING]')
+env.labinstances[env.host_string].check_ubuntu = 'WARNING'
+else:
+logging.info('You are running an up-to-date version of Ubuntu.
[OK]')
+env.labinstances[env.host_string].check_ubuntu = 'OK'
+except ValueError:
+logging.warning('Was not able to determine whether your Ubuntu
installation is up to date. [WARNING')
+env.labinstances[env.host_string].check_ubuntu = 'WARNING'
+
+
+@task
+@check_connection
def test():
with settings(warn_only=True):
execute(detect_databases)
@@ -355,6 +374,7 @@
execute(detect_last_puppet_run)
execute(detect_shared_storage_for_projects)
execute(detect_shared_storage_for_home)
+execute(check_ubuntu)
output_summary()
'''
TODO: There is a weird bug in probably my code where if you call test then
it will keep looping over all
--
To view, visit https://gerrit.wikimedia.org/r/110216
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: Idbb231d9954a4278bbac6e6c6532f02f083df37a
Gerrit-PatchSet: 1
Gerrit-Project: labs/migration-assistant
Gerrit-Branch: master
Gerrit-Owner: Diederik dvanli...@wikimedia.org
___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] Added test to check which version of Ubuntu is running. - change (labs/migration-assistant)
Diederik has submitted this change and it was merged.
Change subject: Added test to check which version of Ubuntu is running.
..
Added test to check which version of Ubuntu is running.
Change-Id: Idbb231d9954a4278bbac6e6c6532f02f083df37a
---
M labs-migration-assistant/fabfile.py
1 file changed, 21 insertions(+), 1 deletion(-)
Approvals:
Diederik: Verified; Looks good to me, approved
jenkins-bot: Verified
diff --git a/labs-migration-assistant/fabfile.py
b/labs-migration-assistant/fabfile.py
index a0eda72..308f8a8 100644
--- a/labs-migration-assistant/fabfile.py
+++ b/labs-migration-assistant/fabfile.py
@@ -68,7 +68,7 @@
def __init__(self, name, project, datacenter):
self.tasks = ['detect_self_puppetmaster', 'detect_last_puppet_run',
'detect_shared_storage_for_projects',
'detect_shared_storage_for_home',
- 'detect_databases', 'detect_mediawiki']
+ 'detect_databases', 'detect_mediawiki', 'check_ubuntu']
self.name = name
self.project = project
self.datacenter = datacenter
@@ -347,6 +347,25 @@
@task
@check_connection
+def check_ubuntu():
+min_ubuntu_version = 11.04
+result = run('lsb_release -a')
+version = run('lsb_release -r -s')
+try:
+version = float(version)
+if min_ubuntu_version version:
+logging.warning('You are running an outdated version of Ubuntu,
please upgrade. [WARNING]')
+env.labinstances[env.host_string].check_ubuntu = 'WARNING'
+else:
+logging.info('You are running an up-to-date version of Ubuntu.
[OK]')
+env.labinstances[env.host_string].check_ubuntu = 'OK'
+except ValueError:
+logging.warning('Was not able to determine whether your Ubuntu
installation is up to date. [WARNING')
+env.labinstances[env.host_string].check_ubuntu = 'WARNING'
+
+
+@task
+@check_connection
def test():
with settings(warn_only=True):
execute(detect_databases)
@@ -355,6 +374,7 @@
execute(detect_last_puppet_run)
execute(detect_shared_storage_for_projects)
execute(detect_shared_storage_for_home)
+execute(check_ubuntu)
output_summary()
'''
TODO: There is a weird bug in probably my code where if you call test then
it will keep looping over all
--
To view, visit https://gerrit.wikimedia.org/r/110216
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: merged
Gerrit-Change-Id: Idbb231d9954a4278bbac6e6c6532f02f083df37a
Gerrit-PatchSet: 1
Gerrit-Project: labs/migration-assistant
Gerrit-Branch: master
Gerrit-Owner: Diederik dvanli...@wikimedia.org
Gerrit-Reviewer: Diederik dvanli...@wikimedia.org
Gerrit-Reviewer: jenkins-bot
___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] Fix ignored hosts not set. - change (labs/migration-assistant)
Diederik has submitted this change and it was merged.
Change subject: Fix ignored hosts not set.
..
Fix ignored hosts not set.
Change-Id: I627fc4d5c4000ce8d5debb73db964a3c68407bf3
---
M labs-migration-assistant/fabfile.py
1 file changed, 2 insertions(+), 0 deletions(-)
Approvals:
Diederik: Verified; Looks good to me, approved
jenkins-bot: Verified
diff --git a/labs-migration-assistant/fabfile.py
b/labs-migration-assistant/fabfile.py
index 51d0536..a0eda72 100644
--- a/labs-migration-assistant/fabfile.py
+++ b/labs-migration-assistant/fabfile.py
@@ -57,6 +57,8 @@
env.disable_known_hosts = True
env.reject_unknown_hosts = False
env.gateway = 'bastion.wmflabs.org'
+if not 'ignored_hosts' in env:
+env.ignored_hosts = ''
env.ignored_hosts = env.ignored_hosts.split(';')
env.key_filename = os.path.join(os.path.expanduser('~'), '.ssh/id_rsa')
--
To view, visit https://gerrit.wikimedia.org/r/108721
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: merged
Gerrit-Change-Id: I627fc4d5c4000ce8d5debb73db964a3c68407bf3
Gerrit-PatchSet: 1
Gerrit-Project: labs/migration-assistant
Gerrit-Branch: master
Gerrit-Owner: Diederik dvanli...@wikimedia.org
Gerrit-Reviewer: Diederik dvanli...@wikimedia.org
Gerrit-Reviewer: jenkins-bot jenkins-...@wikimedia.org
___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] Fix ignored hosts not set. - change (labs/migration-assistant)
Diederik has uploaded a new change for review.
https://gerrit.wikimedia.org/r/108721
Change subject: Fix ignored hosts not set.
..
Fix ignored hosts not set.
Change-Id: I627fc4d5c4000ce8d5debb73db964a3c68407bf3
---
M labs-migration-assistant/fabfile.py
1 file changed, 2 insertions(+), 0 deletions(-)
git pull ssh://gerrit.wikimedia.org:29418/labs/migration-assistant
refs/changes/21/108721/1
diff --git a/labs-migration-assistant/fabfile.py
b/labs-migration-assistant/fabfile.py
index 51d0536..a0eda72 100644
--- a/labs-migration-assistant/fabfile.py
+++ b/labs-migration-assistant/fabfile.py
@@ -57,6 +57,8 @@
env.disable_known_hosts = True
env.reject_unknown_hosts = False
env.gateway = 'bastion.wmflabs.org'
+if not 'ignored_hosts' in env:
+env.ignored_hosts = ''
env.ignored_hosts = env.ignored_hosts.split(';')
env.key_filename = os.path.join(os.path.expanduser('~'), '.ssh/id_rsa')
--
To view, visit https://gerrit.wikimedia.org/r/108721
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: I627fc4d5c4000ce8d5debb73db964a3c68407bf3
Gerrit-PatchSet: 1
Gerrit-Project: labs/migration-assistant
Gerrit-Branch: master
Gerrit-Owner: Diederik dvanli...@wikimedia.org
___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] First release. - change (labs/migration-assistant)
Diederik has submitted this change and it was merged.
Change subject: First release.
..
First release.
- Added check for Mediawiki
- Added check for MySQL
- Update pip to at least version 1.2
- Puppet outdated if last run was older than 24 hours
- Only check pmtpa instances
- Use warning for use of local storage instead of error
- Added decorator to check if host is reachable
Change-Id: I859fcff0d72c284c107c4183c697a82195fc8680
---
M README.md
M labs-migration-assistant/fabfile.py
M requirements.txt
3 files changed, 260 insertions(+), 173 deletions(-)
Approvals:
Diederik: Looks good to me, approved
diff --git a/README.md b/README.md
index 96a21bc..77d486f 100644
--- a/README.md
+++ b/README.md
@@ -21,7 +21,7 @@
``` shell
cd labs-migration-assistant
-fab migrate_ready --set wiki_username=YOUR_WIKI_USERNAME
+fab test --set wiki_username=YOUR_WIKI_USERNAME
# use your username for Wikitech
```
@@ -31,7 +31,18 @@
on Python 2.5 and 2.6 as well and I do not foresee problems with Linux either.
Butbug reports are
always welcome and pull requests even more!
-Use a recent version of pip.
+Use a recent version of pip (1.2 or higher)
+
+## Troubleshooting
+
+Sometimes a lab instance might be totally unresponsive and SSH will not
time-out. If that happens then
+abort the script using CTRL+Z or CTRL+C and then invoke the labs migration
assistant as follows
+
+``` shell
+fab test --set wiki_username=YOUR_WIKI_USERNAME --exclude_hosts=host1,host2
+```
+You should only use the name of the instance, not the FQDN.
+
## Contributing
1. Fork it
diff --git a/labs-migration-assistant/fabfile.py
b/labs-migration-assistant/fabfile.py
index ff29abb..51d0536 100644
--- a/labs-migration-assistant/fabfile.py
+++ b/labs-migration-assistant/fabfile.py
@@ -50,12 +50,14 @@
# fabric settings
env.timeout = 10
env.forward_agent = True
+env.skip_bad_hosts = True
env.colorize_errors = True
env.abort_on_prompts = True
env.connection_attempts = 3
env.disable_known_hosts = True
env.reject_unknown_hosts = False
env.gateway = 'bastion.wmflabs.org'
+env.ignored_hosts = env.ignored_hosts.split(';')
env.key_filename = os.path.join(os.path.expanduser('~'), '.ssh/id_rsa')
@@ -63,13 +65,14 @@
def __init__(self, name, project, datacenter):
self.tasks = ['detect_self_puppetmaster', 'detect_last_puppet_run',
- 'detect_shared_storage_for_projects',
'detect_shared_storage_for_home']
+ 'detect_shared_storage_for_projects',
'detect_shared_storage_for_home',
+ 'detect_databases', 'detect_mediawiki']
self.name = name
self.project = project
self.datacenter = datacenter
self.connect = None
for task in self.tasks:
-setattr(self, task, True)
+setattr(self, task, 'FAIL')
def __str__(self):
return '%s.%s.wmflabs' % (self.name, self.datacenter)
@@ -78,32 +81,26 @@
return str(self)
def count_errors(self):
-return sum([1 for task in self.tasks if getattr(self, task) is False])
+return sum([1 for task in self.tasks if getattr(self, task) == 'FAIL'
or getattr(self, task) == 'WARNING'])
-def check_connection(func, *args, **kwargs):
-'''Check connection decorator'''
-@functools.wraps(func)
-def wrapper(*args, **kwargs):
-print kwargs
-print args
-output = func(*args, **kwargs)
-print output
-# if not labinstances[env.host_string].connect:
-# logging.info('Skipping task because during first task was not able
to connect to labsinstance.')
-return output
-return wrapper
+def load_lab_instances():
+'''
+Entry point for collecting all lab instances of specified user
+TODO: maybe filter for those instances where the user is admin?
+'''
+if 'debug' in env and env.debug is True:
+test_instance = LabInstance('limn0', 'analytics', 'pmtpa')
+labinstances = {'limn0.pmtpa.wmflabs': test_instance}
+else:
+labinstances = fetch_lab_instances()
+labinstances = parse_lab_instances(labinstances)
+if len(labinstances) == 0:
+logging.error(
+'I was either not able to parse the Wikitech page containing your
lab instances or you are not the administrator for any lab instance.')
-def logged(func):
-'''Logging decorator.'''
-@functools.wraps(func)
-def wrapper(*args, **kwargs):
-with hide('output'):
-output = func(*args, **kwargs)
-logging.info(output)
-return output
-return wrapper
+return labinstances
def parse_lab_instances(labinstances):
@@ -113,18 +110,21 @@
names = labinstance.get('printouts', {}).get('Instance Name', None)
project = labinstance.get('printouts', {}).get('Project', None)
dc = resource.split('.')[1]
-for
[MediaWiki-commits] [Gerrit] Incorporated feedback Andrew Bogott - change (labs/migration-assistant)
Diederik has uploaded a new change for review.
https://gerrit.wikimedia.org/r/108340
Change subject: Incorporated feedback Andrew Bogott
..
Incorporated feedback Andrew Bogott
- Added check for Mediawiki
- Added check for MySQL
- Update pip to at least version 1.2
- Puppet outdated if last run was older than 24 hours
- Only check pmtpa instances
- Use warning for use of local storage instead of error
Change-Id: I859fcff0d72c284c107c4183c697a82195fc8680
---
M labs-migration-assistant/fabfile.py
M requirements.txt
2 files changed, 60 insertions(+), 12 deletions(-)
git pull ssh://gerrit.wikimedia.org:29418/labs/migration-assistant
refs/changes/40/108340/1
diff --git a/labs-migration-assistant/fabfile.py
b/labs-migration-assistant/fabfile.py
index ff29abb..fbd6bf4 100644
--- a/labs-migration-assistant/fabfile.py
+++ b/labs-migration-assistant/fabfile.py
@@ -113,12 +113,13 @@
names = labinstance.get('printouts', {}).get('Instance Name', None)
project = labinstance.get('printouts', {}).get('Project', None)
dc = resource.split('.')[1]
-for name in names:
-if not name.startswith('tools'):
-# ignore all tool-labs instances as they are managed by
-# WMF.
-labinstance = LabInstance(name, project, dc)
-results[str(labinstance)] = labinstance
+if dc == 'pmtpa': #only analyse instances in Tampa
+for name in names:
+if not name.startswith('tools'):
+# ignore all tool-labs instances as they are managed by
+# WMF.
+labinstance = LabInstance(name, project, dc)
+results[str(labinstance)] = labinstance
return results
@@ -148,7 +149,7 @@
def detect_self_puppetmaster(labinstances):
try:
result = run(
-'grep ^server = virt0.wikimedia.org /etc/puppet/puppet.conf | wc
-l')
+'grep ^server = virt0.wikimedia.org|^server = .*localhost.*
/etc/puppet/puppet.conf | wc -l')
if result == '0':
logging.info('You are not using a self-hosted puppet master. [OK]')
else:
@@ -184,7 +185,7 @@
epoch = datetime.fromtimestamp(epoch)
now = datetime.now()
dt = now - epoch
-if dt.days 29:
+if dt.days 1:
logging.error(
'The last puppet run was %d days ago, please run puppet.
[FAIL]' % dt.days)
labinstances[env.host_string].detect_last_puppet_run = False
@@ -206,8 +207,7 @@
return
try:
-result = run('ls -l | wc -l')
-result = int(result)
+result = int(run('ls -l | wc -l'))
if result == 0:
logging.info(
'You seem not to be using your home folder for storing files.
[OK]')
@@ -238,10 +238,55 @@
logging.info(
'You seem to be using the shared storage space for your home
folder. [OK]')
else:
-logging.error(
-'You do not seem to be using the shared storage space for your
home folder. [FAIL]')
+logging.warn(
+'You do not seem to be using the shared storage space for your
home folder. Please make sure that you have backups. [WARNING]')
labinstances[
env.host_string].detect_shared_storage_for_home = False
+except SystemExit:
+logging.error('Could not connect to %s.' % env.host_string)
+
+@task
+def detect_databases():
+'''
+TODO: refactor this check into a decorator function, but Fabric is not
really cooperating
+'''
+if not labinstances[env.host_string].connect:
+if not labinstances[env.host_string].connect:
+logging.info(
+'Skipping task because during first task was not able to
connect to labsinstance.')
+return
+
+try:
+result = int(run('ls /etc/init.d | grep mysql | wc -l'))
+if result == 0:
+logging.info('You are not running a MySQL instance and hence you
do not have to make any backups. [OK]')
+else:
+result = int(run('mysql -e show databases | wc -l'))
+if result 1:
+# ignore information_schema database
+logging.warn('You are running a MySQL instance and hence you
should probably make a backup. [OK]')
+else:
+logging.info('You are running a MySQL database instance but it
does not seem to have any databases.')
+except SystemExit:
+logging.error('Could not connect to %s.' % env.host_string)
+
+@task
+def detect_mediawiki():
+'''
+TODO: refactor this check into a decorator function, but Fabric is not
really cooperating
+'''
+if not
[MediaWiki-commits] [Gerrit] Discovered that Semantic Mediawiki as an API that I can query! - change (labs/migration-assistant)
Diederik has submitted this change and it was merged.
Change subject: Discovered that Semantic Mediawiki as an API that I can query!
..
Discovered that Semantic Mediawiki as an API that I can query!
Deprecate screen scraping :) simplify code :) yay !
Change-Id: Ia038326b3c3780d6b0ced5e69a2e3e20d83216d3
---
M labs-migration-assistant/fabfile.py
M requirements.txt
2 files changed, 41 insertions(+), 58 deletions(-)
Approvals:
Diederik: Verified; Looks good to me, approved
diff --git a/labs-migration-assistant/fabfile.py
b/labs-migration-assistant/fabfile.py
index bd7fcd2..fe7b162 100644
--- a/labs-migration-assistant/fabfile.py
+++ b/labs-migration-assistant/fabfile.py
@@ -24,7 +24,6 @@
'''
import os
-import json
import logging
import functools
@@ -32,7 +31,6 @@
import requests
from fabric.api import *
-from bs4 import BeautifulSoup
from datetime import datetime
from ansistrm import ColorizingStreamHandler
@@ -56,6 +54,7 @@
env.abort_on_prompts = True
env.connection_attempts = 3
env.disable_known_hosts = True
+env.reject_unknown_hosts = False
env.gateway = 'bastion.wmflabs.org'
env.key_filename = os.path.join(os.path.expanduser('~'), '.ssh/id_rsa')
@@ -69,7 +68,6 @@
self.project = project
self.datacenter = datacenter
self.connect = None
-self.add_instance_to_fabric_env()
for task in self.tasks:
setattr(self, task, True)
@@ -78,9 +76,6 @@
def __repr__(self):
return str(self)
-
-def add_instance_to_fabric_env(self):
-env.hosts.append(str(self))
def count_errors(self):
return sum([1 for task in self.tasks if getattr(self, task) is False])
@@ -95,7 +90,7 @@
output = func(*args, **kwargs)
print output
# if not labinstances[env.host_string].connect:
-# logging.info('Skipping task because during first task was not
able to connect to labsinstance.')
+# logging.info('Skipping task because during first task was not able
to connect to labsinstance.')
return output
return wrapper
@@ -111,55 +106,42 @@
return wrapper
-def parse_lab_instances(html_raw):
-labinstances = {}
-soup = BeautifulSoup(html_raw)
-container = soup.find('div', {'id': 'mw-content-text'})
-if container:
-for tag in container.children:
-if tag.name == 'h2':
-project = tag.get('id')
-logging.info('Found project: %s' % project)
-elif tag.name == 'div':
-dc = tag.find('h3').text.strip()
-logging.info('Datacenter: %s' % dc)
-table = tag.find('table')
-'''
-TODO: the last tr from the table is not found, probably me
doing something dumb
-but that means that the final labs instance will not be
analyzed :(
-'''
-cells = table.find('tr')
-for cell in cells:
-if 'novainstancename' in cell.attrs['class']:
-logging.info('Instance: %s' % cell.text)
-labinstance = LabInstance(cell.text, project, dc)
-labinstances[str(labinstance)] = labinstance
-return labinstances
+def parse_lab_instances(labinstances):
+results = {}
+for group in labinstances:
+for resource, labinstance in group.get('results', {}).iteritems():
+names = labinstance.get('printouts', {}).get('Instance Name', None)
+project = labinstance.get('printouts', {}).get('Project', None)
+dc = resource.split('.')[1]
+for name in names:
+if not name.startswith('tools'):
+# ignore all tool-labs instances as they are managed by
+# WMF.
+labinstance = LabInstance(name, project, dc)
+results[str(labinstance)] = labinstance
+return results
def fetch_lab_instances():
-get_token_url =
'https://wikitech.wikimedia.org/w/api.php?action=loginlgname=%slgpassword=%sformat=json'
% (
-env.wiki_username, env.wiki_password)
-url = 'https://wikitech.wikimedia.org/wiki/Special:NovaResources'
+projects_url =
'https://wikitech.wikimedia.org/w/api.php?action=askquery=[[Member::User:%s]]format=json'
% env.wiki_username
verify = False
-result = ''
+labinstances = []
try:
-session = requests.Session()
-token_request = session.post(get_token_url, verify=verify)
-lgtoken = json.loads(token_request.text).get('login', {}).get('token')
-sessionid = json.loads(token_request.text).get(
-'login', {}).get('sessionid')
-confirm_token_url = '%slgtoken=%s' % (get_token_url, lgtoken)
-headers = {'sessionid': sessionid}
-confirm_request = session.post(
-confirm_token_url,
[MediaWiki-commits] [Gerrit] Discovered that Semantic Mediawiki as an API that I can query! - change (labs/migration-assistant)
Diederik has uploaded a new change for review.
https://gerrit.wikimedia.org/r/108063
Change subject: Discovered that Semantic Mediawiki as an API that I can query!
..
Discovered that Semantic Mediawiki as an API that I can query!
Deprecate screen scraping :) simplify code :) yay !
Change-Id: Ia038326b3c3780d6b0ced5e69a2e3e20d83216d3
---
M labs-migration-assistant/fabfile.py
M requirements.txt
2 files changed, 41 insertions(+), 58 deletions(-)
git pull ssh://gerrit.wikimedia.org:29418/labs/migration-assistant
refs/changes/63/108063/1
diff --git a/labs-migration-assistant/fabfile.py
b/labs-migration-assistant/fabfile.py
index bd7fcd2..fe7b162 100644
--- a/labs-migration-assistant/fabfile.py
+++ b/labs-migration-assistant/fabfile.py
@@ -24,7 +24,6 @@
'''
import os
-import json
import logging
import functools
@@ -32,7 +31,6 @@
import requests
from fabric.api import *
-from bs4 import BeautifulSoup
from datetime import datetime
from ansistrm import ColorizingStreamHandler
@@ -56,6 +54,7 @@
env.abort_on_prompts = True
env.connection_attempts = 3
env.disable_known_hosts = True
+env.reject_unknown_hosts = False
env.gateway = 'bastion.wmflabs.org'
env.key_filename = os.path.join(os.path.expanduser('~'), '.ssh/id_rsa')
@@ -69,7 +68,6 @@
self.project = project
self.datacenter = datacenter
self.connect = None
-self.add_instance_to_fabric_env()
for task in self.tasks:
setattr(self, task, True)
@@ -78,9 +76,6 @@
def __repr__(self):
return str(self)
-
-def add_instance_to_fabric_env(self):
-env.hosts.append(str(self))
def count_errors(self):
return sum([1 for task in self.tasks if getattr(self, task) is False])
@@ -95,7 +90,7 @@
output = func(*args, **kwargs)
print output
# if not labinstances[env.host_string].connect:
-# logging.info('Skipping task because during first task was not
able to connect to labsinstance.')
+# logging.info('Skipping task because during first task was not able
to connect to labsinstance.')
return output
return wrapper
@@ -111,55 +106,42 @@
return wrapper
-def parse_lab_instances(html_raw):
-labinstances = {}
-soup = BeautifulSoup(html_raw)
-container = soup.find('div', {'id': 'mw-content-text'})
-if container:
-for tag in container.children:
-if tag.name == 'h2':
-project = tag.get('id')
-logging.info('Found project: %s' % project)
-elif tag.name == 'div':
-dc = tag.find('h3').text.strip()
-logging.info('Datacenter: %s' % dc)
-table = tag.find('table')
-'''
-TODO: the last tr from the table is not found, probably me
doing something dumb
-but that means that the final labs instance will not be
analyzed :(
-'''
-cells = table.find('tr')
-for cell in cells:
-if 'novainstancename' in cell.attrs['class']:
-logging.info('Instance: %s' % cell.text)
-labinstance = LabInstance(cell.text, project, dc)
-labinstances[str(labinstance)] = labinstance
-return labinstances
+def parse_lab_instances(labinstances):
+results = {}
+for group in labinstances:
+for resource, labinstance in group.get('results', {}).iteritems():
+names = labinstance.get('printouts', {}).get('Instance Name', None)
+project = labinstance.get('printouts', {}).get('Project', None)
+dc = resource.split('.')[1]
+for name in names:
+if not name.startswith('tools'):
+# ignore all tool-labs instances as they are managed by
+# WMF.
+labinstance = LabInstance(name, project, dc)
+results[str(labinstance)] = labinstance
+return results
def fetch_lab_instances():
-get_token_url =
'https://wikitech.wikimedia.org/w/api.php?action=loginlgname=%slgpassword=%sformat=json'
% (
-env.wiki_username, env.wiki_password)
-url = 'https://wikitech.wikimedia.org/wiki/Special:NovaResources'
+projects_url =
'https://wikitech.wikimedia.org/w/api.php?action=askquery=[[Member::User:%s]]format=json'
% env.wiki_username
verify = False
-result = ''
+labinstances = []
try:
-session = requests.Session()
-token_request = session.post(get_token_url, verify=verify)
-lgtoken = json.loads(token_request.text).get('login', {}).get('token')
-sessionid = json.loads(token_request.text).get(
-'login', {}).get('sessionid')
-confirm_token_url = '%slgtoken=%s' % (get_token_url, lgtoken)
-headers = {'sessionid': sessionid}
-
[MediaWiki-commits] [Gerrit] Update documentation. - change (labs/migration-assistant)
Diederik has uploaded a new change for review. https://gerrit.wikimedia.org/r/108064 Change subject: Update documentation. .. Update documentation. Change-Id: Id39539d845c612449e9525042c2d80d9bfe26ee8 --- M README.md M labs-migration-assistant/fabfile.py 2 files changed, 3 insertions(+), 3 deletions(-) git pull ssh://gerrit.wikimedia.org:29418/labs/migration-assistant refs/changes/64/108064/1 diff --git a/README.md b/README.md index c2382f1..96a21bc 100644 --- a/README.md +++ b/README.md @@ -21,8 +21,8 @@ ``` shell cd labs-migration-assistant -fab migrate_ready --set wiki_username=YOUR_WIKI_USERNAME,wiki_password=YOUR_WIKI_PASSWORD -# use your credentials for Wikitech +fab migrate_ready --set wiki_username=YOUR_WIKI_USERNAME +# use your username for Wikitech ``` ## Requirements diff --git a/labs-migration-assistant/fabfile.py b/labs-migration-assistant/fabfile.py index fe7b162..ff29abb 100644 --- a/labs-migration-assistant/fabfile.py +++ b/labs-migration-assistant/fabfile.py @@ -298,7 +298,7 @@ def main(): print 'You should not run this script directly but instead call it as:' -print 'fab migrate_ready --set wiki_username=YOUR_WIKI_USERNAME,wiki_password=YOUR_WIKI_PASSWORD' +print 'fab migrate_ready --set wiki_username=YOUR_WIKI_USERNAME' print print 'The wiki that we are referring to is Wikitech.' print 'You might need to pass additional paramaters like your password for your SSH key, but this' -- To view, visit https://gerrit.wikimedia.org/r/108064 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: newchange Gerrit-Change-Id: Id39539d845c612449e9525042c2d80d9bfe26ee8 Gerrit-PatchSet: 1 Gerrit-Project: labs/migration-assistant Gerrit-Branch: master Gerrit-Owner: Diederik dvanli...@wikimedia.org ___ MediaWiki-commits mailing list MediaWiki-commits@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] Update documentation. - change (labs/migration-assistant)
Diederik has submitted this change and it was merged. Change subject: Update documentation. .. Update documentation. Change-Id: Id39539d845c612449e9525042c2d80d9bfe26ee8 --- M README.md M labs-migration-assistant/fabfile.py 2 files changed, 3 insertions(+), 3 deletions(-) Approvals: Diederik: Verified; Looks good to me, approved jenkins-bot: Verified diff --git a/README.md b/README.md index c2382f1..96a21bc 100644 --- a/README.md +++ b/README.md @@ -21,8 +21,8 @@ ``` shell cd labs-migration-assistant -fab migrate_ready --set wiki_username=YOUR_WIKI_USERNAME,wiki_password=YOUR_WIKI_PASSWORD -# use your credentials for Wikitech +fab migrate_ready --set wiki_username=YOUR_WIKI_USERNAME +# use your username for Wikitech ``` ## Requirements diff --git a/labs-migration-assistant/fabfile.py b/labs-migration-assistant/fabfile.py index fe7b162..ff29abb 100644 --- a/labs-migration-assistant/fabfile.py +++ b/labs-migration-assistant/fabfile.py @@ -298,7 +298,7 @@ def main(): print 'You should not run this script directly but instead call it as:' -print 'fab migrate_ready --set wiki_username=YOUR_WIKI_USERNAME,wiki_password=YOUR_WIKI_PASSWORD' +print 'fab migrate_ready --set wiki_username=YOUR_WIKI_USERNAME' print print 'The wiki that we are referring to is Wikitech.' print 'You might need to pass additional paramaters like your password for your SSH key, but this' -- To view, visit https://gerrit.wikimedia.org/r/108064 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: merged Gerrit-Change-Id: Id39539d845c612449e9525042c2d80d9bfe26ee8 Gerrit-PatchSet: 1 Gerrit-Project: labs/migration-assistant Gerrit-Branch: master Gerrit-Owner: Diederik dvanli...@wikimedia.org Gerrit-Reviewer: Diederik dvanli...@wikimedia.org Gerrit-Reviewer: jenkins-bot ___ MediaWiki-commits mailing list MediaWiki-commits@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] Disable SSL certificate verification. - change (labs/migration-assistant)
Diederik has uploaded a new change for review.
https://gerrit.wikimedia.org/r/107898
Change subject: Disable SSL certificate verification.
..
Disable SSL certificate verification.
Change-Id: Iaf7fe811a9403709f7e54b24cdc9f042c9e6e1d8
---
M labs-migration-assistant/fabfile.py
1 file changed, 26 insertions(+), 24 deletions(-)
git pull ssh://gerrit.wikimedia.org:29418/labs/migration-assistant
refs/changes/98/107898/1
diff --git a/labs-migration-assistant/fabfile.py
b/labs-migration-assistant/fabfile.py
index 77a86e7..51f1e91 100644
--- a/labs-migration-assistant/fabfile.py
+++ b/labs-migration-assistant/fabfile.py
@@ -111,45 +111,47 @@
labinstances = {}
soup = BeautifulSoup(html_raw)
container = soup.find('div', {'id': 'mw-content-text'})
- for tag in container.children:
- if tag.name == 'h2':
- project = tag.get('id')
- logging.info('Found project: %s' % project)
- elif tag.name == 'div':
- dc = tag.find('h3').text.strip()
- logging.info('Datacenter: %s' % dc)
- table = tag.find('table')
- '''
- TODO: the last tr from the table is not found,
probably me doing something dumb
- but that means that the final labs instance will not be
analyzed :(
- '''
- cells = table.find('tr')
- for cell in cells:
- if 'novainstancename' in cell.attrs['class']:
- logging.info('Instance: %s' % cell.text)
- labinstance = LabInstance(cell.text,
project, dc)
- labinstances[str(labinstance)] =
labinstance
+ if container:
+ for tag in container.children:
+ if tag.name == 'h2':
+ project = tag.get('id')
+ logging.info('Found project: %s' % project)
+ elif tag.name == 'div':
+ dc = tag.find('h3').text.strip()
+ logging.info('Datacenter: %s' % dc)
+ table = tag.find('table')
+ '''
+ TODO: the last tr from the table is not
found, probably me doing something dumb
+ but that means that the final labs instance
will not be analyzed :(
+ '''
+ cells = table.find('tr')
+ for cell in cells:
+ if 'novainstancename' in
cell.attrs['class']:
+ logging.info('Instance: %s' %
cell.text)
+ labinstance =
LabInstance(cell.text, project, dc)
+ labinstances[str(labinstance)]
= labinstance
return labinstances
def fetch_lab_instances():
get_token_url =
'https://wikitech.wikimedia.org/w/api.php?action=loginlgname=%slgpassword=%sformat=json'
% (env.wiki_username, env.wiki_password)
url = 'https://wikitech.wikimedia.org/wiki/Special:NovaResources'
-
- request = ''
+ verify = False
+ result = ''
try:
session = requests.Session()
- token_request = session.post(get_token_url)
+ token_request = session.post(get_token_url, verify=verify)
lgtoken = json.loads(token_request.text).get('login',
{}).get('token')
sessionid = json.loads(token_request.text).get('login',
{}).get('sessionid')
confirm_token_url = '%slgtoken=%s' % (get_token_url, lgtoken)
headers = {'sessionid' : sessionid}
- confirm_request = session.post(confirm_token_url,
headers=headers)
- request = session.get(url)
+ confirm_request = session.post(confirm_token_url,
headers=headers, verify=verify)
+ request = session.get(url, verify=verify)
+ result = request.text
except requests.exceptions.ConnectionError, e:
logging.error(e)
finally:
session.close()
- return request.text
+ return result
@task
def detect_self_puppetmaster(labinstances):
--
To view, visit https://gerrit.wikimedia.org/r/107898
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: Iaf7fe811a9403709f7e54b24cdc9f042c9e6e1d8
Gerrit-PatchSet: 1
Gerrit-Project: labs/migration-assistant
Gerrit-Branch: master
Gerrit-Owner: Diederik dvanli...@wikimedia.org
[MediaWiki-commits] [Gerrit] Make sure that some labs instances were found. - change (labs/migration-assistant)
Diederik has uploaded a new change for review.
https://gerrit.wikimedia.org/r/107899
Change subject: Make sure that some labs instances were found.
..
Make sure that some labs instances were found.
Change-Id: I731c8f5928dc6e23eb87f7e7ed9da32496cfb783
---
M labs-migration-assistant/fabfile.py
1 file changed, 5 insertions(+), 0 deletions(-)
git pull ssh://gerrit.wikimedia.org:29418/labs/migration-assistant
refs/changes/99/107899/1
diff --git a/labs-migration-assistant/fabfile.py
b/labs-migration-assistant/fabfile.py
index 51f1e91..afcc178 100644
--- a/labs-migration-assistant/fabfile.py
+++ b/labs-migration-assistant/fabfile.py
@@ -247,6 +247,10 @@
labinstances = parse_lab_instances(html_raw)
hosts = ['%s.%s.wmflabs' % (labinstance.name, labinstance.datacenter)
for labinstance in labinstances.values()]
+ if len(hosts) == 0:
+ logging.error('I was either not able to parse the Wikitech page
containing your lab instances or you are not the administrator for any lab
instance.')
+ exit(-1)
+
execute(detect_self_puppetmaster, hosts=hosts,
labinstances=labinstances)
execute(detect_last_puppet_run, hosts=hosts, labinstances=labinstances)
execute(detect_shared_storage_for_projects, hosts=hosts,
labinstances=labinstances)
@@ -271,6 +275,7 @@
print 'You should not run this script directly but instead call it as:'
print 'fab migrate_ready --set
wiki_username=YOUR_WIKI_USERNAME,wiki_password=YOUR_WIKI_PASSWORD'
print
+ print 'The wiki that we are referring to is Wikitech.'
print 'You might need to pass additional paramaters like your password
for your SSH key, but this'
print 'depends on the actual setup of your system.'
print 'For an overview of possible parameters run fab --help'
--
To view, visit https://gerrit.wikimedia.org/r/107899
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: I731c8f5928dc6e23eb87f7e7ed9da32496cfb783
Gerrit-PatchSet: 1
Gerrit-Project: labs/migration-assistant
Gerrit-Branch: master
Gerrit-Owner: Diederik dvanli...@wikimedia.org
___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] Make sure that some labs instances were found. - change (labs/migration-assistant)
Diederik has submitted this change and it was merged.
Change subject: Make sure that some labs instances were found.
..
Make sure that some labs instances were found.
Change-Id: I731c8f5928dc6e23eb87f7e7ed9da32496cfb783
---
M labs-migration-assistant/fabfile.py
1 file changed, 5 insertions(+), 0 deletions(-)
Approvals:
Diederik: Verified; Looks good to me, approved
diff --git a/labs-migration-assistant/fabfile.py
b/labs-migration-assistant/fabfile.py
index 51f1e91..afcc178 100644
--- a/labs-migration-assistant/fabfile.py
+++ b/labs-migration-assistant/fabfile.py
@@ -247,6 +247,10 @@
labinstances = parse_lab_instances(html_raw)
hosts = ['%s.%s.wmflabs' % (labinstance.name, labinstance.datacenter)
for labinstance in labinstances.values()]
+ if len(hosts) == 0:
+ logging.error('I was either not able to parse the Wikitech page
containing your lab instances or you are not the administrator for any lab
instance.')
+ exit(-1)
+
execute(detect_self_puppetmaster, hosts=hosts,
labinstances=labinstances)
execute(detect_last_puppet_run, hosts=hosts, labinstances=labinstances)
execute(detect_shared_storage_for_projects, hosts=hosts,
labinstances=labinstances)
@@ -271,6 +275,7 @@
print 'You should not run this script directly but instead call it as:'
print 'fab migrate_ready --set
wiki_username=YOUR_WIKI_USERNAME,wiki_password=YOUR_WIKI_PASSWORD'
print
+ print 'The wiki that we are referring to is Wikitech.'
print 'You might need to pass additional paramaters like your password
for your SSH key, but this'
print 'depends on the actual setup of your system.'
print 'For an overview of possible parameters run fab --help'
--
To view, visit https://gerrit.wikimedia.org/r/107899
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: merged
Gerrit-Change-Id: I731c8f5928dc6e23eb87f7e7ed9da32496cfb783
Gerrit-PatchSet: 1
Gerrit-Project: labs/migration-assistant
Gerrit-Branch: master
Gerrit-Owner: Diederik dvanli...@wikimedia.org
Gerrit-Reviewer: Diederik dvanli...@wikimedia.org
___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] Disable SSL certificate verification. - change (labs/migration-assistant)
Diederik has submitted this change and it was merged.
Change subject: Disable SSL certificate verification.
..
Disable SSL certificate verification.
Change-Id: Iaf7fe811a9403709f7e54b24cdc9f042c9e6e1d8
---
M labs-migration-assistant/fabfile.py
1 file changed, 26 insertions(+), 24 deletions(-)
Approvals:
Diederik: Verified; Looks good to me, approved
diff --git a/labs-migration-assistant/fabfile.py
b/labs-migration-assistant/fabfile.py
index 77a86e7..51f1e91 100644
--- a/labs-migration-assistant/fabfile.py
+++ b/labs-migration-assistant/fabfile.py
@@ -111,45 +111,47 @@
labinstances = {}
soup = BeautifulSoup(html_raw)
container = soup.find('div', {'id': 'mw-content-text'})
- for tag in container.children:
- if tag.name == 'h2':
- project = tag.get('id')
- logging.info('Found project: %s' % project)
- elif tag.name == 'div':
- dc = tag.find('h3').text.strip()
- logging.info('Datacenter: %s' % dc)
- table = tag.find('table')
- '''
- TODO: the last tr from the table is not found,
probably me doing something dumb
- but that means that the final labs instance will not be
analyzed :(
- '''
- cells = table.find('tr')
- for cell in cells:
- if 'novainstancename' in cell.attrs['class']:
- logging.info('Instance: %s' % cell.text)
- labinstance = LabInstance(cell.text,
project, dc)
- labinstances[str(labinstance)] =
labinstance
+ if container:
+ for tag in container.children:
+ if tag.name == 'h2':
+ project = tag.get('id')
+ logging.info('Found project: %s' % project)
+ elif tag.name == 'div':
+ dc = tag.find('h3').text.strip()
+ logging.info('Datacenter: %s' % dc)
+ table = tag.find('table')
+ '''
+ TODO: the last tr from the table is not
found, probably me doing something dumb
+ but that means that the final labs instance
will not be analyzed :(
+ '''
+ cells = table.find('tr')
+ for cell in cells:
+ if 'novainstancename' in
cell.attrs['class']:
+ logging.info('Instance: %s' %
cell.text)
+ labinstance =
LabInstance(cell.text, project, dc)
+ labinstances[str(labinstance)]
= labinstance
return labinstances
def fetch_lab_instances():
get_token_url =
'https://wikitech.wikimedia.org/w/api.php?action=loginlgname=%slgpassword=%sformat=json'
% (env.wiki_username, env.wiki_password)
url = 'https://wikitech.wikimedia.org/wiki/Special:NovaResources'
-
- request = ''
+ verify = False
+ result = ''
try:
session = requests.Session()
- token_request = session.post(get_token_url)
+ token_request = session.post(get_token_url, verify=verify)
lgtoken = json.loads(token_request.text).get('login',
{}).get('token')
sessionid = json.loads(token_request.text).get('login',
{}).get('sessionid')
confirm_token_url = '%slgtoken=%s' % (get_token_url, lgtoken)
headers = {'sessionid' : sessionid}
- confirm_request = session.post(confirm_token_url,
headers=headers)
- request = session.get(url)
+ confirm_request = session.post(confirm_token_url,
headers=headers, verify=verify)
+ request = session.get(url, verify=verify)
+ result = request.text
except requests.exceptions.ConnectionError, e:
logging.error(e)
finally:
session.close()
- return request.text
+ return result
@task
def detect_self_puppetmaster(labinstances):
--
To view, visit https://gerrit.wikimedia.org/r/107898
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: merged
Gerrit-Change-Id: Iaf7fe811a9403709f7e54b24cdc9f042c9e6e1d8
Gerrit-PatchSet: 1
Gerrit-Project: labs/migration-assistant
Gerrit-Branch: master
Gerrit-Owner: Diederik dvanli...@wikimedia.org
Gerrit-Reviewer: Diederik dvanli...@wikimedia.org
___
MediaWiki-commits
[MediaWiki-commits] [Gerrit] Minor tweaks. - change (labs/migration-assistant)
Diederik has uploaded a new change for review.
https://gerrit.wikimedia.org/r/107909
Change subject: Minor tweaks.
..
Minor tweaks.
Change-Id: I89d8a548d91e960d521a5a09a70dd329efaf484e
---
M labs-migration-assistant/fabfile.py
1 file changed, 6 insertions(+), 2 deletions(-)
git pull ssh://gerrit.wikimedia.org:29418/labs/migration-assistant
refs/changes/09/107909/1
diff --git a/labs-migration-assistant/fabfile.py
b/labs-migration-assistant/fabfile.py
index 011022d..a3b56c0 100644
--- a/labs-migration-assistant/fabfile.py
+++ b/labs-migration-assistant/fabfile.py
@@ -162,10 +162,11 @@
else:
logging.error('You are running your own self-hosted
puppet master. [FAIL]')
labinstances[env.host_string].detect_self_puppetmaster
= False
+ labinstances[env.host_string].connect = True
except SystemExit:
logging.error('Could not connect to %s.' % env.host_string)
labinstances[env.host_string].connect = False # we only to set
this once if we cannot connect to the instance.
- labinstances[env.host_string].connect = True
+
@task
def detect_last_puppet_run(labinstances):
@@ -258,7 +259,7 @@
for labsinstance in labinstances.values():
if not labsinstance.connect:
- logging.error('There were problems connecting
to instance %s, fix those problems first and then rerun this script.' %
labsinstance)
+ logging.error('There were problems connecting
to instance %s, please fix those problems first and then rerun this script.' %
labsinstance)
else:
logging.error('%s does not yet seem to be ready
for migration to eqiad. Please fix the %d identified problems.' %
(labsinstance, labsinstance.count_errors()))
logging.info('Summary of failed tasks:')
@@ -268,6 +269,9 @@
if not result:
logging.info('Problem %d: task
%s [FAIL]' % (problems, task))
problems +=1
+ if problems == 1:
+ logging.info('Congratulations! %s seems
ready for migration!' % labinstance)
+
run = logged(run)
--
To view, visit https://gerrit.wikimedia.org/r/107909
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: I89d8a548d91e960d521a5a09a70dd329efaf484e
Gerrit-PatchSet: 1
Gerrit-Project: labs/migration-assistant
Gerrit-Branch: master
Gerrit-Owner: Diederik dvanli...@wikimedia.org
___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] Use the appropriate bastion host. - change (labs/migration-assistant)
Diederik has uploaded a new change for review.
https://gerrit.wikimedia.org/r/107908
Change subject: Use the appropriate bastion host.
..
Use the appropriate bastion host.
Change-Id: Ib0d90c9d7eae0428e14b6a92997fe7c6abe8c97a
---
M labs-migration-assistant/fabfile.py
1 file changed, 1 insertion(+), 1 deletion(-)
git pull ssh://gerrit.wikimedia.org:29418/labs/migration-assistant
refs/changes/08/107908/1
diff --git a/labs-migration-assistant/fabfile.py
b/labs-migration-assistant/fabfile.py
index afcc178..011022d 100644
--- a/labs-migration-assistant/fabfile.py
+++ b/labs-migration-assistant/fabfile.py
@@ -57,7 +57,7 @@
env.abort_on_prompts = True
env.connection_attempts = 3
env.disable_known_hosts = True
-env.gateway = 'bast1001.wikimedia.org'
+env.gateway = 'bastion.wmflabs.org'
env.key_filename = os.path.join(os.path.expanduser('~'), '.ssh/id_rsa')
--
To view, visit https://gerrit.wikimedia.org/r/107908
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: Ib0d90c9d7eae0428e14b6a92997fe7c6abe8c97a
Gerrit-PatchSet: 1
Gerrit-Project: labs/migration-assistant
Gerrit-Branch: master
Gerrit-Owner: Diederik dvanli...@wikimedia.org
___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] Use the appropriate bastion host. - change (labs/migration-assistant)
Diederik has submitted this change and it was merged.
Change subject: Use the appropriate bastion host.
..
Use the appropriate bastion host.
Change-Id: Ib0d90c9d7eae0428e14b6a92997fe7c6abe8c97a
---
M labs-migration-assistant/fabfile.py
1 file changed, 1 insertion(+), 1 deletion(-)
Approvals:
Diederik: Verified; Looks good to me, approved
diff --git a/labs-migration-assistant/fabfile.py
b/labs-migration-assistant/fabfile.py
index afcc178..011022d 100644
--- a/labs-migration-assistant/fabfile.py
+++ b/labs-migration-assistant/fabfile.py
@@ -57,7 +57,7 @@
env.abort_on_prompts = True
env.connection_attempts = 3
env.disable_known_hosts = True
-env.gateway = 'bast1001.wikimedia.org'
+env.gateway = 'bastion.wmflabs.org'
env.key_filename = os.path.join(os.path.expanduser('~'), '.ssh/id_rsa')
--
To view, visit https://gerrit.wikimedia.org/r/107908
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: merged
Gerrit-Change-Id: Ib0d90c9d7eae0428e14b6a92997fe7c6abe8c97a
Gerrit-PatchSet: 1
Gerrit-Project: labs/migration-assistant
Gerrit-Branch: master
Gerrit-Owner: Diederik dvanli...@wikimedia.org
Gerrit-Reviewer: Diederik dvanli...@wikimedia.org
___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] basic pep8/flake8 configuration - change (labs/migration-assistant)
Diederik has submitted this change and it was merged. Change subject: basic pep8/flake8 configuration .. basic pep8/flake8 configuration Ignore errors: W191 indentation contains tabs E501 line too long (X 79 characters) Configuration is done in tox.ini which both flake8 and pep8 can read from. Change-Id: I3ad46aa869d060619fd62e9200e3b2ab315d3192 --- A tox.ini 1 file changed, 9 insertions(+), 0 deletions(-) Approvals: Diederik: Verified; Looks good to me, approved diff --git a/tox.ini b/tox.ini new file mode 100644 index 000..f1c0ad0 --- /dev/null +++ b/tox.ini @@ -0,0 +1,9 @@ +[flake8] +; W191 indentation contains tabs +; E501 line too long (X 79 characters) +ignore=W191,E501 + +[pep8] +; W191 indentation contains tabs +; E501 line too long (X 79 characters) +ignore=W191,E501 -- To view, visit https://gerrit.wikimedia.org/r/107912 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: merged Gerrit-Change-Id: I3ad46aa869d060619fd62e9200e3b2ab315d3192 Gerrit-PatchSet: 1 Gerrit-Project: labs/migration-assistant Gerrit-Branch: master Gerrit-Owner: Hashar has...@free.fr Gerrit-Reviewer: Diederik dvanli...@wikimedia.org ___ MediaWiki-commits mailing list MediaWiki-commits@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] Make code pep8 compliant. - change (labs/migration-assistant)
Diederik has uploaded a new change for review.
https://gerrit.wikimedia.org/r/107915
Change subject: Make code pep8 compliant.
..
Make code pep8 compliant.
Change-Id: I99e2cd4a0f444ec6b03474639072a482eb363255
---
M labs-migration-assistant/ansistrm.py
M labs-migration-assistant/fabfile.py
2 files changed, 241 insertions(+), 199 deletions(-)
git pull ssh://gerrit.wikimedia.org:29418/labs/migration-assistant
refs/changes/15/107915/1
diff --git a/labs-migration-assistant/ansistrm.py
b/labs-migration-assistant/ansistrm.py
index f53a558..4d4c2a3 100644
--- a/labs-migration-assistant/ansistrm.py
+++ b/labs-migration-assistant/ansistrm.py
@@ -4,7 +4,8 @@
import ctypes
import logging
import os
-
+
+
class ColorizingStreamHandler(logging.StreamHandler):
# color names to indices
color_map = {
@@ -17,8 +18,8 @@
'cyan': 6,
'white': 7,
}
-
-#levels to (background, foreground, bold/intense)
+
+# levels to (background, foreground, bold/intense)
if os.name == 'nt':
level_map = {
logging.DEBUG: (None, 'green', True),
@@ -37,12 +38,12 @@
}
csi = '\x1b['
reset = '\x1b[0m'
-
+
@property
def is_tty(self):
isatty = getattr(self.stream, 'isatty', None)
return isatty and isatty()
-
+
def emit(self, record):
try:
message = self.format(record)
@@ -57,14 +58,14 @@
raise
except:
self.handleError(record)
-
+
if os.name != 'nt':
def output_colorized(self, message):
self.stream.write(message)
else:
import re
ansi_esc = re.compile(r'\x1b\[((?:\d+)(?:;(?:\d+))*)m')
-
+
nt_color_map = {
0: 0x00,# black
1: 0x04,# red
@@ -75,7 +76,7 @@
6: 0x03,# cyan
7: 0x07,# white
}
-
+
def output_colorized(self, message):
parts = self.ansi_esc.split(message)
write = self.stream.write
@@ -83,7 +84,7 @@
fd = getattr(self.stream, 'fileno', None)
if fd is not None:
fd = fd()
-if fd in (1, 2): # stdout or stderr
+if fd in (1, 2): # stdout or stderr
h = ctypes.windll.kernel32.GetStdHandle(-10 - fd)
while parts:
text = parts.pop(0)
@@ -100,13 +101,14 @@
elif 30 = p = 37:
color |= self.nt_color_map[p - 30]
elif p == 1:
-color |= 0x08 # foreground intensity on
-elif p == 0: # reset to default color
+color |= 0x08 # foreground intensity on
+elif p == 0: # reset to default color
color = 0x07
else:
-pass # error condition ignored
-ctypes.windll.kernel32.SetConsoleTextAttribute(h,
color)
-
+pass # error condition ignored
+ctypes.windll.kernel32.SetConsoleTextAttribute(
+h, color)
+
def colorize(self, message, record):
if record.levelno in self.level_map:
bg, fg, bold = self.level_map[record.levelno]
@@ -121,7 +123,7 @@
message = ''.join((self.csi, ';'.join(params),
'm', message, self.reset))
return message
-
+
def format(self, record):
message = logging.StreamHandler.format(self, record)
if self.is_tty:
@@ -130,7 +132,8 @@
parts[0] = self.colorize(parts[0], record)
message = '\n'.join(parts)
return message
-
+
+
def main():
root = logging.getLogger()
root.setLevel(logging.DEBUG)
@@ -140,6 +143,6 @@
logging.warning('WARNING')
logging.error('ERROR')
logging.critical('CRITICAL')
-
+
if __name__ == '__main__':
-main()
\ No newline at end of file
+main()
diff --git a/labs-migration-assistant/fabfile.py
b/labs-migration-assistant/fabfile.py
index a3b56c0..82c0230 100644
--- a/labs-migration-assistant/fabfile.py
+++ b/labs-migration-assistant/fabfile.py
@@ -4,7 +4,7 @@
'''
labs-migration-assistant: a script to assess the readyness of
a Wikimedia Labs instance to be migrated from the Tampa dc to
-the Ashburn dc.
+the Ashburn dc.
Copyright (C) 2014 Diederik van Liere, Wikimedia Foundation
@@ -41,12 +41,13 @@
logger = logging.getLogger()
logger.setLevel(logging.INFO)
-formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s -
%(message)s')
+formatter = logging.Formatter(
+'%(asctime)s - %(name)s - %(levelname)s - %(message)s')
handler = ColorizingStreamHandler()
handler.setFormatter(formatter)
[MediaWiki-commits] [Gerrit] Make code pep8 compliant. - change (labs/migration-assistant)
Diederik has submitted this change and it was merged.
Change subject: Make code pep8 compliant.
..
Make code pep8 compliant.
Change-Id: I99e2cd4a0f444ec6b03474639072a482eb363255
---
M labs-migration-assistant/ansistrm.py
M labs-migration-assistant/fabfile.py
2 files changed, 241 insertions(+), 199 deletions(-)
Approvals:
Diederik: Verified; Looks good to me, approved
diff --git a/labs-migration-assistant/ansistrm.py
b/labs-migration-assistant/ansistrm.py
index f53a558..4d4c2a3 100644
--- a/labs-migration-assistant/ansistrm.py
+++ b/labs-migration-assistant/ansistrm.py
@@ -4,7 +4,8 @@
import ctypes
import logging
import os
-
+
+
class ColorizingStreamHandler(logging.StreamHandler):
# color names to indices
color_map = {
@@ -17,8 +18,8 @@
'cyan': 6,
'white': 7,
}
-
-#levels to (background, foreground, bold/intense)
+
+# levels to (background, foreground, bold/intense)
if os.name == 'nt':
level_map = {
logging.DEBUG: (None, 'green', True),
@@ -37,12 +38,12 @@
}
csi = '\x1b['
reset = '\x1b[0m'
-
+
@property
def is_tty(self):
isatty = getattr(self.stream, 'isatty', None)
return isatty and isatty()
-
+
def emit(self, record):
try:
message = self.format(record)
@@ -57,14 +58,14 @@
raise
except:
self.handleError(record)
-
+
if os.name != 'nt':
def output_colorized(self, message):
self.stream.write(message)
else:
import re
ansi_esc = re.compile(r'\x1b\[((?:\d+)(?:;(?:\d+))*)m')
-
+
nt_color_map = {
0: 0x00,# black
1: 0x04,# red
@@ -75,7 +76,7 @@
6: 0x03,# cyan
7: 0x07,# white
}
-
+
def output_colorized(self, message):
parts = self.ansi_esc.split(message)
write = self.stream.write
@@ -83,7 +84,7 @@
fd = getattr(self.stream, 'fileno', None)
if fd is not None:
fd = fd()
-if fd in (1, 2): # stdout or stderr
+if fd in (1, 2): # stdout or stderr
h = ctypes.windll.kernel32.GetStdHandle(-10 - fd)
while parts:
text = parts.pop(0)
@@ -100,13 +101,14 @@
elif 30 = p = 37:
color |= self.nt_color_map[p - 30]
elif p == 1:
-color |= 0x08 # foreground intensity on
-elif p == 0: # reset to default color
+color |= 0x08 # foreground intensity on
+elif p == 0: # reset to default color
color = 0x07
else:
-pass # error condition ignored
-ctypes.windll.kernel32.SetConsoleTextAttribute(h,
color)
-
+pass # error condition ignored
+ctypes.windll.kernel32.SetConsoleTextAttribute(
+h, color)
+
def colorize(self, message, record):
if record.levelno in self.level_map:
bg, fg, bold = self.level_map[record.levelno]
@@ -121,7 +123,7 @@
message = ''.join((self.csi, ';'.join(params),
'm', message, self.reset))
return message
-
+
def format(self, record):
message = logging.StreamHandler.format(self, record)
if self.is_tty:
@@ -130,7 +132,8 @@
parts[0] = self.colorize(parts[0], record)
message = '\n'.join(parts)
return message
-
+
+
def main():
root = logging.getLogger()
root.setLevel(logging.DEBUG)
@@ -140,6 +143,6 @@
logging.warning('WARNING')
logging.error('ERROR')
logging.critical('CRITICAL')
-
+
if __name__ == '__main__':
-main()
\ No newline at end of file
+main()
diff --git a/labs-migration-assistant/fabfile.py
b/labs-migration-assistant/fabfile.py
index a3b56c0..82c0230 100644
--- a/labs-migration-assistant/fabfile.py
+++ b/labs-migration-assistant/fabfile.py
@@ -4,7 +4,7 @@
'''
labs-migration-assistant: a script to assess the readyness of
a Wikimedia Labs instance to be migrated from the Tampa dc to
-the Ashburn dc.
+the Ashburn dc.
Copyright (C) 2014 Diederik van Liere, Wikimedia Foundation
@@ -41,12 +41,13 @@
logger = logging.getLogger()
logger.setLevel(logging.INFO)
-formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s -
%(message)s')
+formatter = logging.Formatter(
+'%(asctime)s - %(name)s - %(levelname)s - %(message)s')
handler = ColorizingStreamHandler()
handler.setFormatter(formatter)
logger.addHandler(handler)
'''
-Fabric logging messages have their
[MediaWiki-commits] [Gerrit] pep8/flake8 now exclude some standard directories - change (labs/migration-assistant)
Diederik has submitted this change and it was merged. Change subject: pep8/flake8 now exclude some standard directories .. pep8/flake8 now exclude some standard directories Change-Id: Id9343e3c6419fa0098f54ad16742dfefacb01e59 --- M tox.ini 1 file changed, 2 insertions(+), 0 deletions(-) Approvals: Diederik: Verified; Looks good to me, approved diff --git a/tox.ini b/tox.ini index f1c0ad0..f8ad589 100644 --- a/tox.ini +++ b/tox.ini @@ -1,9 +1,11 @@ [flake8] +exclude=.venv,.tox,dist,doc,build,*.egg ; W191 indentation contains tabs ; E501 line too long (X 79 characters) ignore=W191,E501 [pep8] +exclude=.venv,.tox,dist,doc,build,*.egg ; W191 indentation contains tabs ; E501 line too long (X 79 characters) ignore=W191,E501 -- To view, visit https://gerrit.wikimedia.org/r/107913 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: merged Gerrit-Change-Id: Id9343e3c6419fa0098f54ad16742dfefacb01e59 Gerrit-PatchSet: 2 Gerrit-Project: labs/migration-assistant Gerrit-Branch: master Gerrit-Owner: Hashar has...@free.fr Gerrit-Reviewer: Diederik dvanli...@wikimedia.org Gerrit-Reviewer: Hashar has...@free.fr Gerrit-Reviewer: jenkins-bot ___ MediaWiki-commits mailing list MediaWiki-commits@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] Fix pyflakes errors. Two remain because they are not real er... - change (labs/migration-assistant)
Diederik has submitted this change and it was merged. Change subject: Fix pyflakes errors. Two remain because they are not real errors. .. Fix pyflakes errors. Two remain because they are not real errors. Change-Id: If672638df066d717142ae09eb23f88cf0ab43a15 --- M labs-migration-assistant/fabfile.py 1 file changed, 0 insertions(+), 2 deletions(-) Approvals: Diederik: Looks good to me, approved diff --git a/labs-migration-assistant/fabfile.py b/labs-migration-assistant/fabfile.py index 82c0230..bd7fcd2 100644 --- a/labs-migration-assistant/fabfile.py +++ b/labs-migration-assistant/fabfile.py @@ -24,13 +24,11 @@ ''' import os -import sys import json import logging import functools import yaml -import fabric import requests from fabric.api import * -- To view, visit https://gerrit.wikimedia.org/r/107919 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: merged Gerrit-Change-Id: If672638df066d717142ae09eb23f88cf0ab43a15 Gerrit-PatchSet: 1 Gerrit-Project: labs/migration-assistant Gerrit-Branch: master Gerrit-Owner: Diederik dvanli...@wikimedia.org Gerrit-Reviewer: Diederik dvanli...@wikimedia.org Gerrit-Reviewer: jenkins-bot ___ MediaWiki-commits mailing list MediaWiki-commits@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] Fix pyflakes errors. Two remain because they are not real er... - change (labs/migration-assistant)
Diederik has uploaded a new change for review. https://gerrit.wikimedia.org/r/107919 Change subject: Fix pyflakes errors. Two remain because they are not real errors. .. Fix pyflakes errors. Two remain because they are not real errors. Change-Id: If672638df066d717142ae09eb23f88cf0ab43a15 --- M labs-migration-assistant/fabfile.py 1 file changed, 0 insertions(+), 2 deletions(-) git pull ssh://gerrit.wikimedia.org:29418/labs/migration-assistant refs/changes/19/107919/1 diff --git a/labs-migration-assistant/fabfile.py b/labs-migration-assistant/fabfile.py index 82c0230..bd7fcd2 100644 --- a/labs-migration-assistant/fabfile.py +++ b/labs-migration-assistant/fabfile.py @@ -24,13 +24,11 @@ ''' import os -import sys import json import logging import functools import yaml -import fabric import requests from fabric.api import * -- To view, visit https://gerrit.wikimedia.org/r/107919 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: newchange Gerrit-Change-Id: If672638df066d717142ae09eb23f88cf0ab43a15 Gerrit-PatchSet: 1 Gerrit-Project: labs/migration-assistant Gerrit-Branch: master Gerrit-Owner: Diederik dvanli...@wikimedia.org ___ MediaWiki-commits mailing list MediaWiki-commits@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] Remove p/p wrapping around label that shows in tooltip - change (mediawiki...SecurePoll)
Diederik has uploaded a new change for review. https://gerrit.wikimedia.org/r/90555 Change subject: Remove p/p wrapping around label that shows in tooltip .. Remove p/p wrapping around label that shows in tooltip Bug 30399 Change-Id: I45a7651803124e255105feb7d05a4bcd0dc9342c --- M includes/ballots/RadioRangeBallot.php 1 file changed, 1 insertion(+), 1 deletion(-) git pull ssh://gerrit.wikimedia.org:29418/mediawiki/extensions/SecurePoll refs/changes/55/90555/1 diff --git a/includes/ballots/RadioRangeBallot.php b/includes/ballots/RadioRangeBallot.php index 3256ce7..8f35487 100644 --- a/includes/ballots/RadioRangeBallot.php +++ b/includes/ballots/RadioRangeBallot.php @@ -155,7 +155,7 @@ $s .= Xml::tags( 'td', array(), Xml::radio( $inputId, $score, !strcmp( $oldValue, $score ), - array( 'title' = $label ) ) + array( 'title' = Sanitizer::stripAllTags( $label ) ) ) ) . \n; } $s .= /tr\n; -- To view, visit https://gerrit.wikimedia.org/r/90555 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I45a7651803124e255105feb7d05a4bcd0dc9342c Gerrit-PatchSet: 1 Gerrit-Project: mediawiki/extensions/SecurePoll Gerrit-Branch: master Gerrit-Owner: Diederik dvanli...@wikimedia.org ___ MediaWiki-commits mailing list MediaWiki-commits@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] Abstract dependencies to separate requirements.txt file - change (analytics/wikimetrics)
Diederik has uploaded a new change for review.
https://gerrit.wikimedia.org/r/90682
Change subject: Abstract dependencies to separate requirements.txt file
..
Abstract dependencies to separate requirements.txt file
Make it easier to install all the dependencies for Wikimetrics
by enabling pip install -r requirements.txt
Also made the test script work on OSX.
Change-Id: I488826379e7e338605b1e02e108ba5b08ed92419
---
A requirements.txt
M scripts/test
M setup.py
3 files changed, 31 insertions(+), 23 deletions(-)
git pull ssh://gerrit.wikimedia.org:29418/analytics/wikimetrics
refs/changes/82/90682/1
diff --git a/requirements.txt b/requirements.txt
new file mode 100644
index 000..3b68938
--- /dev/null
+++ b/requirements.txt
@@ -0,0 +1,14 @@
+distribute=0.6.28
+sqlalchemy==0.8.1
+mysql-python==1.2.4
+requests==1.2.3
+flask==0.9
+flask-login==0.2.4
+flask-oauth==0.12
+wtforms==1.0.4
+# needed to run tests but bad in prod environment
+# because tests are currently distructive nose ===1.3.0
+coverage==3.6
+celery==3.0
+celery-with-redis==3.0
+PyYAML==3.10
diff --git a/scripts/test b/scripts/test
index 83a5573..fc20168 100755
--- a/scripts/test
+++ b/scripts/test
@@ -1,4 +1,4 @@
# for example:
# scripts/test tests/test_controllers/test_cohorts.py:CohortsControllerTest
# rm .coverage *.db
-find -name *.pyc | xargs rm ; nosetests --cover-erase $1
+find . -name *.pyc | xargs rm ; nosetests --cover-erase $1
diff --git a/setup.py b/setup.py
index b1e33a4..c0dc051 100644
--- a/setup.py
+++ b/setup.py
@@ -2,32 +2,26 @@
# follow the frog
from setuptools import setup
+from pip.req import parse_requirements
+
+# parse_requirements() returns generator of pip.req.InstallRequirement objects
+install_reqs = parse_requirements('requirements.txt')
+
+# reqs is a list of requirement
+# e.g. ['flask==0.9', 'sqlalchemye==0.8.1']
+reqs = [str(ir.req) for ir in install_reqs]
setup(
-name='wikimetrics',
-version='0.0.1',
-description='Wikipedia User Analysis Tool',
-url='http://www.github.com/wikimedia/analytics-wikimetrics',
-author='Andrew Otto, Dan Andreescu, Evan Rosen',
-packages=[
+name = 'wikimetrics',
+version = '0.0.1',
+description = 'Wikipedia Cohort Analysis Tool',
+url = 'http://www.github.com/wikimedia/analytics-wikimetrics',
+author = 'Andrew Otto, Dan Andreescu, Evan Rosen, Stefan Petrea'
+packages = [
'wikimetrics',
],
-install_requires=[
-'sqlalchemy == 0.8.1',
-'mysql-python == 1.2.4',
-'requests == 1.2.3',
-'flask == 0.9',
-'flask-login == 0.2.4',
-'flask-oauth == 0.12',
-'wtforms == 1.0.4',
-# needed to run tests but bad in prod environment
-# because tests are currently distructive 'nose == 1.3.0',
-'coverage == 3.6',
-'celery == 3.0',
-'celery-with-redis == 3.0',
-'PyYAML==3.10'
-],
-entry_points={
+install_requires = reqs
+entry_points = {
'console_scripts': [
'wikimetrics = wikimetrics.run:main'
]
--
To view, visit https://gerrit.wikimedia.org/r/90682
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: I488826379e7e338605b1e02e108ba5b08ed92419
Gerrit-PatchSet: 1
Gerrit-Project: analytics/wikimetrics
Gerrit-Branch: master
Gerrit-Owner: Diederik dvanli...@wikimedia.org
___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] Add dependencies for Wikimetrics to Jenkins to enable CI. - change (operations/puppet)
Diederik has uploaded a new change for review.
https://gerrit.wikimedia.org/r/90684
Change subject: Add dependencies for Wikimetrics to Jenkins to enable CI.
..
Add dependencies for Wikimetrics to Jenkins to enable CI.
Wikimetrics is webapp developed by the Analytics team and
hosted under the analytics/wikimetrics Gerrit repo. It's
written in Python and uses Flask and SQLAlchemy as core
dependencies. This patchset includes all Python dependencies.
Change-Id: I306ad8c75fb99bed7e338b7da03fc39936abb0b2
---
M modules/contint/manifests/packages.pp
1 file changed, 15 insertions(+), 0 deletions(-)
git pull ssh://gerrit.wikimedia.org:29418/operations/puppet
refs/changes/84/90684/1
diff --git a/modules/contint/manifests/packages.pp
b/modules/contint/manifests/packages.pp
index 069241a..7f48f79 100644
--- a/modules/contint/manifests/packages.pp
+++ b/modules/contint/manifests/packages.pp
@@ -101,6 +101,21 @@
ensure = 'installed',
}
+ # Dependencies for Wikimetrics
+ package { [
+'python-mysqldb',
+'python-nose',
+'python-dev',
+'python-sqlalchemy',
+'python-requests',
+'python-flask',
+'python-celery',
+'python-yaml',
+'python-coverage',
+'libmysqlclient-dev',
+]: ensure = 'installed',
+ }
+
# Used to build analytics udp-filters
package { ['libcidr0-dev', 'libanon0-dev']:
ensure = 'latest',
--
To view, visit https://gerrit.wikimedia.org/r/90684
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: I306ad8c75fb99bed7e338b7da03fc39936abb0b2
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Diederik dvanli...@wikimedia.org
___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] Bug 55763 - change (mediawiki...RSS)
Diederik has submitted this change and it was merged.
Change subject: Bug 55763
..
Bug 55763
Change-Id: I4329c4e652f3025ddcca4bcd38dedc33b1a378a6
---
M RELEASE-NOTES
M RSS.php
M RSSData.php
M RSSParser.php
4 files changed, 29 insertions(+), 26 deletions(-)
Approvals:
Wikinaut: Checked; Looks good to me, but someone else must approve
Diederik: Verified; Looks good to me, approved
jenkins-bot: Checked
diff --git a/RELEASE-NOTES b/RELEASE-NOTES
index 0891205..1d85971 100644
--- a/RELEASE-NOTES
+++ b/RELEASE-NOTES
@@ -1,6 +1,6 @@
RELEASE NOTES of the MediaWiki extension RSS
-Version 2.23 20130903
+Version 2.24 20131017
Manual http://www.mediawiki.org/wiki/Extension:RSS
@@ -27,6 +27,9 @@
The length limitation must be HTML tag-safe, but it is not at the moment.
Length limitation is disabled by default.
+=== Version 2.24 2013-10-17 ==
+* fix for Bug 55763 Undefined variable: txt in RSSParser.php on line 376
+
=== Version 2.23 2013-09-03 ===
* Fix parsing from CLI:
removed references to $wgTitle, $wgUser, and $wgLang.
diff --git a/RSS.php b/RSS.php
index c3aad25..05f4b05 100644
--- a/RSS.php
+++ b/RSS.php
@@ -4,7 +4,7 @@
*
* @file
* @ingroup Extensions
- * @version 2.22
+ * @version 2.24
* @author mutante, Daniel Kinzler, Rdb, Mafs, Thomas Gries, Alxndr, Chris
Reigrut, K001
* @author Kellan Elliott-McCrea kel...@protest.net -- author of MagpieRSS
* @author Jeroen De Dauw
@@ -14,7 +14,7 @@
* @link http://www.mediawiki.org/wiki/Extension:RSS Documentation
*/
-define( EXTENSION_RSS_VERSION, 2.23 );
+define( EXTENSION_RSS_VERSION, 2.24 );
if ( !defined( 'MEDIAWIKI' ) ) {
die( This is not a valid entry point.\n );
diff --git a/RSSData.php b/RSSData.php
index 96f0ea3..9ac81c5 100644
--- a/RSSData.php
+++ b/RSSData.php
@@ -41,7 +41,7 @@
* mark up their RSS, some more
precautions are
* needed.
*/
- $bit[$name] = $n-nodeValue;
+ $bit[$name] = trim( $n-nodeValue );
}
}
$this-items[] = $bit;
diff --git a/RSSParser.php b/RSSParser.php
index f313c48..a0cc138 100644
--- a/RSSParser.php
+++ b/RSSParser.php
@@ -363,30 +363,30 @@
// use the overloaded multi byte wrapper functions in
GlobalFunctions.php
foreach ( array_keys( $item ) as $info ) {
- switch ( $info ) {
- // ATOM id elements and RSS link elements are item
link urls
- case 'id':
- $txt = $this-sanitizeUrl( $item['id'] );
- $renderedItem = str_replace( '{{{link}}}',
$txt, $renderedItem );
- break;
- case 'link':
- if ( !isset( $item['id'] ) ) {
+ if ( $item[$info] != ) {
+ switch ( $info ) {
+ // ATOM id elements and RSS link elements
are item link urls
+ case 'id':
+ $txt = $this-sanitizeUrl( $item['id']
);
+ $renderedItem = str_replace(
'{{{link}}}', $txt, $renderedItem );
+ break;
+ case 'link':
$txt = $this-sanitizeUrl(
$item['link'] );
+ $renderedItem = str_replace(
'{{{link}}}', $txt, $renderedItem );
+ break;
+ case 'date':
+ $tempTimezone =
date_default_timezone_get();
+ date_default_timezone_set( 'UTC' );
+ $txt = date( $this-date, strtotime(
$this-escapeTemplateParameter( $item['date'] ) ) );
+ date_default_timezone_set(
$tempTimezone );
+ $renderedItem = str_replace(
'{{{date}}}', $txt, $renderedItem );
+ break;
+ default:
+ $str = $this-escapeTemplateParameter(
$item[$info] );
+ $str =
$parser-getFunctionLang()-truncate( $str, $this-ItemMaxLength );
+ $str = $this-highlightTerms( $str );
+ $renderedItem = str_replace( '{{{' .
$info . '}}}', $parser-insertStripItem( $str ), $renderedItem );
}
- $renderedItem = str_replace( '{{{link}}}',
$txt,
[MediaWiki-commits] [Gerrit] Whitelist Mingle Analytics RSS feed for RSS extension on Med... - change (operations/mediawiki-config)
Diederik has uploaded a new change for review. https://gerrit.wikimedia.org/r/89100 Change subject: Whitelist Mingle Analytics RSS feed for RSS extension on Mediawiki. .. Whitelist Mingle Analytics RSS feed for RSS extension on Mediawiki. Change-Id: I2a12f6ae20b333e6c70a1b2e35eb5bb06db97c30 --- M wmf-config/InitialiseSettings.php 1 file changed, 1 insertion(+), 0 deletions(-) git pull ssh://gerrit.wikimedia.org:29418/operations/mediawiki-config refs/changes/00/89100/1 diff --git a/wmf-config/InitialiseSettings.php b/wmf-config/InitialiseSettings.php index ff5f11e..00041b6 100644 --- a/wmf-config/InitialiseSettings.php +++ b/wmf-config/InitialiseSettings.php @@ -10632,6 +10632,7 @@ 'mediawikiwiki' = array( 'http://blog.wikimedia.org/feed/', 'https://git.wikimedia.org/feed/mediawiki/extensions/Translate.git', + 'https://mingle.corp.wikimedia.org/projects/analytics/feeds/8z8k6vUfniLmWc2qGe6GMQsxBujvKRKuybLd8mdTbMFHwGfLH3oxK*MU0E8zM6go.atom', ), ), -- To view, visit https://gerrit.wikimedia.org/r/89100 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I2a12f6ae20b333e6c70a1b2e35eb5bb06db97c30 Gerrit-PatchSet: 1 Gerrit-Project: operations/mediawiki-config Gerrit-Branch: master Gerrit-Owner: Diederik dvanli...@wikimedia.org ___ MediaWiki-commits mailing list MediaWiki-commits@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] Fix determining latest commit from git. - change (analytics/wikimetrics)
Diederik has uploaded a new change for review. https://gerrit.wikimedia.org/r/86293 Change subject: Fix determining latest commit from git. .. Fix determining latest commit from git. Change-Id: I1711516d780d1d60d1f104eaad7af8a60b5297b1 --- M wikimetrics/configurables.py 1 file changed, 8 insertions(+), 1 deletion(-) git pull ssh://gerrit.wikimedia.org:29418/analytics/wikimetrics refs/changes/93/86293/1 diff --git a/wikimetrics/configurables.py b/wikimetrics/configurables.py index bd6733d..bd83924 100644 --- a/wikimetrics/configurables.py +++ b/wikimetrics/configurables.py @@ -93,11 +93,18 @@ queue.config_from_object(args.override_config) +def get_absolute_path(): +return os.path.dirname(os.path.abspath(__file__)) + '/' + def get_wikimetrics_version(): Returns a tuple of the form (pretty version string, latest commit sha) +path = get_absolute_path() +orig_wd = os.getcwd() # remember our original working directory +os.chdir(path) +print path cmd = ['git', 'log', '--date', 'relative', --pretty=format:'%an %ar %h', '-n', '1'] p = subprocess.Popen(cmd, shell=False, stdout=subprocess.PIPE) version, err = p.communicate() @@ -108,5 +115,5 @@ latest, err = p.communicate() if err is not None: latest = 'unknown' - +os.chdir(orig_wd) return version, latest -- To view, visit https://gerrit.wikimedia.org/r/86293 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I1711516d780d1d60d1f104eaad7af8a60b5297b1 Gerrit-PatchSet: 1 Gerrit-Project: analytics/wikimetrics Gerrit-Branch: master Gerrit-Owner: Diederik dvanli...@wikimedia.org ___ MediaWiki-commits mailing list MediaWiki-commits@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] Remove global_south_editor_fractions - change (analytics...dashboard-data)
Diederik has submitted this change and it was merged. Change subject: Remove global_south_editor_fractions .. Remove global_south_editor_fractions The global_south_editor_fractions are now generated through the geowiki repository, and can be found in the geowiki-data repository. Change-Id: I42ced8c70746d2d2bc7b23c0a88b5533ca4ca295 --- D datafiles/global_south_editor_fractions.csv D datasources/global_south_editor_fractions.json D graphs/global_south_editor_fractions.json 3 files changed, 0 insertions(+), 518 deletions(-) Approvals: Diederik: Verified; Looks good to me, approved diff --git a/datafiles/global_south_editor_fractions.csv b/datafiles/global_south_editor_fractions.csv deleted file mode 100644 index 06d8fd0..000 --- a/datafiles/global_south_editor_fractions.csv +++ /dev/null @@ -1,368 +0,0 @@ -date,Global South Fraction (100+),Global South Fraction (5+),Global South Fraction (all) -2012/08/19,0.1559566787,0.18435727563,0.19409049224 -2012/08/20,0.155141202028,0.184653110048,0.193631058009 -2012/08/21,0.15473399459,0.184198618719,0.193330784884 -2012/08/22,0.154421033675,0.183876757759,0.193318817945 -2012/08/23,0.153263613415,0.183807179634,0.19337114226 -2012/08/24,0.153399132921,0.183518509534,0.192605639353 -2012/08/25,0.153637840765,0.183560400721,0.192820580078 -2012/08/26,0.153063063063,0.18411954012,0.193057913893 -2012/08/27,0.152690300522,0.184006406043,0.192392611468 -2012/08/28,0.151848867401,0.183461241002,0.191893085505 -2012/08/29,0.152195296432,0.183515364022,0.191909280082 -2012/08/30,0.151376558156,0.183538977587,0.192226690372 -2012/08/31,0.151643023882,0.183432865946,0.192178681412 -2012/09/01,0.151161747555,0.182936212019,0.191870810899 -2012/09/02,0.151022972003,0.183263251407,0.191912635076 -2012/09/03,0.151526032316,0.183229813665,0.192164806604 -2012/09/04,0.150794363163,0.182582610955,0.191798690285 -2012/09/05,0.149865229111,0.183120868744,0.191869826296 -2012/09/06,0.149708127526,0.182907150681,0.191403311826 -2012/09/07,0.149860573896,0.183141970428,0.191456978215 -2012/09/08,0.14889788574,0.182833481219,0.191408087399 -2012/09/09,0.149064411659,0.182694702387,0.191414293272 -2012/09/10,0.14848812095,0.182221378934,0.191296332973 -2012/09/11,0.147781477815,0.182338608568,0.190740684572 -2012/09/12,0.146550944761,0.182137368483,0.190496709644 -2012/09/13,0.14656654302,0.181938596906,0.190449548637 -2012/09/14,0.145944964127,0.182530927896,0.190336799533 -2012/09/15,0.14589121,0.182712295247,0.190604515936 -2012/09/16,0.146210045662,0.182789154614,0.190827950586 -2012/09/17,0.145721315957,0.182806619272,0.190828277883 -2012/09/18,0.145031905196,0.182714637893,0.190828323466 -2012/09/19,0.145232005846,0.182800813346,0.191085318425 -2012/09/20,0.14503396365,0.18260496388,0.190954296272 -2012/09/21,0.145756796473,0.183376219257,0.191040421699 -2012/09/22,0.146714588517,0.183140511737,0.190926521477 -2012/09/23,0.146662979351,0.184022841722,0.19119114509 -2012/09/24,0.14767699115,0.184284784024,0.191323774258 -2012/09/25,0.145775814467,0.183432024023,0.190653675222 -2012/09/26,0.145580110497,0.18296994341,0.190560210045 -2012/09/27,0.145950948635,0.182518006477,0.190331516055 -2012/09/28,0.14612464117,0.182335768192,0.19003710568 -2012/09/29,0.146667905939,0.182503495653,0.18953777106 -2012/09/30,0.147198657593,0.182611344666,0.18937278519 -2012/10/01,0.14734389562,0.182566532726,0.189051458069 -2012/10/02,0.147253567099,0.182287660697,0.188659494039 -2012/10/03,0.147347740668,0.181267224349,0.188016592886 -2012/10/04,0.147464141746,0.181866978968,0.187831117505 -2012/10/05,0.147445529677,0.181662717123,0.188005309988 -2012/10/06,0.146909637423,0.181899302033,0.188331401232 -2012/10/07,0.145809951366,0.182258478774,0.188531181651 -2012/10/08,0.145328719723,0.182017141669,0.188430794628 -2012/10/09,0.144750349977,0.181300108607,0.188318938113 -2012/10/10,0.144036009002,0.181234826951,0.188279311384 -2012/10/11,0.144559099437,0.180512995813,0.188596220356 -2012/10/12,0.144105213715,0.18055894557,0.188613157854 -2012/10/13,0.144557982563,0.180466129704,0.188737566236 -2012/10/14,0.144317329332,0.180674173825,0.188447922779 -2012/10/15,0.145589200337,0.180598074992,0.187964024165 -2012/10/16,0.144505958525,0.179990496034,0.187197652646 -2012/10/17,0.144744291376,0.180184673779,0.186959294374 -2012/10/18,0.144533838145,0.180148357471,0.186743724038 -2012/10/19,0.144384520768,0.180797427966,0.186618920092 -2012/10/20,0.144373874301,0.18059932111,0.186394442822 -2012/10/21,0.144612476371,0.180438941763,0.186651675183 -2012/10/22,0.143785310734,0.179874305581,0.186453392422 -2012/10/23,0.143717080512,0.179032902624,0.186314433088 -2012/10/24,0.142978642979,0.178724434225,0.185662409789 -2012/10/25,0.141951358001,0.178957564017,0.185233086493 -2012/10/26,0.142127659574,0.179219397807,0.185106958905 -2012/10/27,0.141495185954,0.179197389097,0.184982453464
[MediaWiki-commits] [Gerrit] Linkify references to Analytics Mingle projects. - change (operations/puppet)
Diederik has uploaded a new change for review.
https://gerrit.wikimedia.org/r/84338
Change subject: Linkify references to Analytics Mingle projects.
..
Linkify references to Analytics Mingle projects.
Change-Id: Ic79b7e74ca716670512af4e0496474c6b97f8761
---
M templates/gerrit/gerrit.config.erb
1 file changed, 3 insertions(+), 0 deletions(-)
git pull ssh://gerrit.wikimedia.org:29418/operations/puppet
refs/changes/38/84338/1
diff --git a/templates/gerrit/gerrit.config.erb
b/templates/gerrit/gerrit.config.erb
index 883d76e..8a91415 100644
--- a/templates/gerrit/gerrit.config.erb
+++ b/templates/gerrit/gerrit.config.erb
@@ -80,6 +80,9 @@
[commentlink cve]
match = \\b(CVE\\-\\d{4}\\-\\d+)\\b
link = https://cve.mitre.org/cgi-bin/cvename.cgi?name=$1
+[commentlink mingle]
+match = \\b#[for|finish]\\s(\\d+)\\.\\d+\\b
+link = https://mingle.corp.wikimedia.org/projects/analytics/cards/$1
[mimetype application/javascript]
safe = true
[mimetype application/x-php]
--
To view, visit https://gerrit.wikimedia.org/r/84338
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: Ic79b7e74ca716670512af4e0496474c6b97f8761
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Diederik dvanli...@wikimedia.org
___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] Add test data for unit-tests - change (analytics/kraken)
Diederik has uploaded a new change for review. https://gerrit.wikimedia.org/r/83962 Change subject: Add test data for unit-tests .. Add test data for unit-tests Change-Id: I0038c1c5ea7ad08fecf33710ba12791fac67f4e0 --- M .gitignore A kraken-pig/src/test/resources/testdata_desktop.csv A kraken-pig/src/test/resources/testdata_mobile.csv A kraken-pig/src/test/resources/testdata_session.csv 4 files changed, 53 insertions(+), 1 deletion(-) git pull ssh://gerrit.wikimedia.org:29418/analytics/kraken refs/changes/62/83962/1 diff --git a/.gitignore b/.gitignore index 123ae76..c5ebe20 100644 --- a/.gitignore +++ b/.gitignore @@ -17,7 +17,7 @@ doc/ deploy.sh -testdata* +testdata.csv kraken-dclass/src/main/java/org/openddr/ *.gz tmp/ diff --git a/kraken-pig/src/test/resources/testdata_desktop.csv b/kraken-pig/src/test/resources/testdata_desktop.csv new file mode 100644 index 000..bc96671 --- /dev/null +++ b/kraken-pig/src/test/resources/testdata_desktop.csv @@ -0,0 +1,25 @@ +knsq24.knams.wikimedia.org 560388694 2013-02-13T07:05:54.593 0 10.180.33.0 TCP_MEM_HIT/200 5041GET http://commons.wikimedia.org/w/index.php?title=MediaWiki:TextCleaner.jsaction=rawctype=text/javascript NONE/- text/javascript http://commons.wikimedia.org/wiki/File:Victoria_(Seychelles).jpg?uselang=ru - Mozilla/5.0%20(Windows%20NT%206.1;%20WOW64)%20AppleWebKit/537.17%20(KHTML,%20like%20Gecko)%20Chrome/24.0.1312.57%20Safari/537.17 ru-RU,ru;q=0.8,en-US;q=0.6,en;q=0.4 - +cp1036.eqiad.wmnet 1987998022 2013-02-13T07:05:54 0.001018524 10.167.250.0miss/3040 GET http://upload.wikimedia.org/wikipedia/commons/thumb/2/2c/PPD-34_and_PPD-34_38.JPG/200px-PPD-34_and_PPD-34_38.JPG - image/jpeg http://en.wikipedia.org/wiki/List_of_Russian_weaponry - Mozilla/4.0%20(compatible;%20MSIE%208.0;%20Windows%20NT%206.1;%20Trident/4.0;%20%20SLCC2;%20.NET%20CLR%202.0.50727;%20.NET%20CLR%203.5.30729;%20.NET%20CLR%203.0.30729;%20Media%20Center%20PC%206.0;%20.NET4.0C;%20.NET4.0E;%20) en-US - +cp1012.eqiad.wmnet 839078746 2013-02-13T07:05:54.672 0 10.185.212.0TCP_MEM_HIT/200 489 GET http://meta.wikimedia.org/wiki/Special:RecordImpression?banner=B13_BlankA_FRcampaign=C13_wpnd_mlWW_FRresult=showcountry=ARuserlang=esproject=wikipediadb=eswikisitename=Wikipediabucket=0 NONE/- image/png http://es.wikipedia.org/wiki/D%C3%ADa_de_San_Valent%C3%ADn - Mozilla/5.0%20(Windows%20NT%205.1)%20AppleWebKit/537.17%20(KHTML,%20like%20Gecko)%20Chrome/24.0.1312.57%20Safari/537.17 es-ES,es;q=0.8 - +cp1021.eqiad.wmnet 1978892196 2013-02-13T07:05:54 0.40531 10.49.217.0 hit/200 5738GET http://upload.wikimedia.org/wikipedia/commons/thumb/3/3d/Mediawiki-logo.png/35px-Mediawiki-logo.png - image/png http://en.wikipedia.org/wiki/Main_Page - Mozilla/5.0%20(Windows%20NT%205.1)%20AppleWebKit/537.17%20(KHTML,%20like%20Gecko)%20Chrome/24.0.1312.57%20Safari/537.17 en-US,en;q=0.8 - +amssq57.esams.wikimedia.org825729523 2013-02-13T07:05:54.661 15 10.138.51.0 TCP_MISS/20047879 GET http://upload.wikimedia.org/wikipedia/commons/thumb/6/60/Wiki-mam-intcs.png/220px-Wiki-mam-intcs.png CARP/91.198.174.57 image/png http://de.wikipedia.org/wiki/Cumshot- Mozilla/4.0%20(compatible;%20MSIE%208.0;%20Windows%20NT%205.2;%20Trident/4.0;%20.NET%20CLR%201.1.4322;%20.NET%20CLR%202.0.50727;%20.NET%20CLR%203.0.4506.2152;%20.NET%20CLR%203.5.30729;%20.NET4.0C;%20.NET4.0E;%20OfficeLiveConnector.1.5;%20OfficeLivePatch.1.3) de - +cp3010.esams.wikimedia.org 3167536827 2013-02-13T07:05:54 0.55790 10.123.230.0hit/200 508 GET http://upload.wikimedia.org/wikipedia/commons/thumb/3/3e/Flag_of_New_Zealand.svg/22px-Flag_of_New_Zealand.svg.png - image/png http://ru.wikipedia.org/wiki/Bell_UH-1_Iroquois - Opera/9.80%20(Windows%20NT%206.1;%20WOW64;%20U;%20ru)%20Presto/2.10.289%20Version/12.00 ru-RU,ru;q=0.9,en;q=0.8 - +cp1013.eqiad.wmnet 841173907 2013-02-13T07:05:54.739 1 10.43.19.0 TCP_MISS/20014279 GET http://ja.wikipedia.org/wiki/%E3%83%AD%E3%82%B7%E3%82%A2%E8%AA%9E%E4%BC%9A%E8%A9%B1 CARP/10.64.0.133text/html http://www.google.co.jp/url?sa=trct=jq=%E3%83%AD%E3%82%B7%E3%82%A2%E8%AA%9E%E3%80%80%E8%AC%9B%E5%BA%A7%E3%80%80%E3%82%BF%E3%83%AC%E3%83%B3%E3%83%88source=webcd=1ved=0CDAQFjAAurl=http%3A%2F%2Fja.wikipedia.org%2Fwiki%2F%25E3%2583%25AD%25E3%2582%25B7%25E3%2582%25A2%25E8%25AA%259E%25E4%25BC%259A%25E8%25A9%25B1ei=JjsbUfjkKeTAmQWNj4CoBAusg=AFQjCNHIxiPlFmQzx2Rr1-tCquUoSRfY1wbvm=bv.42261806,d.dGY - Mozilla/5.0%20(compatible;%20MSIE%209.0;%20Windows%20NT%206.1;%20Trident/5.0) ja-JP -
[MediaWiki-commits] [Gerrit] Add test data for unit-tests - change (analytics/kraken)
Diederik has submitted this change and it was merged. Change subject: Add test data for unit-tests .. Add test data for unit-tests Change-Id: I0038c1c5ea7ad08fecf33710ba12791fac67f4e0 --- M .gitignore A kraken-pig/src/test/resources/testdata_desktop.csv A kraken-pig/src/test/resources/testdata_mobile.csv A kraken-pig/src/test/resources/testdata_session.csv 4 files changed, 53 insertions(+), 1 deletion(-) Approvals: Diederik: Verified; Looks good to me, approved diff --git a/.gitignore b/.gitignore index 123ae76..c5ebe20 100644 --- a/.gitignore +++ b/.gitignore @@ -17,7 +17,7 @@ doc/ deploy.sh -testdata* +testdata.csv kraken-dclass/src/main/java/org/openddr/ *.gz tmp/ diff --git a/kraken-pig/src/test/resources/testdata_desktop.csv b/kraken-pig/src/test/resources/testdata_desktop.csv new file mode 100644 index 000..bc96671 --- /dev/null +++ b/kraken-pig/src/test/resources/testdata_desktop.csv @@ -0,0 +1,25 @@ +knsq24.knams.wikimedia.org 560388694 2013-02-13T07:05:54.593 0 10.180.33.0 TCP_MEM_HIT/200 5041GET http://commons.wikimedia.org/w/index.php?title=MediaWiki:TextCleaner.jsaction=rawctype=text/javascript NONE/- text/javascript http://commons.wikimedia.org/wiki/File:Victoria_(Seychelles).jpg?uselang=ru - Mozilla/5.0%20(Windows%20NT%206.1;%20WOW64)%20AppleWebKit/537.17%20(KHTML,%20like%20Gecko)%20Chrome/24.0.1312.57%20Safari/537.17 ru-RU,ru;q=0.8,en-US;q=0.6,en;q=0.4 - +cp1036.eqiad.wmnet 1987998022 2013-02-13T07:05:54 0.001018524 10.167.250.0miss/3040 GET http://upload.wikimedia.org/wikipedia/commons/thumb/2/2c/PPD-34_and_PPD-34_38.JPG/200px-PPD-34_and_PPD-34_38.JPG - image/jpeg http://en.wikipedia.org/wiki/List_of_Russian_weaponry - Mozilla/4.0%20(compatible;%20MSIE%208.0;%20Windows%20NT%206.1;%20Trident/4.0;%20%20SLCC2;%20.NET%20CLR%202.0.50727;%20.NET%20CLR%203.5.30729;%20.NET%20CLR%203.0.30729;%20Media%20Center%20PC%206.0;%20.NET4.0C;%20.NET4.0E;%20) en-US - +cp1012.eqiad.wmnet 839078746 2013-02-13T07:05:54.672 0 10.185.212.0TCP_MEM_HIT/200 489 GET http://meta.wikimedia.org/wiki/Special:RecordImpression?banner=B13_BlankA_FRcampaign=C13_wpnd_mlWW_FRresult=showcountry=ARuserlang=esproject=wikipediadb=eswikisitename=Wikipediabucket=0 NONE/- image/png http://es.wikipedia.org/wiki/D%C3%ADa_de_San_Valent%C3%ADn - Mozilla/5.0%20(Windows%20NT%205.1)%20AppleWebKit/537.17%20(KHTML,%20like%20Gecko)%20Chrome/24.0.1312.57%20Safari/537.17 es-ES,es;q=0.8 - +cp1021.eqiad.wmnet 1978892196 2013-02-13T07:05:54 0.40531 10.49.217.0 hit/200 5738GET http://upload.wikimedia.org/wikipedia/commons/thumb/3/3d/Mediawiki-logo.png/35px-Mediawiki-logo.png - image/png http://en.wikipedia.org/wiki/Main_Page - Mozilla/5.0%20(Windows%20NT%205.1)%20AppleWebKit/537.17%20(KHTML,%20like%20Gecko)%20Chrome/24.0.1312.57%20Safari/537.17 en-US,en;q=0.8 - +amssq57.esams.wikimedia.org825729523 2013-02-13T07:05:54.661 15 10.138.51.0 TCP_MISS/20047879 GET http://upload.wikimedia.org/wikipedia/commons/thumb/6/60/Wiki-mam-intcs.png/220px-Wiki-mam-intcs.png CARP/91.198.174.57 image/png http://de.wikipedia.org/wiki/Cumshot- Mozilla/4.0%20(compatible;%20MSIE%208.0;%20Windows%20NT%205.2;%20Trident/4.0;%20.NET%20CLR%201.1.4322;%20.NET%20CLR%202.0.50727;%20.NET%20CLR%203.0.4506.2152;%20.NET%20CLR%203.5.30729;%20.NET4.0C;%20.NET4.0E;%20OfficeLiveConnector.1.5;%20OfficeLivePatch.1.3) de - +cp3010.esams.wikimedia.org 3167536827 2013-02-13T07:05:54 0.55790 10.123.230.0hit/200 508 GET http://upload.wikimedia.org/wikipedia/commons/thumb/3/3e/Flag_of_New_Zealand.svg/22px-Flag_of_New_Zealand.svg.png - image/png http://ru.wikipedia.org/wiki/Bell_UH-1_Iroquois - Opera/9.80%20(Windows%20NT%206.1;%20WOW64;%20U;%20ru)%20Presto/2.10.289%20Version/12.00 ru-RU,ru;q=0.9,en;q=0.8 - +cp1013.eqiad.wmnet 841173907 2013-02-13T07:05:54.739 1 10.43.19.0 TCP_MISS/20014279 GET http://ja.wikipedia.org/wiki/%E3%83%AD%E3%82%B7%E3%82%A2%E8%AA%9E%E4%BC%9A%E8%A9%B1 CARP/10.64.0.133text/html http://www.google.co.jp/url?sa=trct=jq=%E3%83%AD%E3%82%B7%E3%82%A2%E8%AA%9E%E3%80%80%E8%AC%9B%E5%BA%A7%E3%80%80%E3%82%BF%E3%83%AC%E3%83%B3%E3%83%88source=webcd=1ved=0CDAQFjAAurl=http%3A%2F%2Fja.wikipedia.org%2Fwiki%2F%25E3%2583%25AD%25E3%2582%25B7%25E3%2582%25A2%25E8%25AA%259E%25E4%25BC%259A%25E8%25A9%25B1ei=JjsbUfjkKeTAmQWNj4CoBAusg=AFQjCNHIxiPlFmQzx2Rr1-tCquUoSRfY1wbvm=bv.42261806,d.dGY - Mozilla/5.0%20(compatible;%20MSIE%209.0;%20Windows%20NT%206.1;%20Trident/5.0) ja-JP - +amssq36.esams.wikimedia.org558274766 2013-02-13T07:05:54.709 0
[MediaWiki-commits] [Gerrit] Made a small change for Christian to demo gingle #for 1112.3 - change (analytics/wikimetrics)
Diederik has uploaded a new change for review. https://gerrit.wikimedia.org/r/81942 Change subject: Made a small change for Christian to demo gingle #for 1112.3 .. Made a small change for Christian to demo gingle #for 1112.3 Change-Id: I5f809844ec60e7215d4f564a5b52cf9167ca7d2a --- M README.md 1 file changed, 1 insertion(+), 0 deletions(-) git pull ssh://gerrit.wikimedia.org:29418/analytics/wikimetrics refs/changes/42/81942/1 diff --git a/README.md b/README.md index 7fe7125..fdf15a1 100644 --- a/README.md +++ b/README.md @@ -1,3 +1,4 @@ + Contents: * [Setting up your Development Environment](#development-environment) -- To view, visit https://gerrit.wikimedia.org/r/81942 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I5f809844ec60e7215d4f564a5b52cf9167ca7d2a Gerrit-PatchSet: 1 Gerrit-Project: analytics/wikimetrics Gerrit-Branch: master Gerrit-Owner: Diederik dvanli...@wikimedia.org ___ MediaWiki-commits mailing list MediaWiki-commits@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] Name should be a string, not an array. - change (analytics...dashboard-data)
Diederik has uploaded a new change for review. https://gerrit.wikimedia.org/r/81547 Change subject: Name should be a string, not an array. .. Name should be a string, not an array. Change-Id: Ic29afc95cc8ce01beea16e29c153ea5937d85f8b --- M datasources/grants_spending_by_country.json 1 file changed, 2 insertions(+), 4 deletions(-) git pull ssh://gerrit.wikimedia.org:29418/analytics/global-dev/dashboard-data refs/changes/47/81547/1 diff --git a/datasources/grants_spending_by_country.json b/datasources/grants_spending_by_country.json index dd1a7bf..61e2c52 100644 --- a/datasources/grants_spending_by_country.json +++ b/datasources/grants_spending_by_country.json @@ -16,9 +16,7 @@ label: Amount Funded } ], -name: [ -Grants Spending by Country -], +name: Grants Spending by Country, format: json, url: /data/datafiles/gp/grants_spending_by_country.json, shortName: , @@ -27,4 +25,4 @@ id: grants_spending_by_country, notes: , slug: grants_spending_by_country -} \ No newline at end of file +} -- To view, visit https://gerrit.wikimedia.org/r/81547 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: newchange Gerrit-Change-Id: Ic29afc95cc8ce01beea16e29c153ea5937d85f8b Gerrit-PatchSet: 1 Gerrit-Project: analytics/global-dev/dashboard-data Gerrit-Branch: master Gerrit-Owner: Diederik dvanli...@wikimedia.org ___ MediaWiki-commits mailing list MediaWiki-commits@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] Updating Kraken to cope with libdclass 2.2.2 - change (analytics/kraken)
Diederik has submitted this change and it was merged. Change subject: Updating Kraken to cope with libdclass 2.2.2 .. Updating Kraken to cope with libdclass 2.2.2 Although libdclass 2.2.2 can now be used, we still rely on the dtree file from libdclass-data 2.0.14 (i.e.: the dtree file in version 1.13), due significant changes in the patterns of newer dtree files. Change-Id: I3621ed24a83fe8cfdc431e74c3867e7b92b5603c --- M README.md M kraken-dclass/pom.xml A kraken-dclass/src/main/java/dclass/dClass.java D kraken-dclass/src/main/java/org/wikimedia/analytics/dclassjni/DclassWrapper.java M kraken-dclass/src/main/java/org/wikimedia/analytics/dclassjni/DeviceClassification.java D kraken-dclass/src/test/java/org/wikimedia/analytics/dclassjni/DclassLoaderTest.java A kraken-dclass/src/test/java/org/wikimedia/analytics/dclassjni/DclassWrapperTest.java A kraken-dclass/src/test/java/org/wikimedia/analytics/dclassjni/DeviceClassificationTest.java 8 files changed, 229 insertions(+), 181 deletions(-) Approvals: Diederik: Verified; Looks good to me, approved diff --git a/README.md b/README.md index ddde037..b2f04e4 100644 --- a/README.md +++ b/README.md @@ -38,7 +38,7 @@ ### Debian-like systems -If your an a Debian system, you can probably just grab the most recent .deb files of libdclass0 and libdclassjni0 for your architecture from [garage-coding](http://garage-coding.com/releases/libdclass-dev/) and install them using dpkg and can proceed to the next step. +If your an a Debian system, you can probably just grab the most recent .deb files of libdclass0 and libdclass-jni for your architecture from [Wikimedia's apt repository](http://apt.wikimedia.org/wikimedia/pool/main/d/dclass/) and install them using dpkg and can proceed to the next step. ### Non-Debian-like systems @@ -46,7 +46,7 @@ Getting files by hand -For an amd64 system with the most recent version being 2.0.14, +For an amd64 system with the most recent version being 2.2.0, that would be: mkdir dclass @@ -54,19 +54,19 @@ mkdir libdclass0 cd libdclass0 -wget http://garage-coding.com/releases/libdclass-dev/libdclass0_2.0.14_amd64.deb -ar x libdclass0_2.0.14_amd64.deb +wget http://apt.wikimedia.org/wikimedia/pool/main/d/dclass/libdclass0_2.2.2-1_amd64.deb +ar x libdclass0_2.2.2-1_amd64.deb tar -xzvf data.tar.gz -sudo mv usr/lib/libdclass.so.0.0.0 /usr/lib/libdclass.so.0 +sudo mv usr/lib/x86_64-linux-gnu/libdclass.so.0.0.0 /usr/lib/libdclass.so.0 sudo chmod +x /usr/lib/libdclass.so.0 cd .. -mkdir libdclassjni0 -cd libdclassjni0 -wget http://garage-coding.com/releases/libdclass-dev/libdclassjni0_2.0.14_amd64.deb -ar x libdclassjni0_2.0.14_amd64.deb +mkdir libdclass-jni +cd libdclass-jni +wget http://apt.wikimedia.org/wikimedia/pool/main/d/dclass/libdclass-jni_2.2.2-1_amd64.deb +ar x libdclass-jni_2.2.2-1_amd64.deb tar -xzvf data.tar.gz -sudo mv usr/lib/libdclassjni.so.0.0.0 /usr/lib/libdclassjni.so +sudo mv usr/lib/x86_64-linux-gnu/jni/libdclassjni.so.0.0.0 /usr/lib/libdclassjni.so sudo chmod +x /usr/lib/libdclassjni.so cd .. @@ -104,9 +104,9 @@ ## openddr.dtree file -Get a openddr.dtree file from an [Analytics Team member](http://www.mediawiki.org/wiki/Analytics), and store it as +Get a openddr.dtree file (version 1.13) from an [Analytics Team member](http://www.mediawiki.org/wiki/Analytics), and store it as -/usr/share/libdclass/dtrees/openddr.dtree +/usr/share/libdclass/openddr.dtree ## GeoIP files diff --git a/kraken-dclass/pom.xml b/kraken-dclass/pom.xml index f91dab5..cf755d8 100644 --- a/kraken-dclass/pom.xml +++ b/kraken-dclass/pom.xml @@ -17,13 +17,6 @@ /parent dependencies -!-- -dependency -groupIdorg.apache.devicemap/groupId -artifactIdopenddr-java/artifactId -version0.9.9-SNAPSHOT/version -/dependency --- /dependencies properties diff --git a/kraken-dclass/src/main/java/dclass/dClass.java b/kraken-dclass/src/main/java/dclass/dClass.java new file mode 100644 index 000..bee86b7 --- /dev/null +++ b/kraken-dclass/src/main/java/dclass/dClass.java @@ -0,0 +1,144 @@ +/** + * Copyright (C) 2012-2013 Wikimedia Foundation + + * This program is free software; you can redistribute it and/or + * modify it under the terms of the GNU General Public License + * as published by the Free Software Foundation; either version 2 + * of the License, or (at your option) any later version. + + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + + * You should have received a copy of the GNU General Public License + * along with
[MediaWiki-commits] [Gerrit] Add Maven Checkstyle - change (analytics/kraken)
Diederik has uploaded a new change for review.
https://gerrit.wikimedia.org/r/78960
Change subject: Add Maven Checkstyle
..
Add Maven Checkstyle
Change-Id: I8ad7109f8fa62441cafbe48df0897a2f1c8e084c
---
A checkstyle.xml
M pom.xml
2 files changed, 141 insertions(+), 0 deletions(-)
git pull ssh://gerrit.wikimedia.org:29418/analytics/kraken
refs/changes/60/78960/1
diff --git a/checkstyle.xml b/checkstyle.xml
new file mode 100644
index 000..c60c940
--- /dev/null
+++ b/checkstyle.xml
@@ -0,0 +1,135 @@
+?xml version=1.0?
+!DOCTYPE module PUBLIC
+-//Puppy Crawl//DTD Check Configuration 1.2//EN
+http://www.puppycrawl.com/dtds/configuration_1_2.dtd;
+module name=Checker
+
+ !-- Checks whether files end with a new line. --
+ !-- See http://checkstyle.sf.net/config_misc.html#NewlineAtEndOfFile
--
+ module name=NewlineAtEndOfFile /
+
+ !-- Checks that property files contain the same keys. --
+ !-- See http://checkstyle.sf.net/config_misc.html#Translation --
+ module name=Translation /
+ module name=FileLength /
+
+ module name=TreeWalker
+ !-- Checks for Javadoc comments. --
+ !-- See http://checkstyle.sf.net/config_javadoc.html --
+ module name=JavadocMethod
+ property name=scope value=protected /
+ /module
+ module name=JavadocType
+ property name=scope value=protected /
+ property name=allowUnknownTags value=true/
+ /module
+ module name=JavadocVariable
+ property name=scope value=protected /
+ /module
+ module name=JavadocStyle
+ property name=scope value=protected /
+ /module
+
+ !-- Checks for Naming Conventions. --
+ !-- See http://checkstyle.sf.net/config_naming.html --
+ module name=ConstantName /
+ module name=LocalFinalVariableName /
+ module name=LocalVariableName /
+ module name=MemberName
+ property name=format value=^[_a-z][a-zA-Z0-9]*$/
+ /module
+ module name=MethodName /
+ module name=PackageName /
+ module name=ParameterName /
+ module name=StaticVariableName
+ property name=format value=^[_a-z][a-zA-Z0-9]*$/
+ /module
+ module name=TypeName /
+
+ !-- Checks for imports --
+ !-- See http://checkstyle.sf.net/config_import.html --
+ module name=AvoidStarImport /
+ module name=IllegalImport / !-- defaults to sun.* packages
--
+ module name=RedundantImport /
+ module name=UnusedImports /
+
+
+ !-- Checks for Size Violations. --
+ !-- See http://checkstyle.sf.net/config_sizes.html --
+ !-- module name=LineLength --
+ !-- property name=max value=160/ --
+ !-- /module --
+ module name=MethodLength /
+ module name=ParameterNumber /
+
+
+ !-- Checks for whitespace --
+ !-- See http://checkstyle.sf.net/config_whitespace.html --
+ module name=EmptyForIteratorPad
+ property name=option value=space /
+ /module
+ module name=MethodParamPad /
+ module name=NoWhitespaceAfter /
+ module name=NoWhitespaceBefore /
+ module name=OperatorWrap /
+ module name=ParenPad /
+ module name=TypecastParenPad /
+ !-- module name=TabCharacter/ --
+ module name=WhitespaceAfter /
+ module name=WhitespaceAround /
+
+
+ !-- Modifier Checks --
+ !-- See http://checkstyle.sf.net/config_modifiers.html --
+ module name=ModifierOrder /
+ module name=RedundantModifier /
+
+
+ !-- Checks for blocks. You know, those {}'s --
+ !-- See http://checkstyle.sf.net/config_blocks.html --
+ module name=AvoidNestedBlocks /
+ module name=EmptyBlock /
+ module name=LeftCurly /
+ module name=NeedBraces /
+ module name=RightCurly /
+
+
+ !-- Checks for common coding problems --
+ !-- See http://checkstyle.sf.net/config_coding.html --
+ module name=AvoidInlineConditionals /
+ module name=DoubleCheckedLocking /!-- MY FAVOURITE --
+ module name=EmptyStatement /
+ module name=EqualsHashCode /
+ module name=HiddenField /
+ module name=IllegalInstantiation /
+ module name=InnerAssignment /
+
[MediaWiki-commits] [Gerrit] Switch Zero's returned schema to tuple - change (analytics/kraken)
Diederik has submitted this change and it was merged.
Change subject: Switch Zero's returned schema to tuple
..
Switch Zero's returned schema to tuple
When using current svn's pig, zero_carrier.pig failed with
file zero_carrier.pig, line 37, column 14 pig script failed to validate:
org.apache.pig.impl.logicalLayer.FrontendException: ERROR 0: Given UDF returns
an improper Schema. Schema should only contain one field of a Tuple, Bag, or a
single type. Returns: {chararray,chararray}
. We fix the output schema of Zero to return a tuple.
Change-Id: I09f18af429bd655b673072e37336f02d98811ba2
---
M kraken-pig/src/main/java/org/wikimedia/analytics/kraken/pig/Zero.java
1 file changed, 11 insertions(+), 8 deletions(-)
Approvals:
Diederik: Verified; Looks good to me, approved
diff --git
a/kraken-pig/src/main/java/org/wikimedia/analytics/kraken/pig/Zero.java
b/kraken-pig/src/main/java/org/wikimedia/analytics/kraken/pig/Zero.java
index 91e5c9b..13aa086 100644
--- a/kraken-pig/src/main/java/org/wikimedia/analytics/kraken/pig/Zero.java
+++ b/kraken-pig/src/main/java/org/wikimedia/analytics/kraken/pig/Zero.java
@@ -27,14 +27,13 @@
import org.apache.pig.data.DataType;
import org.apache.pig.data.Tuple;
import org.apache.pig.data.TupleFactory;
+import org.apache.pig.impl.logicalLayer.FrontendException;
import org.wikimedia.analytics.kraken.schemas.JsonToClassConverter;
import org.wikimedia.analytics.kraken.schemas.MccMnc;
import org.wikimedia.analytics.kraken.schemas.Schema;
import java.io.IOException;
-import java.util.ArrayList;
import java.util.HashMap;
-import java.util.List;
/**
@@ -132,11 +131,15 @@
throw new RuntimeException(e);
}
-
-Listorg.apache.pig.impl.logicalLayer.schema.Schema.FieldSchema
fields = new
ArrayListorg.apache.pig.impl.logicalLayer.schema.Schema.FieldSchema();
-// Carrier and ISO country are chararrays
-fields.add(new
org.apache.pig.impl.logicalLayer.schema.Schema.FieldSchema(null,
DataType.CHARARRAY));
-fields.add(new
org.apache.pig.impl.logicalLayer.schema.Schema.FieldSchema(null,
DataType.CHARARRAY));
-return new org.apache.pig.impl.logicalLayer.schema.Schema(fields);
+org.apache.pig.impl.logicalLayer.schema.Schema tupleSchema = new
org.apache.pig.impl.logicalLayer.schema.Schema();
+tupleSchema.add(new
org.apache.pig.impl.logicalLayer.schema.Schema.FieldSchema(carrier,
DataType.CHARARRAY));
+tupleSchema.add(new
org.apache.pig.impl.logicalLayer.schema.Schema.FieldSchema(iso,
DataType.CHARARRAY));
+org.apache.pig.impl.logicalLayer.schema.Schema ret;
+try {
+ ret = new org.apache.pig.impl.logicalLayer.schema.Schema(new
org.apache.pig.impl.logicalLayer.schema.Schema.FieldSchema(null,tupleSchema,
DataType.TUPLE));
+} catch (FrontendException e) {
+ throw new RuntimeException(e);
+}
+return ret;
}
}
--
To view, visit https://gerrit.wikimedia.org/r/73747
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: merged
Gerrit-Change-Id: I09f18af429bd655b673072e37336f02d98811ba2
Gerrit-PatchSet: 1
Gerrit-Project: analytics/kraken
Gerrit-Branch: master
Gerrit-Owner: QChris christ...@quelltextlich.at
Gerrit-Reviewer: Diederik dvanli...@wikimedia.org
___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] Switch PageViewEvalFunc's returned schema to tuple - change (analytics/kraken)
Diederik has submitted this change and it was merged.
Change subject: Switch PageViewEvalFunc's returned schema to tuple
..
Switch PageViewEvalFunc's returned schema to tuple
When using current svn's pig, zero_country.pig failed with
file zero_country.pig, line 37, column 13 pig script failed to validate:
org.apache.pig.impl.logicalLayer.FrontendException: ERROR 0: Given UDF returns
an improper Schema. Schema should only contain one field of a Tuple, Bag, or a
single type. Returns: {chararray,chararray,chararray}
. We fix the output schema of PageViewEvalFunc to return a tuple.
Change-Id: Iccaed180ebcadcc8896a1e5ba30bc841a5b17af7
---
M
kraken-pig/src/main/java/org/wikimedia/analytics/kraken/pig/PageViewEvalFunc.java
1 file changed, 12 insertions(+), 7 deletions(-)
Approvals:
Diederik: Verified; Looks good to me, approved
diff --git
a/kraken-pig/src/main/java/org/wikimedia/analytics/kraken/pig/PageViewEvalFunc.java
b/kraken-pig/src/main/java/org/wikimedia/analytics/kraken/pig/PageViewEvalFunc.java
index 94ebbee..b787552 100644
---
a/kraken-pig/src/main/java/org/wikimedia/analytics/kraken/pig/PageViewEvalFunc.java
+++
b/kraken-pig/src/main/java/org/wikimedia/analytics/kraken/pig/PageViewEvalFunc.java
@@ -23,13 +23,12 @@
import org.apache.pig.data.DataType;
import org.apache.pig.data.Tuple;
import org.apache.pig.data.TupleFactory;
+import org.apache.pig.impl.logicalLayer.FrontendException;
import org.apache.pig.impl.logicalLayer.schema.Schema;
import org.wikimedia.analytics.kraken.pageview.ProjectInfo;
import java.net.MalformedURLException;
import java.net.URL;
-import java.util.ArrayList;
-import java.util.List;
/**
* Entry point for the Pig UDF class that uses the Pageview filter logic.
@@ -135,10 +134,16 @@
throw new RuntimeException(e);
}
-ListSchema.FieldSchema fields = new ArrayListSchema.FieldSchema();
-fields.add(new Schema.FieldSchema(null, DataType.CHARARRAY)); //
language
-fields.add(new Schema.FieldSchema(null, DataType.CHARARRAY)); //
project
-fields.add(new Schema.FieldSchema(null, DataType.CHARARRAY)); //
site_version
-return new Schema(fields);
+Schema tupleSchema = new Schema();
+tupleSchema.add(new Schema.FieldSchema(language,
DataType.CHARARRAY));
+tupleSchema.add(new Schema.FieldSchema(project, DataType.CHARARRAY));
+tupleSchema.add(new Schema.FieldSchema(site_version,
DataType.CHARARRAY));
+Schema ret;
+try {
+ ret = new Schema(new Schema.FieldSchema(page_view, tupleSchema,
DataType.TUPLE));
+} catch (FrontendException e) {
+ throw new RuntimeException(e);
+}
+return ret;
}
}
--
To view, visit https://gerrit.wikimedia.org/r/73746
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: merged
Gerrit-Change-Id: Iccaed180ebcadcc8896a1e5ba30bc841a5b17af7
Gerrit-PatchSet: 1
Gerrit-Project: analytics/kraken
Gerrit-Branch: master
Gerrit-Owner: QChris christ...@quelltextlich.at
Gerrit-Reviewer: Diederik dvanli...@wikimedia.org
___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] Simplify regex by removing duplicate language code for 'ar' - change (operations/puppet)
Diederik has uploaded a new change for review.
https://gerrit.wikimedia.org/r/68657
Change subject: Simplify regex by removing duplicate language code for 'ar'
..
Simplify regex by removing duplicate language code for 'ar'
Change-Id: I85e8ee23476b8b4d098bf0b2eefffc799eb5a9ff
---
M templates/varnish/zero.inc.vcl.erb
1 file changed, 1 insertion(+), 1 deletion(-)
git pull ssh://gerrit.wikimedia.org:29418/operations/puppet
refs/changes/57/68657/1
diff --git a/templates/varnish/zero.inc.vcl.erb
b/templates/varnish/zero.inc.vcl.erb
index fdda943..385f3ba 100644
--- a/templates/varnish/zero.inc.vcl.erb
+++ b/templates/varnish/zero.inc.vcl.erb
@@ -318,7 +318,7 @@
} else if (client.ip ~ carrier_623_01) {
set req.http.X-DfltLang = fr;
if (req.http.X-Subdomain == M) {
- if (req.http.host ~
(^(fr|ar|sg|en|es|zh|ar|ha|ln|eo)\.m|^m)\.([a-zA-Z0-9-]+)\.org) {
+ if (req.http.host ~
(^(fr|ar|sg|en|es|zh|ha|ln|eo)\.m|^m)\.([a-zA-Z0-9-]+)\.org) {
set req.http.X-Carrier = Orange Central
African Republic;
set req.http.X-CS = 623-01;
}
--
To view, visit https://gerrit.wikimedia.org/r/68657
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: I85e8ee23476b8b4d098bf0b2eefffc799eb5a9ff
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Diederik dvanli...@wikimedia.org
___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] Separation of libraries in Makefile.am - change (analytics/dclass)
Diederik has submitted this change and it was merged. Change subject: Separation of libraries in Makefile.am .. Separation of libraries in Makefile.am * libdclass is created * libdclassjni depends on libdclass Change-Id: I0845ce8526d629334faf4d9cbdf7f42284c76da0 --- M Makefile.am M configure.ac 2 files changed, 20 insertions(+), 11 deletions(-) Approvals: Diederik: Verified; Looks good to me, approved diff --git a/Makefile.am b/Makefile.am index c2f5e42..9d4b397 100644 --- a/Makefile.am +++ b/Makefile.am @@ -1,19 +1,27 @@ CFLAGS= LDFLAGS= SOVERSION = 2 -lib_LTLIBRARIES = libdclassjni.la +lib_LTLIBRARIES = libdclass.la libdclassjni.la AUTOMAKE_OPTIONS = foreign -libdclassjni_la_SOURCES = src/dtree_core.c \ - src/openddr_client.c \ - src/dclass_file.c \ - src/dtree_mem.c\ - src/dtree_util.c \ - src/dclass_client.c\ - java-wikimedia/dclass-wrapper.c +libdclass_la_SOURCES = src/dtree_core.c \ + src/openddr_client.c \ + src/dclass_file.c \ + src/dtree_mem.c \ + src/dtree_util.c \ + src/dclass_client.c + +libdclass_la_LDFLAGS = -D__X86_64__ -fPIC -shared -DSHARED_OBJECT -Wl +libdclass_ladir = $(CURDIR) -libdclassjni_la_CFLAGS = -g -I$(JAVA_HOME)/include -I$(JAVA_HOME)/include/linux -Wall -I ./jni/ -I./src/ -#libdclassjni_la_LDFLAGS = -D__X86_64__ -fPIC -shared -DSHARED_OBJECT -Wl,--no-undefined -libdclassjni_la_LDFLAGS = -D__X86_64__ -fPIC -shared -DSHARED_OBJECT -Wl + +libdclassjni_la_SOURCES = java-wikimedia/dclass-wrapper.c +libdclassjni_ladir = $(CURDIR)/.libs +#libdclassjni_la_LIBRARIES = libdclass.so +libdclassjni_la_CFLAGS = -g -I$(CURDIR)/.libs -I$(JAVA_HOME)/include -I$(JAVA_HOME)/include/linux -Wall -I ./jni/ -I./src/ -ldclass + + +#libdclassjni_la_CFLAGS = -g -I$(JAVA_HOME)/include -I$(JAVA_HOME)/include/linux -Wall -I ./jni/ -I./src/ +#libdclassjni_la_LDFLAGS = -D__X86_64__ -fPIC -shared -DSHARED_OBJECT -Wl diff --git a/configure.ac b/configure.ac index 16e731b..1f207ea 100644 --- a/configure.ac +++ b/configure.ac @@ -6,6 +6,7 @@ AC_CONFIG_HEADERS([config.h]) AM_INIT_AUTOMAKE AC_USE_SYSTEM_EXTENSIONS +LT_INIT([disable-static]) # Checks for programs. AC_PROG_AWK -- To view, visit https://gerrit.wikimedia.org/r/68703 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: merged Gerrit-Change-Id: I0845ce8526d629334faf4d9cbdf7f42284c76da0 Gerrit-PatchSet: 1 Gerrit-Project: analytics/dclass Gerrit-Branch: wikimedia Gerrit-Owner: Stefan.petrea ste...@garage-coding.com Gerrit-Reviewer: Diederik dvanli...@wikimedia.org ___ MediaWiki-commits mailing list MediaWiki-commits@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] Fixed 341 - change (analytics/wikistats)
Diederik has submitted this change and it was merged.
Change subject: Fixed 341
..
Fixed 341
* Merged Australia into Oceania region
* Removed Australia as region
* Added test to check if Australia still appears in SquidReportCountryData.htm
* removed output from test 09
* added dummy file(blank.sh) to testdata/merge-australia-into-oceania to make
sure the directory is versioned
* fixed CommonConfig.pm with $WORKSPACE variable
* updated with pipe-separated 5-th field containing both IP and
Country code
Change-Id: I1474fcd79ddc5d958e45961e37e31d29bd9677fa
---
A .travis.yml
M squids/.gitignore
M squids/csv/.gitignore
M squids/csv/meta/RegionCodes.csv
M squids/perl/SquidCountArchiveProcessLogRecord.pm
M squids/perl/SquidCountArchiveReadInput.pm
M squids/perl/SquidReportArchive.pl
M squids/t/06-regression-mismatch-world-north-south-unknown.t
M squids/t/09-regression-totals-fixes-squidreportclients.t
A squids/t/11-merge-australia-into-oceania.t
M squids/t/CommonConfig.pm
M squids/t/Generate/Squid.pm
A squids/testdata/merge-australia-into-oceania/SquidCountArchiveConfig.pm
A squids/testdata/merge-australia-into-oceania/SquidReportArchiveConfig.pm
A squids/testdata/merge-australia-into-oceania/blank.txt
15 files changed, 189 insertions(+), 79 deletions(-)
Approvals:
Erik Zachte: Checked
Diederik: Looks good to me, approved
jenkins-bot: Verified
diff --git a/.travis.yml b/.travis.yml
new file mode 100644
index 000..e950543
--- /dev/null
+++ b/.travis.yml
@@ -0,0 +1,23 @@
+
+language: perl
+perl:
+ - 5.16
+ - 5.14
+ - 5.12
+ - 5.10
+
+
+before_install:
+
+ - sudo apt-get update -qq
+ - # sudo apt-get install -qq
+ - cpanm --quiet --notest Template
+ - cpanm --quiet --notest JSON::XS HTML::TreeBuilder::XPath
+
+install:
+
+script:
+ - hostname
+ - pwd
+ - echo ==
+ - cd squids; prove -v -Iperl/ t/
diff --git a/squids/.gitignore b/squids/.gitignore
index a25cb96..3b337db 100644
--- a/squids/.gitignore
+++ b/squids/.gitignore
@@ -29,8 +29,11 @@
!*.sh
# file specifying line-ending characters
!.gitattributes
+!testdata
!testdata/
!testdata/*
+!testdata/*/*.pm
+!testdata/*/*.txt
!testdata/*/*.gz
# and readme's
@@ -45,6 +48,7 @@
!logs_basic/
+!csv
# advice Linus
# use add -f to add few files in ignored folders
diff --git a/squids/csv/.gitignore b/squids/csv/.gitignore
index 6309fb0..ddeaea3 100644
--- a/squids/csv/.gitignore
+++ b/squids/csv/.gitignore
@@ -1,6 +1,11 @@
# ignore all
*
+
+!meta
+!meta/
+!meta/*.csv
+
# expect this file
!.gitignore
diff --git a/squids/csv/meta/RegionCodes.csv b/squids/csv/meta/RegionCodes.csv
index 5a32a01..35cc5c2 100644
--- a/squids/csv/meta/RegionCodes.csv
+++ b/squids/csv/meta/RegionCodes.csv
@@ -30,7 +30,7 @@
AR,SA,S,Argentina
AS,OC,S,American Samoa
AT,EU,N,Austria
-AU,AU,N,Australia
+AU,OC,N,Australia
AW,SA,S,Aruba
AX,EU,N,Aland Islands
AZ,AS,S,Azerbaijan
@@ -110,7 +110,7 @@
GW,AF,S,Guinea-Bissau
GY,SA,S,Guyana
HK,AS,S,Hong Kong
-HM,AU,N,Heard Island and McDonald Islands
+HM,OC,N,Heard Island and McDonald Islands
HN,CA,S,Honduras
HR,EU,N,Croatia
HT,CA,S,Haiti
diff --git a/squids/perl/SquidCountArchiveProcessLogRecord.pm
b/squids/perl/SquidCountArchiveProcessLogRecord.pm
index 7d32ef1..d537512 100755
--- a/squids/perl/SquidCountArchiveProcessLogRecord.pm
+++ b/squids/perl/SquidCountArchiveProcessLogRecord.pm
@@ -97,7 +97,8 @@
$redirected_to_mobile += $count_event ;
return ;
}
- }
+ };
+
$url =~ s/^http\w?\:\/\///o ;
$url =~ s/\%3A/:/gio ;
@@ -142,58 +143,12 @@
else
{ $mimecat = other ; }
+
+
if ($job_runs_on_production_server)
{
$country = ;
-
-#
-# udp-filters has logic for gelocating and it's using
-# the x-forwarded-for field to geolocate first if that's
-# present but only the first ip inside that field.
-#
-# At the present moment there is not enough time to
-# check all the ips in XFF field but we can come back
-# at this later -- Stefan Petrea
-#
-# The reason for this is that now 2012-Dec-04 we have a high
-# priority for getting editor reports for November out
-# there.
-#
-
-#@xffparts = split('%20',$xff) ;
-#foreach $ip (@xffparts)
-#{
- #next if $country ne ;
- #if ($ip =~ /^\d+\.\d+\.\d+\.\d+$/)
- #{
- #$country = $savedipcountry { $ip } ;
- #if ($country eq )
- #{
- #$country = `echo $ip | /usr/local/bin/geoiplogtag 1` ;
- #$country =~ s/.*\s([\w-\(])/$1/ ;
- #$country =~ s/\s//g ;
- #$savedipcountry { $ip } = $country ;
- #}
- #$foundip = $ip ;
- #if ($country =~ /(^(--|-P|A1|A2|AB|BL|G2|GF|KO|MF|O1|TE|TK)$|null)/
) # Non-countries
- #{
[MediaWiki-commits] [Gerrit] Fix for mingle 356 - change (analytics/wikistats)
Diederik has submitted this change and it was merged.
Change subject: Fix for mingle 356
..
Fix for mingle 356
* Added test for 356. It passes. The bug report could not be
reproduced in current version of the code, but a test was written
to make sure it will never occur
* Added travis support (I also added it for the 341 gerrit patchset which is
in review )
* Removed verbose output from test 09
* Modified t/CommonConfig.pm to use $ENV{WORKSPACE} for the path of
the workspace on Jenkins ( I found this out from Krinkle of
#wikimedia-operations, thanks )
* 5-th field is now IP and Country Code separated by | and Wikistats
agrees with this
Change-Id: I5c74df172e1f3505cfd72b068d7d9a4476ce
---
A .travis.yml
M squids/.gitignore
M squids/perl/SquidCountArchiveProcessLogRecord.pm
M squids/perl/SquidCountArchiveReadInput.pm
M squids/t/06-regression-mismatch-world-north-south-unknown.t
M squids/t/09-regression-totals-fixes-squidreportclients.t
A squids/t/10-regression-mingle-356-bugzilla-46269.t
M squids/t/CommonConfig.pm
M squids/t/Generate/Squid.pm
A
squids/testdata/regression-mingle-356-bugzilla-46269/SquidCountArchiveConfig.pm
A
squids/testdata/regression-mingle-356-bugzilla-46269/SquidReportArchiveConfig.pm
A squids/testdata/regression-mingle-356-bugzilla-46269/ua.txt.gz
12 files changed, 232 insertions(+), 64 deletions(-)
Approvals:
Erik Zachte: Checked
Diederik: Looks good to me, approved
jenkins-bot: Verified
diff --git a/.travis.yml b/.travis.yml
new file mode 100644
index 000..e950543
--- /dev/null
+++ b/.travis.yml
@@ -0,0 +1,23 @@
+
+language: perl
+perl:
+ - 5.16
+ - 5.14
+ - 5.12
+ - 5.10
+
+
+before_install:
+
+ - sudo apt-get update -qq
+ - # sudo apt-get install -qq
+ - cpanm --quiet --notest Template
+ - cpanm --quiet --notest JSON::XS HTML::TreeBuilder::XPath
+
+install:
+
+script:
+ - hostname
+ - pwd
+ - echo ==
+ - cd squids; prove -v -Iperl/ t/
diff --git a/squids/.gitignore b/squids/.gitignore
index a25cb96..3b337db 100644
--- a/squids/.gitignore
+++ b/squids/.gitignore
@@ -29,8 +29,11 @@
!*.sh
# file specifying line-ending characters
!.gitattributes
+!testdata
!testdata/
!testdata/*
+!testdata/*/*.pm
+!testdata/*/*.txt
!testdata/*/*.gz
# and readme's
@@ -45,6 +48,7 @@
!logs_basic/
+!csv
# advice Linus
# use add -f to add few files in ignored folders
diff --git a/squids/perl/SquidCountArchiveProcessLogRecord.pm
b/squids/perl/SquidCountArchiveProcessLogRecord.pm
index 7d32ef1..d537512 100755
--- a/squids/perl/SquidCountArchiveProcessLogRecord.pm
+++ b/squids/perl/SquidCountArchiveProcessLogRecord.pm
@@ -97,7 +97,8 @@
$redirected_to_mobile += $count_event ;
return ;
}
- }
+ };
+
$url =~ s/^http\w?\:\/\///o ;
$url =~ s/\%3A/:/gio ;
@@ -142,58 +143,12 @@
else
{ $mimecat = other ; }
+
+
if ($job_runs_on_production_server)
{
$country = ;
-
-#
-# udp-filters has logic for gelocating and it's using
-# the x-forwarded-for field to geolocate first if that's
-# present but only the first ip inside that field.
-#
-# At the present moment there is not enough time to
-# check all the ips in XFF field but we can come back
-# at this later -- Stefan Petrea
-#
-# The reason for this is that now 2012-Dec-04 we have a high
-# priority for getting editor reports for November out
-# there.
-#
-
-#@xffparts = split('%20',$xff) ;
-#foreach $ip (@xffparts)
-#{
- #next if $country ne ;
- #if ($ip =~ /^\d+\.\d+\.\d+\.\d+$/)
- #{
- #$country = $savedipcountry { $ip } ;
- #if ($country eq )
- #{
- #$country = `echo $ip | /usr/local/bin/geoiplogtag 1` ;
- #$country =~ s/.*\s([\w-\(])/$1/ ;
- #$country =~ s/\s//g ;
- #$savedipcountry { $ip } = $country ;
- #}
- #$foundip = $ip ;
- #if ($country =~ /(^(--|-P|A1|A2|AB|BL|G2|GF|KO|MF|O1|TE|TK)$|null)/
) # Non-countries
- #{ $country = ; }
- #}
-#}
-if ($country eq )
-{
- $country = $fields [14] ;
- $foundip = $client_ip ;
-}
-if (($country eq ) || ($country =~ /null/))
-{
- $country = -- ;
- if ($foundip =~ /:/)
- {
-$country = -P ;
- }
-}
-if (IsInternal($foundip))
-{ $country = -X ; }
+($country) = $client_ip =~ /\|(..)$/;
}
else
{
diff --git a/squids/perl/SquidCountArchiveReadInput.pm
b/squids/perl/SquidCountArchiveReadInput.pm
index 6e8ad7d..d578444 100755
--- a/squids/perl/SquidCountArchiveReadInput.pm
+++ b/squids/perl/SquidCountArchiveReadInput.pm
@@ -182,12 +182,12 @@
else
{ open IN,$file_in ; } #
[MediaWiki-commits] [Gerrit] Set cwd for git log when determining version info - change (operations/puppet)
Diederik has uploaded a new change for review.
https://gerrit.wikimedia.org/r/64252
Change subject: Set cwd for git log when determining version info
..
Set cwd for git log when determining version info
Change-Id: I0d189d7469d818497ff797a8b2c68f8295af973d
---
M templates/misc/e3-metrics.settings.py.erb
1 file changed, 7 insertions(+), 8 deletions(-)
git pull ssh://gerrit.wikimedia.org:29418/operations/puppet
refs/changes/52/64252/1
diff --git a/templates/misc/e3-metrics.settings.py.erb
b/templates/misc/e3-metrics.settings.py.erb
index e51a8e3..63fad62 100644
--- a/templates/misc/e3-metrics.settings.py.erb
+++ b/templates/misc/e3-metrics.settings.py.erb
@@ -9,14 +9,6 @@
# Get working set of Python modules
working_set = WorkingSet()
-# Determine UMAPI version using git foo
-cmd = ['git', 'log', '--date', 'relative', --pretty=format:'%an %ar %h',
'-n', '1']
-p = subprocess.Popen(cmd, shell=False, stdout=subprocess.PIPE)
-version, err = p.communicate()
-if err is not None:
-version = 'Unknown version'
-
-
# Project settings
#
@@ -39,6 +31,13 @@
__secret_key__ = '%= secret_key %'
+# Determine UMAPI version using git foo
+cmd = ['git', 'log', '--date', 'relative', --pretty=format:'%an %ar %h',
'-n', '1']
+p = subprocess.Popen(cmd, shell=False, stdout=subprocess.PIPE,
cwd=__project_home__)
+version, err = p.communicate()
+if err is not None:
+version = 'Unknown version'
+
try:
working_set.require('Flask-Login=0.1.2')
__flask_login_exists__ = True
--
To view, visit https://gerrit.wikimedia.org/r/64252
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: I0d189d7469d818497ff797a8b2c68f8295af973d
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Diederik dvanli...@wikimedia.org
___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] Two fixes to rolematcher.py - change (operations/puppet)
Diederik has uploaded a new change for review. https://gerrit.wikimedia.org/r/63668 Change subject: Two fixes to rolematcher.py .. Two fixes to rolematcher.py Change-Id: I13b481707feafac3bd09188ae80a07ff6bc8b8a8 --- M files/misc/PacketLossLogtailer.py M files/misc/rolematcher.py 2 files changed, 2 insertions(+), 2 deletions(-) git pull ssh://gerrit.wikimedia.org:29418/operations/puppet refs/changes/68/63668/1 diff --git a/files/misc/PacketLossLogtailer.py b/files/misc/PacketLossLogtailer.py index 4c2c2f9..42e3d99 100644 --- a/files/misc/PacketLossLogtailer.py +++ b/files/misc/PacketLossLogtailer.py @@ -27,12 +27,12 @@ def __init__(self): '''This function should initialize any data structures or variables needed for the internal state of the line parser.''' +self.last_update_time = time.time() self.reset_state() self.lock = threading.RLock() # a list of rolematchers which are simple object to determine the role of a particular server # this list is obtained from crawling noc.wikimedia.org/pybal and parse the available configurations self.day_in_seconds = 24 * 60 * 60 -self.last_update_time = time.time() self.matchers = rolematcher.init() # this is what will match the packet loss lines # packet loss format : diff --git a/files/misc/rolematcher.py b/files/misc/rolematcher.py index 1a4dd6d..71d0940 100644 --- a/files/misc/rolematcher.py +++ b/files/misc/rolematcher.py @@ -204,7 +204,7 @@ if __name__ == '__main__': -if len(sys.argv) != 1: +if sys.argv[0] == '': print 'Please specify path to packetloss log file, call this file only for testing purposes.' sys.exit(-1) else: -- To view, visit https://gerrit.wikimedia.org/r/63668 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I13b481707feafac3bd09188ae80a07ff6bc8b8a8 Gerrit-PatchSet: 1 Gerrit-Project: operations/puppet Gerrit-Branch: production Gerrit-Owner: Diederik dvanli...@wikimedia.org ___ MediaWiki-commits mailing list MediaWiki-commits@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] Add version info to umapi puppet template. - change (operations/puppet)
Diederik has uploaded a new change for review. https://gerrit.wikimedia.org/r/63717 Change subject: Add version info to umapi puppet template. .. Add version info to umapi puppet template. Change-Id: Ib4c37ac04620ac4e8b56f570572b6250ea56a11b --- M templates/misc/e3-metrics.settings.py.erb 1 file changed, 8 insertions(+), 0 deletions(-) git pull ssh://gerrit.wikimedia.org:29418/operations/puppet refs/changes/17/63717/1 diff --git a/templates/misc/e3-metrics.settings.py.erb b/templates/misc/e3-metrics.settings.py.erb index bf331e4..e51a8e3 100644 --- a/templates/misc/e3-metrics.settings.py.erb +++ b/templates/misc/e3-metrics.settings.py.erb @@ -4,10 +4,18 @@ from os.path import exists from urllib2 import urlopen import json +import subprocess # Get working set of Python modules working_set = WorkingSet() +# Determine UMAPI version using git foo +cmd = ['git', 'log', '--date', 'relative', --pretty=format:'%an %ar %h', '-n', '1'] +p = subprocess.Popen(cmd, shell=False, stdout=subprocess.PIPE) +version, err = p.communicate() +if err is not None: +version = 'Unknown version' + # Project settings # -- To view, visit https://gerrit.wikimedia.org/r/63717 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: newchange Gerrit-Change-Id: Ib4c37ac04620ac4e8b56f570572b6250ea56a11b Gerrit-PatchSet: 1 Gerrit-Project: operations/puppet Gerrit-Branch: production Gerrit-Owner: Diederik dvanli...@wikimedia.org ___ MediaWiki-commits mailing list MediaWiki-commits@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] Non-performant hack: set threads to 1 - change (operations/puppet)
Diederik has uploaded a new change for review. https://gerrit.wikimedia.org/r/63143 Change subject: Non-performant hack: set threads to 1 .. Non-performant hack: set threads to 1 This is a temporarily measure while the job queue system in UMAPI is being replaced. As soon as that code has been deployed then this changeset should be reverted. Change-Id: I0cf8a37b9671c6f2b468bff32eb78715b445ee13 --- M templates/apache/sites/metrics.wikimedia.org.erb 1 file changed, 1 insertion(+), 1 deletion(-) git pull ssh://gerrit.wikimedia.org:29418/operations/puppet refs/changes/43/63143/1 diff --git a/templates/apache/sites/metrics.wikimedia.org.erb b/templates/apache/sites/metrics.wikimedia.org.erb index cfcd3b4..3e4eb78 100644 --- a/templates/apache/sites/metrics.wikimedia.org.erb +++ b/templates/apache/sites/metrics.wikimedia.org.erb @@ -50,7 +50,7 @@ SSLCertificateKeyFile /etc/ssl/private/%= site_name %.key SSLCACertificatePath /etc/ssl/certs/ - WSGIDaemonProcess api user=%= metrics_user % group=wikidev threads=5 python-path=%= e3_analysis_path % + WSGIDaemonProcess api user=%= metrics_user % group=wikidev threads=1 python-path=%= e3_analysis_path % WSGIScriptAlias / %= document_root %/api.wsgi Directory %= document_root % -- To view, visit https://gerrit.wikimedia.org/r/63143 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I0cf8a37b9671c6f2b468bff32eb78715b445ee13 Gerrit-PatchSet: 1 Gerrit-Project: operations/puppet Gerrit-Branch: production Gerrit-Owner: Diederik dvanli...@wikimedia.org ___ MediaWiki-commits mailing list MediaWiki-commits@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] Added per dc / server role breakdown of udp2log packetloss m... - change (operations/puppet)
Diederik has uploaded a new change for review. https://gerrit.wikimedia.org/r/63220 Change subject: Added per dc / server role breakdown of udp2log packetloss monitoring. .. Added per dc / server role breakdown of udp2log packetloss monitoring. Instead of having one overall packetloss metric in Ganglia per udp2log host, this patchset introduces functionality to map a hostname to it's role and dc and to calculate packetloss per dc / role. The rolematcher.py determines the role and dc of a hostname; this information is now hardcoded and was obtained from noc.wikimedia.org/pybal; Not sure if it's worth the effort to dynamically fetching this information. Roles include apache, api, mobile, text, upload, bits, ssl/ip6. The interfaces of PacketLossLogtailer.py have not been changed, this changeset might require some changes in Ganglia as well. A local test of ganglia-logtailer did work. Change-Id: I084c210d996068f213977f2fb942d4a80021be8f --- M files/misc/PacketLossLogtailer.py A files/misc/rolematcher.py 2 files changed, 160 insertions(+), 20 deletions(-) git pull ssh://gerrit.wikimedia.org:29418/operations/puppet refs/changes/20/63220/1 diff --git a/files/misc/PacketLossLogtailer.py b/files/misc/PacketLossLogtailer.py index cdc5078..3ec2f15 100644 --- a/files/misc/PacketLossLogtailer.py +++ b/files/misc/PacketLossLogtailer.py @@ -1,8 +1,8 @@ # -*- coding: utf-8 -*- ### ### This plugin for logtailer will crunch WMF packet loss logs and return: -###* average percent loss -###* ninetieth percentile loss +###* average percent loss per server role +###* ninetieth percentile loss per server role ### It will throw out ###* packet loss numbers greater than 98% ###* large margins of error @@ -17,6 +17,7 @@ # local dependencies from ganglia_logtailer_helper import GangliaMetricObject from ganglia_logtailer_helper import LogtailerParsingException, LogtailerStateException +import rolematcher class PacketLossLogtailer(object): # only used in daemon mode @@ -26,6 +27,8 @@ needed for the internal state of the line parser.''' self.reset_state() self.lock = threading.RLock() +# a list of rolematchers which are simple object to determine the role of a particular server +self.matchers = rolematcher.init() # this is what will match the packet loss lines # packet loss format : # %[%Y-%m-%dT%H:%M:%S]t %server lost: (%percentloss +/- %margin) @@ -51,11 +54,13 @@ # capture data percentloss = float(linebits['percentloss']) margin = float(linebits['margin']) +role = self.determine_role(linebits['server']) # store for 90th % and average calculations # on ssl servers, sequence numbers are out of order. # http://rt.wikimedia.org/Ticket/Display.html?id=1616 if( ( margin = 20 ) and ( percentloss = 98 ) ): -self.percentloss_list.append(percentloss) +self.percentloss_dict.setdefault(role, []) +self.percentloss_dict[role].append(percentloss) else: raise LogtailerParsingException, regmatch failed to match @@ -63,6 +68,17 @@ self.lock.release() raise LogtailerParsingException, regmatch or contents failed with %s % e self.lock.release() + +def determine_role(self, hostname): +if hostname == 'total': +return 'total' +role = 'misc' # default group for when we were not able to determine the role +for matcher in self.matchers: +if matcher == hostname: +role = matcher.get_role() +break; +return role + # example function for deep copy # takes no arguments # returns one object @@ -72,9 +88,10 @@ currently being modified so that the other thread can deal with it without fear of it changing out from under it. The format of this object is internal to the plugin.''' -myret = dict(percentloss_list=self.percentloss_list +myret = dict(percentloss_dict=self.percentloss_dict ) return myret + # example function for reset_state # takes no arguments # returns nothing @@ -87,8 +104,9 @@ reset_state should store now() each time it's called, and get_state will use the time since that now() to do its calculations''' self.num_hits = 0 -self.percentloss_list = list() +self.percentloss_dict = dict() self.last_reset_time = time.time() + # example for keeping track of runtimes # takes no arguments # returns float number of seconds for this run @@ -98,6 +116,7 @@ it.''' self.duration = dur self.dur_override = True + def
[MediaWiki-commits] [Gerrit] Bug fix for 'Group parameter not honored', Mingle #603 - change (analytics/user-metrics)
Diederik has uploaded a new change for review.
https://gerrit.wikimedia.org/r/62974
Change subject: Bug fix for 'Group parameter not honored', Mingle #603
..
Bug fix for 'Group parameter not honored', Mingle #603
Change-Id: I338692301f420baf34ea112e8b6d6a28cc8d9f85
---
M user_metrics/api/engine/request_meta.py
1 file changed, 0 insertions(+), 2 deletions(-)
git pull ssh://gerrit.wikimedia.org:29418/analytics/user-metrics
refs/changes/74/62974/1
diff --git a/user_metrics/api/engine/request_meta.py
b/user_metrics/api/engine/request_meta.py
index bbcb54c..dac1712 100644
--- a/user_metrics/api/engine/request_meta.py
+++ b/user_metrics/api/engine/request_meta.py
@@ -55,10 +55,8 @@
# Default group + structure that maps values in the query string to new ones
DEFAULT_GROUP = 'reg'
REQUEST_VALUE_MAPPING = {
-'group': {
'reg': USER_METRIC_PERIOD_TYPE.REGISTRATION,
'activity': USER_METRIC_PERIOD_TYPE.INPUT,
-}
}
def RequestMetaFactory(cohort_expr, cohort_gen_timestamp, metric_expr):
--
To view, visit https://gerrit.wikimedia.org/r/62974
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: I338692301f420baf34ea112e8b6d6a28cc8d9f85
Gerrit-PatchSet: 1
Gerrit-Project: analytics/user-metrics
Gerrit-Branch: master
Gerrit-Owner: Diederik dvanli...@wikimedia.org
___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] Ensure that all UMAPI files belong to wikidev group - change (operations/puppet)
Diederik has uploaded a new change for review. https://gerrit.wikimedia.org/r/63005 Change subject: Ensure that all UMAPI files belong to wikidev group .. Ensure that all UMAPI files belong to wikidev group Change-Id: I2d3bca7d48446cda22d4bcfd1209ec28e2df222c --- M manifests/misc/statistics.pp 1 file changed, 3 insertions(+), 1 deletion(-) git pull ssh://gerrit.wikimedia.org:29418/operations/puppet refs/changes/05/63005/1 diff --git a/manifests/misc/statistics.pp b/manifests/misc/statistics.pp index 2a41b26..d6515c3 100644 --- a/manifests/misc/statistics.pp +++ b/manifests/misc/statistics.pp @@ -306,7 +306,7 @@ $e3_home = /a/e3 $e3_analysis_path = $e3_home/E3Analysis/ - $metrics_user = $misc::statistics::user::username + $metrics_user = $misc::statistics::user::username $secret_key = $passwords::e3::metrics::secret_key @@ -394,6 +394,8 @@ directory = $e3_analysis_path, origin= https://gerrit.wikimedia.org/r/p/analytics/user-metrics.git;, owner = $metrics_user, + group = wikidev, + recurse = true, require = [Package[python-flask], File[$e3_home], Class[misc::statistics::user], Class[misc::statistics::packages::python]], ensure= latest, } -- To view, visit https://gerrit.wikimedia.org/r/63005 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I2d3bca7d48446cda22d4bcfd1209ec28e2df222c Gerrit-PatchSet: 1 Gerrit-Project: operations/puppet Gerrit-Branch: production Gerrit-Owner: Diederik dvanli...@wikimedia.org ___ MediaWiki-commits mailing list MediaWiki-commits@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] Enable LOAD LOCAL INFILE in MySQL - change (mediawiki/vagrant)
Diederik has uploaded a new change for review. https://gerrit.wikimedia.org/r/62727 Change subject: Enable LOAD LOCAL INFILE in MySQL .. Enable LOAD LOCAL INFILE in MySQL Without 'local-infile=1' you would get error 'The used command is not allowed with this MySQL version' Change-Id: Idf945e28ad8ca97e3f4626dec8137d58bf1456cb --- M puppet/modules/mysql/templates/my.cnf.erb 1 file changed, 5 insertions(+), 0 deletions(-) git pull ssh://gerrit.wikimedia.org:29418/mediawiki/vagrant refs/changes/27/62727/1 diff --git a/puppet/modules/mysql/templates/my.cnf.erb b/puppet/modules/mysql/templates/my.cnf.erb index 9a382aa..5ad7fa3 100644 --- a/puppet/modules/mysql/templates/my.cnf.erb +++ b/puppet/modules/mysql/templates/my.cnf.erb @@ -8,6 +8,7 @@ [mysql] pager = less -inSFX prompt = (\R:\m)\_\u@\h:[\d]\_ +local-infile=1 % if @dbname % database = %= @dbname % @@ -15,3 +16,7 @@ comments = true show-warnings = true default-character-set = utf8 + +[mysqld] +local-infile=1 + -- To view, visit https://gerrit.wikimedia.org/r/62727 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: newchange Gerrit-Change-Id: Idf945e28ad8ca97e3f4626dec8137d58bf1456cb Gerrit-PatchSet: 1 Gerrit-Project: mediawiki/vagrant Gerrit-Branch: master Gerrit-Owner: Diederik dvanli...@wikimedia.org ___ MediaWiki-commits mailing list MediaWiki-commits@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] fixing bugs from DarTar's feedback - change (analytics/user-metrics)
Diederik has submitted this change and it was merged.
Change subject: fixing bugs from DarTar's feedback
..
fixing bugs from DarTar's feedback
Change-Id: Ic27656299347a63763735cff5ac4cbd58d183d84
---
A user_metrics/api/static/css/site.css
M user_metrics/api/templates/csv_upload.html
M user_metrics/api/templates/csv_upload_form.html
M user_metrics/api/templates/layout.html
M user_metrics/api/views.py
5 files changed, 43 insertions(+), 33 deletions(-)
Approvals:
Diederik: Verified; Looks good to me, approved
diff --git a/user_metrics/api/static/css/site.css
b/user_metrics/api/static/css/site.css
new file mode 100644
index 000..061d81f
--- /dev/null
+++ b/user_metrics/api/static/css/site.css
@@ -0,0 +1,6 @@
+body { padding-top: 60px; padding-bottom: 60px; }
+.nav { width: 70%; }
+.nav li:last-child { color: #cacaca; float: right !important; }
+.nav li:last-child a { color: #0088cc; display: inline-block; }
+label.error { color: #af; }
+div.flash { margin: 4px 0 20px 0; }
diff --git a/user_metrics/api/templates/csv_upload.html
b/user_metrics/api/templates/csv_upload.html
index ab487e6..10c4ffe 100644
--- a/user_metrics/api/templates/csv_upload.html
+++ b/user_metrics/api/templates/csv_upload.html
@@ -3,7 +3,7 @@
{% include csv_upload_form.html %}
div
h6File Format/h6
-pUser Metrics expects a CSV file with the following format /p
+pUser Metrics expects a CSV file with the following format (the
username, project header row is not required)/p
precodeusername,project
Evan (WMF),en/code/pre
ul
diff --git a/user_metrics/api/templates/csv_upload_form.html
b/user_metrics/api/templates/csv_upload_form.html
index 3486e47..0c44dfd 100644
--- a/user_metrics/api/templates/csv_upload_form.html
+++ b/user_metrics/api/templates/csv_upload_form.html
@@ -1,13 +1,12 @@
-form enctype=multipart/form-data action=/uploads/cohort method=POST
class=form-horizontal
+form enctype=multipart/form-data action=/uploads/cohort method=POST
class=upload-cohort form-horizontal
div class=control-group
label for=cohort_name class=control-labelCohort Name/label
div class=controls
-input type=text name=cohort_name id=cohort_name required
value={% if cohort_name %}{{cohort_name}}{% endif %}/
-a class=btn validate_cohort_name
href=/validate/cohort/allowedIs Available?/a
+input type=text name=cohort_name id=cohort_name value={%
if cohort_name %}{{cohort_name}}{% endif %}/
+label for=cohort_name(automatically checks
availability)/label
/div
/div
-div class=control-group
-label for=cohort_project class=control-labelWiki Project/label
+div class=control-group label for=cohort_project
class=control-labelWiki Project/label
div class=controls
select name=cohort_project id=cohort_project
option value=(select one or specify in file)/option
@@ -20,31 +19,37 @@
div class=control-group
label for=csv_cohort class=control-labelCSV File/label
div class=controls
-input type=file name=csv_cohort id=csv_cohort required/
+input type=file name=csv_cohort id=csv_cohort/
/div
/div
div class=form-actions
input type=submit class=btn btn-primary value=Upload CSV/
/div
/form
-
+script
src=//ajax.aspnetcdn.com/ajax/jquery.validate/1.11.1/jquery.validate.min.js/script
script
$(document).ready(function(){
-
-$('.validate_cohort_name').on('click', function(event){
-event.preventDefault();
-event.stopPropagation();
-a = $(this);
-
-$.get(a.attr('href'), {cohort_name: $('#cohort_name').val()},
function(response){
-if (JSON.parse(response)){
-alert('Available!');
-} else {
-alert('Cohort already exists.');
+
+jQuery.validator.addMethod('cohortName', function(value, element) {
+return /^[0-9_\-A-Za-z ]*$/.test(value);
+}, 'Cohort names should only contain letters, numbers, spaces, dashes,
and underscores');
+
+$('form.upload-cohort').validate({
+messages: {
+cohort_name: {
+remote: 'This cohort name is taken.',
}
-}).fail(function(){
-alert('Server Error');
-});
-});
+},
+rules: {
+cohort_name: {
+required: true,
+cohortName: true,
+remote: '/validate/cohort/allowed'
+},
+csv_cohort: {
+required: true
+}
+}
});
+});
/script
diff --git a/user_metrics/api/templates/layout.html
b/user_metrics/api/templates/layout.html
index e6c69ea..e46880f
[MediaWiki-commits] [Gerrit] Both the 'pager' and 'prompt' variable are part of the [mysq... - change (mediawiki/vagrant)
Diederik has uploaded a new change for review. https://gerrit.wikimedia.org/r/62018 Change subject: Both the 'pager' and 'prompt' variable are part of the [mysql] section and not the [client] section. This causes mysqldump to fail .. Both the 'pager' and 'prompt' variable are part of the [mysql] section and not the [client] section. This causes mysqldump to fail Change-Id: I7ea6afd1e51393b922bfc28d44afd0bd2cca2192 --- M puppet/modules/mysql/templates/my.cnf.erb 1 file changed, 2 insertions(+), 1 deletion(-) git pull ssh://gerrit.wikimedia.org:29418/mediawiki/vagrant refs/changes/18/62018/1 diff --git a/puppet/modules/mysql/templates/my.cnf.erb b/puppet/modules/mysql/templates/my.cnf.erb index 00f1fe1..0acccb3 100644 --- a/puppet/modules/mysql/templates/my.cnf.erb +++ b/puppet/modules/mysql/templates/my.cnf.erb @@ -1,10 +1,11 @@ [client] user = root password = %= @password % + +[mysql] pager = less -inSFX prompt = (\R:\m)\_\u@\h:[\d]\_ -[mysql] % if @dbname % database = %= @dbname % % end % -- To view, visit https://gerrit.wikimedia.org/r/62018 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I7ea6afd1e51393b922bfc28d44afd0bd2cca2192 Gerrit-PatchSet: 1 Gerrit-Project: mediawiki/vagrant Gerrit-Branch: master Gerrit-Owner: Diederik dvanli...@wikimedia.org ___ MediaWiki-commits mailing list MediaWiki-commits@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] Add s6 and s7 to user-metrics api. - change (operations/puppet)
Diederik has uploaded a new change for review.
https://gerrit.wikimedia.org/r/62099
Change subject: Add s6 and s7 to user-metrics api.
..
Add s6 and s7 to user-metrics api.
Change-Id: I2c31bcf5456fc60173cae9a8384e0862ea928666
---
M manifests/misc/statistics.pp
1 file changed, 12 insertions(+), 0 deletions(-)
git pull ssh://gerrit.wikimedia.org:29418/operations/puppet
refs/changes/99/62099/1
diff --git a/manifests/misc/statistics.pp b/manifests/misc/statistics.pp
index d270bc8..2a41b26 100644
--- a/manifests/misc/statistics.pp
+++ b/manifests/misc/statistics.pp
@@ -356,6 +356,18 @@
'host' = 's5-analytics-slave.eqiad.wmnet',
'port' = 3306,
},
+'s6' = {
+'user' = $passwords::mysql::research::user,
+'passwd' = $passwords::mysql::research::pass,
+'host' = 's6-analytics-slave.eqiad.wmnet',
+'port' = 3306,
+},
+'s7' = {
+'user' = $passwords::mysql::research::user,
+'passwd' = $passwords::mysql::research::pass,
+'host' = 's7-analytics-slave.eqiad.wmnet',
+'port' = 3306,
+},
}
package { [python-flask, python-flask-login]:
--
To view, visit https://gerrit.wikimedia.org/r/62099
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: I2c31bcf5456fc60173cae9a8384e0862ea928666
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Diederik dvanli...@wikimedia.org
___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] Create path using os.join.path instead of ''.join - change (mediawiki/vagrant)
Diederik has uploaded a new change for review. https://gerrit.wikimedia.org/r/61340 Change subject: Create path using os.join.path instead of ''.join .. Create path using os.join.path instead of ''.join Change-Id: Idac914bc58d18e9db55b880a4edf50085c799d5c --- M puppet/modules/user_metrics/templates/settings.py.erb 1 file changed, 3 insertions(+), 2 deletions(-) git pull ssh://gerrit.wikimedia.org:29418/mediawiki/vagrant refs/changes/40/61340/1 diff --git a/puppet/modules/user_metrics/templates/settings.py.erb b/puppet/modules/user_metrics/templates/settings.py.erb index 3019c21..b66bd1a 100644 --- a/puppet/modules/user_metrics/templates/settings.py.erb +++ b/puppet/modules/user_metrics/templates/settings.py.erb @@ -4,6 +4,7 @@ from os.path import exists from urllib2 import urlopen import json +import os # Get working set of Python modules working_set = WorkingSet() @@ -16,8 +17,8 @@ __instance_port__ = 5000 __project_home__= '%= user_metrics_path %' -__web_home__= ''.join([__project_home__, 'user_metrics/api/']) -__data_file_dir__ = ''.join([__project_home__, 'data/']) +__web_home__= os.path.join(__project_home__, 'user_metrics/api/') +__data_file_dir__ = os.path.join(__project_home__, 'data/') __query_module__= 'user_metrics.query.query_calls_sql' __user_thread_max__ = 50 -- To view, visit https://gerrit.wikimedia.org/r/61340 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: newchange Gerrit-Change-Id: Idac914bc58d18e9db55b880a4edf50085c799d5c Gerrit-PatchSet: 1 Gerrit-Project: mediawiki/vagrant Gerrit-Branch: master Gerrit-Owner: Diederik dvanli...@wikimedia.org ___ MediaWiki-commits mailing list MediaWiki-commits@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] Added simple script to create admin account through CLI - change (analytics/E3Analysis)
Diederik has submitted this change and it was merged.
Change subject: Added simple script to create admin account through CLI
..
Added simple script to create admin account through CLI
Change-Id: I1252dd377132ea3b53962775f1ee0aee113d4ce6
---
A scripts/create_account.py
1 file changed, 22 insertions(+), 0 deletions(-)
Approvals:
Diederik: Verified; Looks good to me, approved
diff --git a/scripts/create_account.py b/scripts/create_account.py
new file mode 100644
index 000..ad644ac
--- /dev/null
+++ b/scripts/create_account.py
@@ -0,0 +1,22 @@
+#!/usr/bin/python
+# coding=utf-8
+
+#Create default admin account
+#For puppet from drdee
+
+import argparse
+from user_metrics.api.session import APIUser
+
+def create_user(username, password):
+ u = APIUser(username)
+ u.set_password(password)
+ u.register_user()
+
+
+if __name__ == '__main__':
+ parser = argparse.ArgumentParser()
+ parser.add_argument('-u', '--username', default='admin', help='give
username for usermetrics api')
+ parser.add_argument('-p', '--password', default='vagrant', help='give
password for user')
+ args = parser.parse_args()
+
+ create_user(args.username, args.password)
--
To view, visit https://gerrit.wikimedia.org/r/60930
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: merged
Gerrit-Change-Id: I1252dd377132ea3b53962775f1ee0aee113d4ce6
Gerrit-PatchSet: 2
Gerrit-Project: analytics/E3Analysis
Gerrit-Branch: master
Gerrit-Owner: Diederik dvanli...@wikimedia.org
Gerrit-Reviewer: Diederik dvanli...@wikimedia.org
Gerrit-Reviewer: Erosen ero...@wikimedia.org
Gerrit-Reviewer: Ottomata o...@wikimedia.org
Gerrit-Reviewer: Rfaulk ryan.faulk...@mail.mcgill.ca
___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] Use os.path.join instead of string concatenation to create p... - change (analytics/E3Analysis)
Diederik has uploaded a new change for review. https://gerrit.wikimedia.org/r/60966 Change subject: Use os.path.join instead of string concatenation to create paths. This is required to make the user_metrics puppet module work .. Use os.path.join instead of string concatenation to create paths. This is required to make the user_metrics puppet module work Change-Id: I88c03886896cf48fe13b06233a74e26d43fe56ff --- M user_metrics/api/engine/data.py 1 file changed, 7 insertions(+), 6 deletions(-) git pull ssh://gerrit.wikimedia.org:29418/analytics/E3Analysis refs/changes/66/60966/1 diff --git a/user_metrics/api/engine/data.py b/user_metrics/api/engine/data.py index 301847b..6b452b1 100644 --- a/user_metrics/api/engine/data.py +++ b/user_metrics/api/engine/data.py @@ -53,6 +53,7 @@ from collections import OrderedDict from hashlib import sha1 import cPickle +import os import user_metrics.etl.data_loader as dl from user_metrics.config import logging @@ -254,17 +255,17 @@ def read_pickle_data(): try: -with open(settings.__data_file_dir__ + - 'api_data.pkl', 'rb') as pkl_file: +with open(os.path.join(settings.__data_file_dir__, + 'api_data.pkl'), 'rb') as pkl_file: return cPickle.load(pkl_file) except IOError: -with open(settings.__data_file_dir__ + - 'api_data.pkl', 'wb') as pkl_file: +with open(os.path.join(settings.__data_file_dir__, + 'api_data.pkl'), 'wb') as pkl_file: data = OrderedDict() cPickle.dump(data, pkl_file) return data def write_pickle_data(obj): -with open(settings.__data_file_dir__ + - 'api_data.pkl', 'wb') as pkl_file: +with open(os.path.join(settings.__data_file_dir__, + 'api_data.pkl'), 'wb') as pkl_file: cPickle.dump(obj, pkl_file) \ No newline at end of file -- To view, visit https://gerrit.wikimedia.org/r/60966 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I88c03886896cf48fe13b06233a74e26d43fe56ff Gerrit-PatchSet: 1 Gerrit-Project: analytics/E3Analysis Gerrit-Branch: master Gerrit-Owner: Diederik dvanli...@wikimedia.org ___ MediaWiki-commits mailing list MediaWiki-commits@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] Seed wiki db with random sample from test2wiki revision, pag... - change (analytics/E3Analysis)
Diederik has uploaded a new change for review. https://gerrit.wikimedia.org/r/61147 Change subject: Seed wiki db with random sample from test2wiki revision, page and user entries .. Seed wiki db with random sample from test2wiki revision, page and user entries Change-Id: Ic91716e76d63fe2ec9045fd4909fa84111c8c976 --- A scripts/seed.sql 1 file changed, 148 insertions(+), 0 deletions(-) git pull ssh://gerrit.wikimedia.org:29418/analytics/E3Analysis refs/changes/47/61147/1 -- To view, visit https://gerrit.wikimedia.org/r/61147 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: newchange Gerrit-Change-Id: Ic91716e76d63fe2ec9045fd4909fa84111c8c976 Gerrit-PatchSet: 1 Gerrit-Project: analytics/E3Analysis Gerrit-Branch: master Gerrit-Owner: Diederik dvanli...@wikimedia.org ___ MediaWiki-commits mailing list MediaWiki-commits@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] Add Troubleshooting section - change (mediawiki/vagrant)
Diederik has uploaded a new change for review. https://gerrit.wikimedia.org/r/60891 Change subject: Add Troubleshooting section .. Add Troubleshooting section Change-Id: Id97bde8291d9406a390c152dd803dff6464df83b --- M README 1 file changed, 20 insertions(+), 0 deletions(-) git pull ssh://gerrit.wikimedia.org:29418/mediawiki/vagrant refs/changes/91/60891/1 diff --git a/README b/README index b213c5f..bf1eae5 100644 --- a/README +++ b/README @@ -65,3 +65,23 @@ with the vagrant files run `git pull` and then `vagrant reload`. The latter will restart the VM. +## Troubleshooting + +Sometimes you might get the following error on OSX + +There was an error while executing `VBoxManage`, a CLI used by Vagrant +for controlling VirtualBox. The command and stderr is shown below. + +Command: [hostonlyif, create] + +Stderr: 0%... +Progress state: NS_ERROR_FAILURE +VBoxManage: error: Failed to create the host-only adapter +VBoxManage: error: VBoxNetAdpCtl: Error while adding new interface: failed to open /dev/vboxnetctl: No such file or directory + +VBoxManage: error: Details: code NS_ERROR_FAILURE (0x80004005), component HostNetworkInterface, interface IHostNetworkInterface +VBoxManage: error: Context: int handleCreate(HandlerArg*, int, int*) at line 68 of file VBoxManageHostonly.cpp + +The way to fix this is to run the following command: + +sudo /Library/StartupItems/VirtualBox/VirtualBox restart -- To view, visit https://gerrit.wikimedia.org/r/60891 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: newchange Gerrit-Change-Id: Id97bde8291d9406a390c152dd803dff6464df83b Gerrit-PatchSet: 1 Gerrit-Project: mediawiki/vagrant Gerrit-Branch: master Gerrit-Owner: Diederik dvanli...@wikimedia.org ___ MediaWiki-commits mailing list MediaWiki-commits@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] Add config.h to repo - change (analytics/udp-filters)
Diederik has submitted this change and it was merged. Change subject: Add config.h to repo .. Add config.h to repo Change-Id: I6edb4a2fdee753691849039d18dd88a0184610dd --- A config.h 1 file changed, 137 insertions(+), 0 deletions(-) Approvals: Diederik: Verified; Looks good to me, approved diff --git a/config.h b/config.h new file mode 100644 index 000..0ef4757 --- /dev/null +++ b/config.h @@ -0,0 +1,137 @@ +/* config.h. Generated from config.h.in by configure. */ +/* config.h.in. Generated from configure.ac by autoheader. */ + +/* Define to 1 if you have the arpa/inet.h header file. */ +#define HAVE_ARPA_INET_H 1 + +/* Define to 1 if you have the ctype.h header file. */ +#define HAVE_CTYPE_H 1 + +/* Define to 1 if you have the GeoIPCity.h header file. */ +#define HAVE_GEOIPCITY_H 1 + +/* Define to 1 if you have the GeoIP.h header file. */ +#define HAVE_GEOIP_H 1 + +/* Define to 1 if you have the getopt.h header file. */ +#define HAVE_GETOPT_H 1 + +/* Define to 1 if you have the inttypes.h header file. */ +#define HAVE_INTTYPES_H 1 + +/* Define to 1 if you have the libcidr.h header file. */ +/* #undef HAVE_LIBCIDR_H */ + +/* Define to 1 if you have the `geoip' library (-lgeoip). */ +#define HAVE_LIBGEOIP 1 + +/* Define to 1 if your system has a GNU libc compatible `malloc' function, and + to 0 otherwise. */ +#define HAVE_MALLOC 1 + +/* Define to 1 if you have the memory.h header file. */ +#define HAVE_MEMORY_H 1 + +/* Define to 1 if you have the `memset' function. */ +#define HAVE_MEMSET 1 + +/* Define to 1 if you have the netdb.h header file. */ +#define HAVE_NETDB_H 1 + +/* Define to 1 if you have the netinet/in.h header file. */ +#define HAVE_NETINET_IN_H 1 + +/* Define to 1 if you have the `regcomp' function. */ +#define HAVE_REGCOMP 1 + +/* Define to 1 if you have the regex.h header file. */ +#define HAVE_REGEX_H 1 + +/* Define to 1 if you have the stddef.h header file. */ +#define HAVE_STDDEF_H 1 + +/* Define to 1 if you have the stdint.h header file. */ +#define HAVE_STDINT_H 1 + +/* Define to 1 if you have the stdio.h header file. */ +#define HAVE_STDIO_H 1 + +/* Define to 1 if you have the stdlib.h header file. */ +#define HAVE_STDLIB_H 1 + +/* Define to 1 if you have the `strchr' function. */ +#define HAVE_STRCHR 1 + +/* Define to 1 if you have the `strdup' function. */ +#define HAVE_STRDUP 1 + +/* Define to 1 if you have the strings.h header file. */ +#define HAVE_STRINGS_H 1 + +/* Define to 1 if you have the string.h header file. */ +#define HAVE_STRING_H 1 + +/* Define to 1 if you have the `strstr' function. */ +#define HAVE_STRSTR 1 + +/* Define to 1 if you have the sys/socket.h header file. */ +#define HAVE_SYS_SOCKET_H 1 + +/* Define to 1 if you have the sys/stat.h header file. */ +#define HAVE_SYS_STAT_H 1 + +/* Define to 1 if you have the sys/types.h header file. */ +#define HAVE_SYS_TYPES_H 1 + +/* Define to 1 if you have the unistd.h header file. */ +#define HAVE_UNISTD_H 1 + +/* Name of package */ +#define PACKAGE udp-filters + +/* Define to the address where bug reports for this package should be sent. */ +#define PACKAGE_BUGREPORT dvanli...@wikimedia.org + +/* Define to the full name of this package. */ +#define PACKAGE_NAME udp-filters + +/* Define to the full name and version of this package. */ +#define PACKAGE_STRING udp-filters DEBIANIZE_REPLACES_THIS_VERSION + +/* Define to the one symbol short name of this package. */ +#define PACKAGE_TARNAME udp-filters + +/* Define to the home page for this package. */ +#define PACKAGE_URL + +/* Define to the version of this package. */ +#define PACKAGE_VERSION DEBIANIZE_REPLACES_THIS_VERSION + +/* Define to 1 if you have the ANSI C header files. */ +#define STDC_HEADERS 1 + +/* Version number of package */ +#define VERSION DEBIANIZE_REPLACES_THIS_VERSION + +/* Define to empty if `const' does not conform to ANSI C. */ +/* #undef const */ + +/* Define to rpl_malloc if the replacement function should be used. */ +/* #undef malloc */ + +/* Define to the equivalent of the C99 'restrict' keyword, or to + nothing if this is not supported. Do not define if restrict is + supported directly. */ +#define restrict __restrict +/* Work around a bug in Sun C++: it does not support _Restrict or + __restrict__, even though the corresponding Sun C compiler ends up with + #define restrict _Restrict or #define restrict __restrict__ in the + previous line. Perhaps some future version of Sun C++ will work with + restrict; if so, hopefully it defines __RESTRICT like Sun C does. */ +#if defined __SUNPRO_CC !defined __RESTRICT +# define _Restrict +# define __restrict__ +#endif + +/* Define to `unsigned int' if sys/types.h does not define. */ +/* #undef size_t */ -- To view, visit https://gerrit.wikimedia.org/r/60222 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: merged Gerrit-Change-Id:
[MediaWiki-commits] [Gerrit] Minor cleanup in various files - change (analytics/udp-filters)
Diederik has submitted this change and it was merged.
Change subject: Minor cleanup in various files
..
Minor cleanup in various files
Changed 1 to true in a couple of places that I missed earlier.
-- Add SimpleMake makefile to build without needing autoconf, automake and
friends.
-- countries.{h,c}: fix mismatch in array declaration and add const;
remove superfluous array size; include header in C file so that
mismatches can be caught by compiler.
-- udp-filter.h: remove explicit assignment of values since enums, by default
start at 0 and increment by 1; add last member to track number of values.
-- udp-filter.c: use MAX_FILTER instead of manifest constant; remove
superfluous variable num_predefined_filters; use bool type where appropriate;
remove unused variable found_multiple_ips.
-- utils.c: include header in C file
Change-Id: Id47ec9a4e4855d43e632ea0d71ef83323a30413f
---
A SimpleMake
M src/countries.c
M src/countries.h
M src/udp-filter.c
M src/udp-filter.h
M src/utils.c
6 files changed, 79 insertions(+), 39 deletions(-)
Approvals:
Diederik: Verified; Looks good to me, approved
diff --git a/SimpleMake b/SimpleMake
new file mode 100644
index 000..0239501
--- /dev/null
+++ b/SimpleMake
@@ -0,0 +1,40 @@
+CC = gcc
+CFLAGS = -std=gnu99 -O2 -Wall -Wextra
+LDFLAGS = -lGeoIP -lcidr -lanon
+
+TARGETS = udp-filter
+
+HDRS = src/udp-filter.h \
+ src/countries.h\
+ src/collector-output.h \
+ src/geo.h \
+ src/anonymize.h\
+ src/match.h\
+ src/utils.h
+
+SRCS = src/udp-filter.c \
+ src/countries.c\
+ src/collector-output.c \
+ src/geo.c \
+ src/anonymize.c\
+ src/match.c\
+ src/utils.c
+
+OBJS = $(SRCS:%.c=%.o)
+
+$(TARGETS): $(OBJS)
+ $(CC) -o $@ $^ $(LDFLAGS)
+
+clean:
+ rm -f udp-filter $(OBJS)
+
+.PHONY: clean
+
+# dependencies
+src/udp-filter.o: src/udp-filter.c $(HDRS)
+src/countries.o: src/countries.c src/countries.h
+src/collector-output.o: src/collector-output.c src/collector-output.h
+src/geo.o: src/geo.c src/geo.h src/udp-filter.h src/utils.h
+src/anonymize.o: src/anonymize.c src/anonymize.h src/udp-filter.h
+src/match.o: src/match.c src/match.h src/udp-filter.h src/utils.h
+src/utils.o: src/utils.c src/utils.h
diff --git a/src/countries.c b/src/countries.c
index f9e7440..9e40fd1 100644
--- a/src/countries.c
+++ b/src/countries.c
@@ -19,9 +19,11 @@
#include string.h
#include stdio.h
+#include countries.h
+
// This is an array with valid country codes, the country codes that are
// supplied on the command line will be checked against this array.
-const char *country_code_array[253] = {
+const char *const country_code_array[] = {
A1,//,Anonymous Proxy
A2,//,Satellite Provider
O1,//,Other Country
diff --git a/src/countries.h b/src/countries.h
index 320963c..36c3573 100644
--- a/src/countries.h
+++ b/src/countries.h
@@ -19,5 +19,4 @@
int verify_country_code(char *country_code);
-
-extern const char country_code_array[253];
+extern const char *const country_code_array[];
diff --git a/src/udp-filter.c b/src/udp-filter.c
index a7e7677..8c44f9d 100644
--- a/src/udp-filter.c
+++ b/src/udp-filter.c
@@ -68,8 +68,7 @@
-int params[7]; // Increase this when you add a new filter to ScreenType enum.
-const int num_predefined_filters = sizeof(params)/sizeof(int);
+bool params[ MAX_FILTER ];
int verbose_flag = 0; // this flag indicates whether we
should output detailed debug messages, default is off.
@@ -653,32 +652,32 @@
// DETERMINE NUMBER OF FILTERS
- for(n=0; nnum_predefined_filters; n++){
+ for(n=0; nMAX_FILTER; n++){
switch (n) {
case DOMAIN_FILTER:
- if(params[n] == 1){
+ if(params[n] == true){
num_domain_filters =
determine_num_obs(domain_input,comma_delimiter);
required_hits+=1;
}
break;
case PATH_FILTER:
- if(params[n] ==1 ){
+ if(params[n] == true ){
num_path_filters =
determine_num_obs(path_input,comma_delimiter);
required_hits+=1;
}
break;
case IP_FILTER:
- if(params[n] == 1){
+ if(params[n] == true){
num_ipaddress_filters =
determine_num_obs(ipaddress_input, comma_delimiter);
required_hits+=1;
}
break;
case GEO_FILTER:
- if(params[n] == 1){
+ if(params[n] ==
[MediaWiki-commits] [Gerrit] Add config.h to repo - change (analytics/udp-filters)
Diederik has uploaded a new change for review. https://gerrit.wikimedia.org/r/60222 Change subject: Add config.h to repo .. Add config.h to repo Change-Id: I6edb4a2fdee753691849039d18dd88a0184610dd --- A config.h 1 file changed, 137 insertions(+), 0 deletions(-) git pull ssh://gerrit.wikimedia.org:29418/analytics/udp-filters refs/changes/22/60222/1 diff --git a/config.h b/config.h new file mode 100644 index 000..0ef4757 --- /dev/null +++ b/config.h @@ -0,0 +1,137 @@ +/* config.h. Generated from config.h.in by configure. */ +/* config.h.in. Generated from configure.ac by autoheader. */ + +/* Define to 1 if you have the arpa/inet.h header file. */ +#define HAVE_ARPA_INET_H 1 + +/* Define to 1 if you have the ctype.h header file. */ +#define HAVE_CTYPE_H 1 + +/* Define to 1 if you have the GeoIPCity.h header file. */ +#define HAVE_GEOIPCITY_H 1 + +/* Define to 1 if you have the GeoIP.h header file. */ +#define HAVE_GEOIP_H 1 + +/* Define to 1 if you have the getopt.h header file. */ +#define HAVE_GETOPT_H 1 + +/* Define to 1 if you have the inttypes.h header file. */ +#define HAVE_INTTYPES_H 1 + +/* Define to 1 if you have the libcidr.h header file. */ +/* #undef HAVE_LIBCIDR_H */ + +/* Define to 1 if you have the `geoip' library (-lgeoip). */ +#define HAVE_LIBGEOIP 1 + +/* Define to 1 if your system has a GNU libc compatible `malloc' function, and + to 0 otherwise. */ +#define HAVE_MALLOC 1 + +/* Define to 1 if you have the memory.h header file. */ +#define HAVE_MEMORY_H 1 + +/* Define to 1 if you have the `memset' function. */ +#define HAVE_MEMSET 1 + +/* Define to 1 if you have the netdb.h header file. */ +#define HAVE_NETDB_H 1 + +/* Define to 1 if you have the netinet/in.h header file. */ +#define HAVE_NETINET_IN_H 1 + +/* Define to 1 if you have the `regcomp' function. */ +#define HAVE_REGCOMP 1 + +/* Define to 1 if you have the regex.h header file. */ +#define HAVE_REGEX_H 1 + +/* Define to 1 if you have the stddef.h header file. */ +#define HAVE_STDDEF_H 1 + +/* Define to 1 if you have the stdint.h header file. */ +#define HAVE_STDINT_H 1 + +/* Define to 1 if you have the stdio.h header file. */ +#define HAVE_STDIO_H 1 + +/* Define to 1 if you have the stdlib.h header file. */ +#define HAVE_STDLIB_H 1 + +/* Define to 1 if you have the `strchr' function. */ +#define HAVE_STRCHR 1 + +/* Define to 1 if you have the `strdup' function. */ +#define HAVE_STRDUP 1 + +/* Define to 1 if you have the strings.h header file. */ +#define HAVE_STRINGS_H 1 + +/* Define to 1 if you have the string.h header file. */ +#define HAVE_STRING_H 1 + +/* Define to 1 if you have the `strstr' function. */ +#define HAVE_STRSTR 1 + +/* Define to 1 if you have the sys/socket.h header file. */ +#define HAVE_SYS_SOCKET_H 1 + +/* Define to 1 if you have the sys/stat.h header file. */ +#define HAVE_SYS_STAT_H 1 + +/* Define to 1 if you have the sys/types.h header file. */ +#define HAVE_SYS_TYPES_H 1 + +/* Define to 1 if you have the unistd.h header file. */ +#define HAVE_UNISTD_H 1 + +/* Name of package */ +#define PACKAGE udp-filters + +/* Define to the address where bug reports for this package should be sent. */ +#define PACKAGE_BUGREPORT dvanli...@wikimedia.org + +/* Define to the full name of this package. */ +#define PACKAGE_NAME udp-filters + +/* Define to the full name and version of this package. */ +#define PACKAGE_STRING udp-filters DEBIANIZE_REPLACES_THIS_VERSION + +/* Define to the one symbol short name of this package. */ +#define PACKAGE_TARNAME udp-filters + +/* Define to the home page for this package. */ +#define PACKAGE_URL + +/* Define to the version of this package. */ +#define PACKAGE_VERSION DEBIANIZE_REPLACES_THIS_VERSION + +/* Define to 1 if you have the ANSI C header files. */ +#define STDC_HEADERS 1 + +/* Version number of package */ +#define VERSION DEBIANIZE_REPLACES_THIS_VERSION + +/* Define to empty if `const' does not conform to ANSI C. */ +/* #undef const */ + +/* Define to rpl_malloc if the replacement function should be used. */ +/* #undef malloc */ + +/* Define to the equivalent of the C99 'restrict' keyword, or to + nothing if this is not supported. Do not define if restrict is + supported directly. */ +#define restrict __restrict +/* Work around a bug in Sun C++: it does not support _Restrict or + __restrict__, even though the corresponding Sun C compiler ends up with + #define restrict _Restrict or #define restrict __restrict__ in the + previous line. Perhaps some future version of Sun C++ will work with + restrict; if so, hopefully it defines __RESTRICT like Sun C does. */ +#if defined __SUNPRO_CC !defined __RESTRICT +# define _Restrict +# define __restrict__ +#endif + +/* Define to `unsigned int' if sys/types.h does not define. */ +/* #undef size_t */ -- To view, visit https://gerrit.wikimedia.org/r/60222 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
[MediaWiki-commits] [Gerrit] Disable final two Fundraising filters as they have moved to ... - change (operations/puppet)
Diederik has uploaded a new change for review. https://gerrit.wikimedia.org/r/60030 Change subject: Disable final two Fundraising filters as they have moved to Gadolinium .. Disable final two Fundraising filters as they have moved to Gadolinium Change-Id: I36d8c600bc02c54f201b23a2cbc30d58643344ca --- M templates/udp2log/filters.locke.erb 1 file changed, 2 insertions(+), 2 deletions(-) git pull ssh://gerrit.wikimedia.org:29418/operations/puppet refs/changes/30/60030/1 diff --git a/templates/udp2log/filters.locke.erb b/templates/udp2log/filters.locke.erb index f44ad75..233adef 100644 --- a/templates/udp2log/filters.locke.erb +++ b/templates/udp2log/filters.locke.erb @@ -7,10 +7,10 @@ ## Fundraising # Landing pages -pipe 1 /usr/bin/udp-filter -F '\t' -d wikimediafoundation.org,donate.wikimedia.org /a/squid/fundraising/logs/landingpages.tab.log +#pipe 1 /usr/bin/udp-filter -F '\t' -d wikimediafoundation.org,donate.wikimedia.org /a/squid/fundraising/logs/landingpages.tab.log # Banner Impressions -pipe 100 /usr/bin/udp-filter -F '\t' -p Special:RecordImpression\?banner=,Special:RecordImpression\?result= /a/squid/fundraising/logs/bannerImpressions-sampled100.tab.log +#pipe 100 /usr/bin/udp-filter -F '\t' -p Special:RecordImpression\?banner=,Special:RecordImpression\?result= /a/squid/fundraising/logs/bannerImpressions-sampled100.tab.log -- To view, visit https://gerrit.wikimedia.org/r/60030 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I36d8c600bc02c54f201b23a2cbc30d58643344ca Gerrit-PatchSet: 1 Gerrit-Project: operations/puppet Gerrit-Branch: production Gerrit-Owner: Diederik dvanli...@wikimedia.org ___ MediaWiki-commits mailing list MediaWiki-commits@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] Email serious errors to dev's - change (analytics/E3Analysis)
Diederik has uploaded a new change for review.
https://gerrit.wikimedia.org/r/60044
Change subject: Email serious errors to dev's
..
Email serious errors to dev's
Change-Id: Ib439d014131e9efee37f20df0b8db290a51f66ef
---
M .gitignore
M user_metrics/api/run.py
2 files changed, 25 insertions(+), 1 deletion(-)
git pull ssh://gerrit.wikimedia.org:29418/analytics/E3Analysis
refs/changes/44/60044/1
diff --git a/.gitignore b/.gitignore
index 2db5e13..a42cd08 100644
--- a/.gitignore
+++ b/.gitignore
@@ -3,7 +3,7 @@
MANIFEST
.DS_Store
.idea
-
+*.iml
# ignore docs and build sub-folders
build/*
diff --git a/user_metrics/api/run.py b/user_metrics/api/run.py
index 4a6dc3c..5997052 100644
--- a/user_metrics/api/run.py
+++ b/user_metrics/api/run.py
@@ -109,6 +109,30 @@
app.config['SECRET_KEY'] = settings.__secret_key__
+#Send serious errors to devs
+ADMINS = ['dvanli...@wikimedia.org', 'rfaulk...@wikimedia.org']
+if not app.debug:
+import logging
+from logging.handlers import SMTPHandler
+from logging import Formatter
+
+mail_handler = SMTPHandler('127.0.0.1',
+ 'no-re...@user-metrics.wikimedia.org',
+ ADMINS, 'UserMetrics API encountered serious
error')
+mail_handler.setLevel(logging.ERROR)
+mail_handler.setFormatter(Formatter('''
+Message type: %(levelname)s
+Location: %(pathname)s:%(lineno)d
+Module: %(module)s
+Function: %(funcName)s
+Time: %(asctime)s
+
+Message:
+
+%(message)s
+'''))
+app.logger.addHandler(mail_handler)
+
# With the presence of flask.ext.login module
if settings.__flask_login_exists__:
from user_metrics.api.session import login_manager
--
To view, visit https://gerrit.wikimedia.org/r/60044
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: Ib439d014131e9efee37f20df0b8db290a51f66ef
Gerrit-PatchSet: 1
Gerrit-Project: analytics/E3Analysis
Gerrit-Branch: master
Gerrit-Owner: Diederik dvanli...@wikimedia.org
___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] Make sure that a lock is always released. - change (analytics/E3Analysis)
Diederik has uploaded a new change for review. https://gerrit.wikimedia.org/r/59842 Change subject: Make sure that a lock is always released. .. Make sure that a lock is always released. Change-Id: Ifbd7972a738381d23d23a62cd645a7c25b8f32df --- M user_metrics/api/engine/request_manager.py 1 file changed, 10 insertions(+), 7 deletions(-) git pull ssh://gerrit.wikimedia.org:29418/analytics/E3Analysis refs/changes/42/59842/1 diff --git a/user_metrics/api/engine/request_manager.py b/user_metrics/api/engine/request_manager.py index cd421fa..3d39aff 100644 --- a/user_metrics/api/engine/request_manager.py +++ b/user_metrics/api/engine/request_manager.py @@ -615,8 +615,9 @@ except Empty: logging.error(__name__ + ' :: req_cb_get_url -' ' Block time expired.') -return '' -lock.release() +val = '' +finally: +lock.release() return val @@ -629,8 +630,9 @@ except Empty: logging.error(__name__ + ' :: req_cb_get_cache_keys -' ' Block time expired.') -return [] -lock.release() +val = [] +finally: +lock.release() return val @@ -643,8 +645,9 @@ except Empty: logging.error(__name__ + ' :: req_cb_get_is_running -' ' Block time expired.') -return False -lock.release() +val = False +finally: +lock.release() return val @@ -657,4 +660,4 @@ def req_cb_flag_job_complete(key, lock): lock.acquire() req_notification_queue_in.put([1, key], True) -lock.release() \ No newline at end of file +lock.release() -- To view, visit https://gerrit.wikimedia.org/r/59842 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: newchange Gerrit-Change-Id: Ifbd7972a738381d23d23a62cd645a7c25b8f32df Gerrit-PatchSet: 1 Gerrit-Project: analytics/E3Analysis Gerrit-Branch: master Gerrit-Owner: Diederik dvanli...@wikimedia.org ___ MediaWiki-commits mailing list MediaWiki-commits@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] mod. except clause too broad - pass on empty queue only. - change (analytics/E3Analysis)
Diederik has submitted this change and it was merged. Change subject: mod. except clause too broad - pass on empty queue only. .. mod. except clause too broad - pass on empty queue only. Change-Id: Ibcd1ee38fbae1b9a23dbd28445f2ae21f83fb95d --- M user_metrics/api/engine/request_manager.py 1 file changed, 2 insertions(+), 2 deletions(-) Approvals: Diederik: Verified; Looks good to me, approved diff --git a/user_metrics/api/engine/request_manager.py b/user_metrics/api/engine/request_manager.py index 205c661..b48434d 100644 --- a/user_metrics/api/engine/request_manager.py +++ b/user_metrics/api/engine/request_manager.py @@ -655,7 +655,7 @@ lock.acquire() try: req_notification_queue_in.put([0, key, url]) -except: +except Empty: pass finally: lock.release() @@ -665,7 +665,7 @@ lock.acquire() try: req_notification_queue_in.put([1, key], True) -except: +except Empty: pass finally: lock.release() -- To view, visit https://gerrit.wikimedia.org/r/59867 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: merged Gerrit-Change-Id: Ibcd1ee38fbae1b9a23dbd28445f2ae21f83fb95d Gerrit-PatchSet: 1 Gerrit-Project: analytics/E3Analysis Gerrit-Branch: master Gerrit-Owner: Rfaulk ryan.faulk...@mail.mcgill.ca Gerrit-Reviewer: Diederik dvanli...@wikimedia.org Gerrit-Reviewer: Milimetric dandree...@wikimedia.org ___ MediaWiki-commits mailing list MediaWiki-commits@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] Remove class 'analinterns' it is no longer used. - change (operations/puppet)
Diederik has uploaded a new change for review.
https://gerrit.wikimedia.org/r/59135
Change subject: Remove class 'analinterns' it is no longer used.
..
Remove class 'analinterns' it is no longer used.
Change-Id: I53514b601cce4c7aed32abd6b5eefee383e205c2
---
M manifests/admins.pp
1 file changed, 0 insertions(+), 13 deletions(-)
git pull ssh://gerrit.wikimedia.org:29418/operations/puppet
refs/changes/35/59135/1
diff --git a/manifests/admins.pp b/manifests/admins.pp
index f7fd6c9..920928b 100644
--- a/manifests/admins.pp
+++ b/manifests/admins.pp
@@ -2810,19 +2810,6 @@
include accounts::yurik #rt 4835
}
-class admins::analinterns {
- $gid = 500 # 'wikidev' by default
- include groups::wikidev
-
- include accounts::declerambaul
- include accounts::diederik
- include accounts::halfak
- include accounts::spetrea
- include accounts::shawn # disabled
- include accounts::whym # disabled
-
-}
-
class admins::jenkins {
$gid = 500 # 'wikidev' by default
include groups::wikidev
--
To view, visit https://gerrit.wikimedia.org/r/59135
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: I53514b601cce4c7aed32abd6b5eefee383e205c2
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Diederik dvanli...@wikimedia.org
___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] Changed scope for junit from test to compile so mvn package ... - change (mediawiki...mwdumper)
Diederik has uploaded a new change for review. https://gerrit.wikimedia.org/r/59093 Change subject: Changed scope for junit from test to compile so mvn package actually works .. Changed scope for junit from test to compile so mvn package actually works Real fix is of course to make mwdumper comply with the maven folder structure. The problem was caused by the fact that test and regular source code files are in the same folder. Also added some files to .gitignore Change-Id: Id984bf84fdbafefec894d976f4e6dd00f561 --- M .gitignore M pom.xml 2 files changed, 3 insertions(+), 1 deletion(-) git pull ssh://gerrit.wikimedia.org:29418/mediawiki/tools/mwdumper refs/changes/93/59093/1 diff --git a/.gitignore b/.gitignore index f61aa0d..d926e62 100644 --- a/.gitignore +++ b/.gitignore @@ -2,3 +2,5 @@ .settings .project target/ +.idea/ +mwdumper.iml diff --git a/pom.xml b/pom.xml index 66bd5bc..513fc7e 100644 --- a/pom.xml +++ b/pom.xml @@ -53,7 +53,7 @@ groupIdjunit/groupId artifactIdjunit/artifactId version3.8/version - scopetest/scope + scopecompile/scope /dependency /dependencies -- To view, visit https://gerrit.wikimedia.org/r/59093 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: newchange Gerrit-Change-Id: Id984bf84fdbafefec894d976f4e6dd00f561 Gerrit-PatchSet: 1 Gerrit-Project: mediawiki/tools/mwdumper Gerrit-Branch: master Gerrit-Owner: Diederik dvanli...@wikimedia.org ___ MediaWiki-commits mailing list MediaWiki-commits@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] Disable Fabian Kaelin's account, he no longer works with us. - change (operations/puppet)
Diederik has uploaded a new change for review.
https://gerrit.wikimedia.org/r/58904
Change subject: Disable Fabian Kaelin's account, he no longer works with us.
..
Disable Fabian Kaelin's account, he no longer works with us.
Change-Id: If9635e52c485053c41840328fae9d934bf15274e
---
M manifests/admins.pp
1 file changed, 2 insertions(+), 1 deletion(-)
git pull ssh://gerrit.wikimedia.org:29418/operations/puppet
refs/changes/04/58904/1
diff --git a/manifests/admins.pp b/manifests/admins.pp
index c3a70d2..f7fd6c9 100644
--- a/manifests/admins.pp
+++ b/manifests/admins.pp
@@ -1752,6 +1752,7 @@
$username = declerambaul
$realname = Fabian Kaelin
$uid = 566
+ $enabled = false
unixaccount { $realname: username = $username, uid = $uid,
gid = $gid }
@@ -1760,7 +1761,7 @@
ssh_authorized_key {
declerambaul_wiki:
- ensure = present,
+ ensure = absent,
user= $username,
type= ssh-rsa,
key =
B3NzaC1yc2EBIwAAAQEA3aScUQs0HeQEp/pnMuJ6JKWTwEa8IPVKa7Gkrx2DJ3z1qUtA5wzxSG6m//MJBokIntwBAGuqRazAqs5cB3m4GVqViA1fabxZy55l1/GB+962/d8goVEbtkj/MO47vuUBosVSy5GGjGOs3hWtKId9q6+AU0OCNZwC3j12tXGIX3ztcf4Ef2pdBoCfJMgrvlnIpdFDBftrua9kVvYRQj6tVr5rTbFlEioNgNcdQXhvDP0sU81i1NG/nAeOZMDYOzUscDHa6JcCts3nRyqrqgaMixxGjF7WG42tqS+AEqKi9IOqFnaiHtwipZrnJJ8IxtDOve3HHA3VctBsh03qB4RZ6w==;
--
To view, visit https://gerrit.wikimedia.org/r/58904
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: If9635e52c485053c41840328fae9d934bf15274e
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Diederik dvanli...@wikimedia.org
___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] cleanup duplicate content in UDP-FILTER(1) - change (analytics/udp-filters)
Diederik has submitted this change and it was merged. Change subject: cleanup duplicate content in UDP-FILTER(1) .. cleanup duplicate content in UDP-FILTER(1) Change-Id: I6d663e33e46a6ed9db859f02e298fd25c9d4c002 --- M docs/udp-filter.1 1 file changed, 4 insertions(+), 25 deletions(-) Approvals: Diederik: Verified; Looks good to me, approved diff --git a/docs/udp-filter.1 b/docs/udp-filter.1 index 3729bb0..6e094f7 100644 --- a/docs/udp-filter.1 +++ b/docs/udp-filter.1 @@ -145,36 +145,15 @@ Turns on verbose logging and can help in debugging and verifying that the appropriate hits are being logged. This setting is not recommended in production. +.PP +One of \fB\-\-path\fR, \fB\-\-ip\fR or \fB\-\-domain\fR are mandatory. +You can use all three, but you must give at least one of them. .SH BUGS No known bugs, but let me know if you find any. .SH AUTHOR -Written by Diederik van Liere (dvanli...@wikimedia.org). -udp\-filter 0.2.6 -.PP -Wikimedia's generic webserver access log filtering system. -This new filter system replaces the old collection of udp2log filters written in C. -It is customizable and can be configured using the command line. -.PP -Written by Diederik van Liere (dvanli...@wikimedia.org). -.TP -One of \fB\-\-path\fR, \fB\-\-ip\fR or \fB\-\-domain\fR are mandatory. -You can use all three, but you must give at least one of them. -.PP Written by Diederik van Liere (dvanli...@wikimedia.org). .SH COPYRIGHT Copyright \(co 2012 Wikimedia Foundation, Inc. .br This is free software; see the source copying conditions. There is NO -warrant; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. -.PP -.br -Copyright \(co 2012 Wikimedia Foundation, Inc. -.br -This is free software; see the source copying conditions. There is NO -warrant; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. -.PP -.br -Copyright \(co 2012 Wikimedia Foundation, Inc. -.br -This is free software; see the source copying conditions. There is NO -warrant; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. +warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. -- To view, visit https://gerrit.wikimedia.org/r/57395 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: merged Gerrit-Change-Id: I6d663e33e46a6ed9db859f02e298fd25c9d4c002 Gerrit-PatchSet: 1 Gerrit-Project: analytics/udp-filters Gerrit-Branch: master Gerrit-Owner: Jeremyb jer...@tuxmachine.com Gerrit-Reviewer: Diederik dvanli...@wikimedia.org Gerrit-Reviewer: Ottomata o...@wikimedia.org ___ MediaWiki-commits mailing list MediaWiki-commits@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] Dropping all search and preview url's - change (operations/puppet)
Diederik has uploaded a new change for review.
https://gerrit.wikimedia.org/r/56633
Change subject: Dropping all search and preview url's
..
Dropping all search and preview url's
Change-Id: Ib7911caf7088414392b8ad0e8f63c0e298f77008
---
M templates/misc/email-blog-pageviews.erb
1 file changed, 2 insertions(+), 2 deletions(-)
git pull ssh://gerrit.wikimedia.org:29418/operations/puppet
refs/changes/33/56633/1
diff --git a/templates/misc/email-blog-pageviews.erb
b/templates/misc/email-blog-pageviews.erb
index ae655e8..1cf2535 100644
--- a/templates/misc/email-blog-pageviews.erb
+++ b/templates/misc/email-blog-pageviews.erb
@@ -12,8 +12,8 @@
SUBSTRING_INDEX(SUBSTRING_INDEX(SUBSTRING_INDEX(event_referrerUrl, '/',
3), '/', -1), '.', -2) AS referer
FROM WikimediaBlogVisit_5308166
WHERE DATE(timestamp)=DATE('$DATE')
- AND event_requestUrl NOT LIKE '/?s=%'
- AND event_requestUrl NOT LIKE '/?p=%'
+ AND event_requestUrl NOT LIKE '%?s=%'
+ AND event_requestUrl NOT LIKE '%?p=%'
GROUP BY
event_requestUrl, referer
ORDER BY
--
To view, visit https://gerrit.wikimedia.org/r/56633
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: Ib7911caf7088414392b8ad0e8f63c0e298f77008
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Diederik dvanli...@wikimedia.org
___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] Adding David and Dan to Analytics contact group, removing Fa... - change (operations/puppet)
Diederik has uploaded a new change for review.
https://gerrit.wikimedia.org/r/53624
Change subject: Adding David and Dan to Analytics contact group, removing
Fabrice.
..
Adding David and Dan to Analytics contact group, removing Fabrice.
Change-Id: I4abe7cd544d45d28cd831fbb435e7883cbcc9b73
---
M files/nagios/contactgroups.cfg
1 file changed, 1 insertion(+), 1 deletion(-)
git pull ssh://gerrit.wikimedia.org:29418/operations/puppet
refs/changes/24/53624/1
diff --git a/files/nagios/contactgroups.cfg b/files/nagios/contactgroups.cfg
index 4a2b392..57b948e 100644
--- a/files/nagios/contactgroups.cfg
+++ b/files/nagios/contactgroups.cfg
@@ -21,5 +21,5 @@
define contactgroup {
contactgroup_name analytics
- members dvanliere,ezachte,dtaraborelli,fflorin,otto
+ members
dvanliere,ezachte,dtaraborelli,otto,dsc,dandreescu
}
--
To view, visit https://gerrit.wikimedia.org/r/53624
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: I4abe7cd544d45d28cd831fbb435e7883cbcc9b73
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Diederik dvanli...@wikimedia.org
___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] Fixed segfault - change (analytics/webstatscollector)
Diederik has submitted this change and it was merged.
Change subject: Fixed segfault
..
Fixed segfault
Change-Id: I873600aac2b3f0e45bc3a5f77404f1c2da8fbd1d
---
M filter.c
1 file changed, 5 insertions(+), 2 deletions(-)
Approvals:
Diederik: Verified; Looks good to me, approved
diff --git a/filter.c b/filter.c
index 68daf4b..169a543 100644
--- a/filter.c
+++ b/filter.c
@@ -93,6 +93,7 @@
void replace_space(char *url) {
int len = strlen(url);
+
if (len==0) {
return;
}
@@ -102,7 +103,7 @@
if(url[i] == ' '){
url[i] = '_';
}
-}
+ }
}
bool parse_url(char *url, struct info *in) {
@@ -176,7 +177,9 @@
info.size= FIELD; /* object size */
FIELD;
url=FIELD;
-replace_space(url);
+if(!url)
+ continue;
+replace_space(url);
if (!check_ip(info.ip))
continue;
if (!parse_url(url,info))
--
To view, visit https://gerrit.wikimedia.org/r/52549
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: merged
Gerrit-Change-Id: I873600aac2b3f0e45bc3a5f77404f1c2da8fbd1d
Gerrit-PatchSet: 1
Gerrit-Project: analytics/webstatscollector
Gerrit-Branch: time_travel
Gerrit-Owner: Stefan.petrea ste...@garage-coding.com
Gerrit-Reviewer: Diederik dvanli...@wikimedia.org
___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] Add replace_space function - change (analytics/webstatscollector)
Diederik has submitted this change and it was merged. Change subject: Add replace_space function .. Add replace_space function Added tests, added test file, fixed replace_space Change-Id: Ic795c876105fe8a1a980f83a0c07a5ba2011fff5 --- M Makefile A README.tests A entries-with-urls-with-spaces-2013-02-10.txt M filter.c A test.sh 5 files changed, 80 insertions(+), 35 deletions(-) Approvals: Diederik: Verified; Looks good to me, approved diff --git a/Makefile b/Makefile index b05321b..25451a7 100644 --- a/Makefile +++ b/Makefile @@ -14,12 +14,14 @@ all: collector filter -collector: collector.h collector.c export.c +collector: collector.h collector.c export.c export.o + gcc -o collector collector.c export.o -ldb -lpthread filter: filter.c - cc -o filter filter.c + gcc -o filter filter.c -#export: collector.h export.c +export.o: export.c collector.h collector.c filter.c + gcc -c -o export.o export.c clean: - rm -f collector exporter + rm -f *.o collector filter diff --git a/README.tests b/README.tests new file mode 100644 index 000..c14927e --- /dev/null +++ b/README.tests @@ -0,0 +1,7 @@ + + +The test lines in entries-with-urls-with-spaces-2013-02-10.txt were produced like this: + + zcat /home/user/wikidata/raw_gzips/sampled-1000.tab.log-20130210.gz | perl -ne '@f=split(/\t/); print if index($f[8], )!=-1;' entries-with-urls-with-spaces-2013-02-10.txt + +After this head -40 | tail -30 was applied in order for the filter to accept all of them (some of them were influenced by some the discarding rules of the filter). diff --git a/entries-with-urls-with-spaces-2013-02-10.txt b/entries-with-urls-with-spaces-2013-02-10.txt new file mode 100644 index 000..abb6648 --- /dev/null +++ b/entries-with-urls-with-spaces-2013-02-10.txt @@ -0,0 +1,30 @@ +ssl1002338866312 2013-02-09T10:00:09.126 1.667 0.0.0.0 FAKE_CACHE_STATUS/301 667 GET https://fr.wikipedia.org/wiki/Discussion:Bourail/Droit d'auteur NONE/wikipedia - - - Mozilla/5.0%20(compatible;%20Googlebot/2.1;%20+http://www.google.com/bot.html) - - +ssl1004347282202 2013-02-09T10:18:11.367 0.082 0.0.0.0 FAKE_CACHE_STATUS/301 675 GET https://fr.wikipedia.org/wiki/Discussion:Chemin\xC3\xA9e solaire/Traduction NONE/wikipedia - - - Mozilla/5.0%20(compatible;%20Googlebot/2.1;%20+https://www.google.com/bot.html) - - +ssl1004349592952 2013-02-09T12:19:04.972 0.079 0.0.0.0 FAKE_CACHE_STATUS/301 667 GET https://fr.wikipedia.org/wiki/\xC3\x89tienne Perrot (psychanalyste) NONE/wikipedia - - - Mozilla/5.0%20(compatible;%20Googlebot/2.1;%20+http://www.google.com/bot.html) - - +ssl1001338735382 2013-02-09T12:57:51.173 0.201 0.0.0.0 FAKE_CACHE_STATUS/301 759 GET http://ja.wikipedia.org/wiki/CODE NAME. 1 Brother Sun\xE3\x80\x90\xE5\x88\x9D\xE5\x9B\x9E\xE7\x94\x9F\xE7\x94\xA3\xE9\x99\x90\xE5\xAE\x9A\xE3\x80\x91(\xE7\xB4\x99\xE3\x82\xB8\xE3\x83\xA3\xE3\x82\xB1\xE3\x83\x83\xE3\x83\x88\xE4\xBB\x95\xE6\xA7\x98) NONE/wikipedia - - - foobar2000/1.1.14a - - +ssl1001340768664 2013-02-09T14:29:25.274 0.351 0.0.0.0 FAKE_CACHE_STATUS/301 885 GET http://ja.wikipedia.org/wiki/\xE7\x89\xB9\xE5\x88\xA5:\xE3\x83\x87\xE3\x83\xBC\xE3\x82\xBF\xE6\x9B\xB8\xE3\x81\x8D\xE5\x87\xBA\xE3\x81\x97/\xE3\x82\x82\xE3\x82\x82\xE3\x81\x84\xE3\x82\x8D\xE3\x82\xAF\xE3\x83\xAD\xE3\x83\xBC\xE3\x83\x90\xE3\x83\xBCZ 1st Live \xE3\x81\xAB\xE3\x83\x95\xE3\x83\xA9\xE3\x83\xB3\xE3\x82\xB9\xE4\xBA\xBA\xE5\xA4\xA7\xE8\x88\x88\xE5\xA5\xAE\xEF\xBC\x81\xE3\x83\x91\xE3\x83\xAAde Japan Expo NONE/wikipedia - - - Mozilla/4.0%20(compatible;%20MSIE%207.0;%20Windows%20NT%206.0) - - +ssl3002764325222 2013-02-09T14:35:55.799 0.088 0.0.0.0 FAKE_CACHE_STATUS/301 1247GET http://en.wikipedia.org/wiki/Chopin - The Piano Works (Brilliant Classics) (CD4 of 13)_(album) NONE/wikipedia - - - foobar2000/1.2 - - +ssl1002345415572 2013-02-09T15:10:39.804 0.051 0.0.0.0 FAKE_CACHE_STATUS/301 674 GET https://fr.wikipedia.org/wiki/Portail:Indianapolis/Index th\xC3\xA9matique NONE/wikipedia - - - Mozilla/5.0%20(compatible;%20Googlebot/2.1;%20+http://www.google.com/bot.html) - - +ssl3002769405948 2013-02-09T15:36:59.442 0.408 0.0.0.0 FAKE_CACHE_STATUS/301 1270GET https://en.wikipedia.org/wiki/Kevin Smith NONE/wikipedia - - - Mozilla/5.0%20(X11;%20Linux%20x86_64;%20rv:18.0)%20Gecko/20100101%20Firefox/18.0 en-US,en;q=0.5 - +ssl1004354242982 2013-02-09T15:40:02.553 0.061 0.0.0.0 FAKE_CACHE_STATUS/301 677 GET
[MediaWiki-commits] [Gerrit] Replace FAKE_CACHE_STATUS with FCS to save 14 bytes per log ... - change (operations/puppet)
Diederik has uploaded a new change for review. Change subject: Replace FAKE_CACHE_STATUS with FCS to save 14 bytes per log line .. Replace FAKE_CACHE_STATUS with FCS to save 14 bytes per log line Change-Id: I82870f34b3b0e2ded8bfe129cf6d94b42a2aad65 --- M templates/nginx/nginx.conf.erb 1 file changed, 2 insertions(+), 1 deletion(-) git pull ssh://gerrit.wikimedia.org:29418/operations/puppet refs/changes/29/48129/1 -- To view, visit https://gerrit.wikimedia.org/r/48129 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I82870f34b3b0e2ded8bfe129cf6d94b42a2aad65 Gerrit-PatchSet: 1 Gerrit-Project: operations/puppet Gerrit-Branch: production Gerrit-Owner: Diederik dvanli...@wikimedia.org ___ MediaWiki-commits mailing list MediaWiki-commits@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] Add replace_space function - change (analytics/webstatscollector)
Diederik has uploaded a new change for review. Change subject: Add replace_space function .. Add replace_space function Change-Id: I79a82ecaea96104e5e831bf139a034e020102e27 --- M filter.c 1 file changed, 22 insertions(+), 31 deletions(-) git pull ssh://gerrit.wikimedia.org:29418/analytics/webstatscollector refs/changes/27/47827/1 -- To view, visit https://gerrit.wikimedia.org/r/47827 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I79a82ecaea96104e5e831bf139a034e020102e27 Gerrit-PatchSet: 1 Gerrit-Project: analytics/webstatscollector Gerrit-Branch: time_travel Gerrit-Owner: Diederik dvanli...@wikimedia.org ___ MediaWiki-commits mailing list MediaWiki-commits@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] Enable sampling on Thank_You_Main and country filter on Emer... - change (operations/puppet[production])
Diederik has uploaded a new change for review. Change subject: Enable sampling on Thank_You_Main and country filter on Emery .. Enable sampling on Thank_You_Main and country filter on Emery Change-Id: I55f0a1d7e93c4650d045739987d3891bdd694247 --- M templates/udp2log/filters.emery.erb 1 file changed, 2 insertions(+), 2 deletions(-) git pull ssh://gerrit.wikimedia.org:29418/operations/puppet refs/changes/89/46189/1 -- To view, visit https://gerrit.wikimedia.org/r/46189 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I55f0a1d7e93c4650d045739987d3891bdd694247 Gerrit-PatchSet: 1 Gerrit-Project: operations/puppet Gerrit-Branch: production Gerrit-Owner: Diederik dvanli...@wikimedia.org ___ MediaWiki-commits mailing list MediaWiki-commits@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] Replaced custom CS carrier codes with MCC-MNC mobile carrier... - change (operations/puppet[production])
Diederik has uploaded a new change for review. Change subject: Replaced custom CS carrier codes with MCC-MNC mobile carrier codes. .. Replaced custom CS carrier codes with MCC-MNC mobile carrier codes. Change-Id: I7da99ef223c33696349a33dec03bd8d215293738 --- M templates/varnish/mobile-frontend.inc.vcl.erb 1 file changed, 27 insertions(+), 21 deletions(-) git pull ssh://gerrit.wikimedia.org:29418/operations/puppet refs/changes/53/45353/1 -- To view, visit https://gerrit.wikimedia.org/r/45353 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I7da99ef223c33696349a33dec03bd8d215293738 Gerrit-PatchSet: 1 Gerrit-Project: operations/puppet Gerrit-Branch: production Gerrit-Owner: Diederik dvanli...@wikimedia.org ___ MediaWiki-commits mailing list MediaWiki-commits@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] Updated IP blocks that should be filtered due to duplicate s... - change (analytics/webstatscollector[master])
Diederik has uploaded a new change for review. Change subject: Updated IP blocks that should be filtered due to duplicate squid requests. .. Updated IP blocks that should be filtered due to duplicate squid requests. Change-Id: I2fbdaba7f54d76cc891f50f0cf68aedcecbd6bff --- A .gitreview M filter.c 2 files changed, 23 insertions(+), 6 deletions(-) git pull ssh://gerrit.wikimedia.org:29418/analytics/webstatscollector refs/changes/83/44083/1 -- To view, visit https://gerrit.wikimedia.org/r/44083 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I2fbdaba7f54d76cc891f50f0cf68aedcecbd6bff Gerrit-PatchSet: 1 Gerrit-Project: analytics/webstatscollector Gerrit-Branch: master Gerrit-Owner: Diederik dvanli...@wikimedia.org ___ MediaWiki-commits mailing list MediaWiki-commits@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] Add Stefan Petrea as whitelisted developer - change (integration/zuul-config[master])
Diederik has uploaded a new change for review. Change subject: Add Stefan Petrea as whitelisted developer .. Add Stefan Petrea as whitelisted developer Change-Id: Idf96ce3aef7007840c0cbc8af8a0dba31a245663 --- M layout.yaml 1 file changed, 1 insertion(+), 0 deletions(-) git pull ssh://gerrit.wikimedia.org:29418/integration/zuul-config refs/changes/90/39590/1 -- To view, visit https://gerrit.wikimedia.org/r/39590 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: newchange Gerrit-Change-Id: Idf96ce3aef7007840c0cbc8af8a0dba31a245663 Gerrit-PatchSet: 1 Gerrit-Project: integration/zuul-config Gerrit-Branch: master Gerrit-Owner: Diederik dvanli...@wikimedia.org ___ MediaWiki-commits mailing list MediaWiki-commits@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] Moved message to debug mode - change (analytics/webstatscollector[master])
Diederik has submitted this change and it was merged. Change subject: Moved message to debug mode .. Moved message to debug mode Change-Id: I69f27b184419d3986a8fcfccdf9cb6d4e65e2bb6 --- M src/collector.c 1 file changed, 3 insertions(+), 1 deletion(-) Approvals: Diederik: Verified; Looks good to me, approved jenkins-bot: Verified -- To view, visit https://gerrit.wikimedia.org/r/36944 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: merged Gerrit-Change-Id: I69f27b184419d3986a8fcfccdf9cb6d4e65e2bb6 Gerrit-PatchSet: 1 Gerrit-Project: analytics/webstatscollector Gerrit-Branch: master Gerrit-Owner: Stefan.petrea ste...@garage-coding.com Gerrit-Reviewer: Diederik dvanli...@wikimedia.org Gerrit-Reviewer: jenkins-bot ___ MediaWiki-commits mailing list MediaWiki-commits@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] Refactoring bash scripts for generating reports - change (analytics/wikistats[master])
Diederik has submitted this change and it was merged. Change subject: Refactoring bash scripts for generating reports .. Refactoring bash scripts for generating reports * Refactored SquidCountArchive.sh , added command-line switches and checks. Added option -s and -e for processing editors or sampled squid log files (uses the configuration we made in confi-editors) * Refactored SquidReportArchive.sh , added command-line switches -s and -e as above. Added 3 more switches for processing quarters, months, and country-scan * Removed the use of /usr/local/bin/geoiplogtag in favor of the same functionality which is provided now by udp-filters. So now the expected input logs will be pre-geocoded so we can just used the 15th field for that. * Removed dependencies so tests can run on jenkins * Added tests for domain name in external origin which looks like an ipv6 IP * Added Jenkins specific paths to test config file * More jenkins debugging * Made test config more generic so it can be used for all tests * added debug statements for jenkins, Identified the problem with ipv6 at line 1185 in SquidCountArchiveProcessLogRecord.pm * Fixed bug, if at line 1185 $referer is - then we can't find out external origin * Removed Perl scripts output so Jenkins gives green * CalcPercentages has a bug related to illegal division. Wrote test to catch it * Added multiplechecks for denominator and nominator so as to avoid illegal divisions in SquidReportArchive.pl All tests are passing now * Removed .gz files for tests from the repository, moved their contents and creating the .gz files from the test * Removed ips from data test file contents Change-Id: Ibf7fa4bbb34f3be9ce49c685ea98eb991ea0850a --- M squids/.gitignore M squids/bash/SquidCountArchive.sh M squids/bash/SquidReportArchive.sh M squids/perl/SquidCountArchive.pl M squids/perl/SquidCountArchiveConfig.pm M squids/perl/SquidCountArchiveProcessLogRecord.pm M squids/perl/SquidCountArchiveReadInput.pm M squids/perl/SquidReportArchive.pl M squids/perl/UA/iPad.pm D squids/t/02-ua-ipad-tests.t A squids/t/03-regression-wrong-domains-actually-ipv6.t A squids/t/04-regression-countries-count-arithmetic.t A squids/t/CommonConfig.pm A squids/t/Generate/Squid.pm A squids/testdata/regression-countries-count-arithmetic/SquidCountArchiveConfig.pm A squids/testdata/regression-countries-count-arithmetic/SquidReportArchiveConfig.pm A squids/testdata/regression-test-ipv6-wrong-external-domain/SquidCountArchiveConfig.pm A squids/testdata/regression-test-ipv6-wrong-external-domain/SquidReportArchiveConfig.pm 18 files changed, 772 insertions(+), 140 deletions(-) Approvals: Diederik: Verified; Looks good to me, approved jenkins-bot: Verified -- To view, visit https://gerrit.wikimedia.org/r/37859 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: merged Gerrit-Change-Id: Ibf7fa4bbb34f3be9ce49c685ea98eb991ea0850a Gerrit-PatchSet: 1 Gerrit-Project: analytics/wikistats Gerrit-Branch: master Gerrit-Owner: Stefan.petrea ste...@garage-coding.com Gerrit-Reviewer: Diederik dvanli...@wikimedia.org Gerrit-Reviewer: jenkins-bot ___ MediaWiki-commits mailing list MediaWiki-commits@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] Fixed configure.in - change (analytics/libanon[master])
Diederik has submitted this change and it was merged. Change subject: Fixed configure.in .. Fixed configure.in Deleted autom5te.cache from git Removed aclocal.m4 config.log config.status Change-Id: I29b01e049a918cc5bbd6b65ec357eac1da8751dc --- D Makefile M Makefile.in M README D aclocal.m4 D autom4te.cache/output.0 D autom4te.cache/output.1 D autom4te.cache/requests D autom4te.cache/traces.0 D autom4te.cache/traces.1 M config.h M config.h.in D config.log D config.status D configure M configure.in M libanon.lsm M libtool M src/Makefile M src/Makefile.in M test/Makefile M test/Makefile.in 21 files changed, 582 insertions(+), 62,509 deletions(-) Approvals: Diederik: Verified; Looks good to me, approved jenkins-bot: Verified -- To view, visit https://gerrit.wikimedia.org/r/31858 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: merged Gerrit-Change-Id: I29b01e049a918cc5bbd6b65ec357eac1da8751dc Gerrit-PatchSet: 3 Gerrit-Project: analytics/libanon Gerrit-Branch: master Gerrit-Owner: Stefan.petrea ste...@garage-coding.com Gerrit-Reviewer: Diederik dvanli...@wikimedia.org Gerrit-Reviewer: Stefan.petrea ste...@garage-coding.com Gerrit-Reviewer: jenkins-bot ___ MediaWiki-commits mailing list MediaWiki-commits@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] Added test for countries arithmetic - change (analytics/wikistats[master])
Diederik has submitted this change and it was merged. Change subject: Added test for countries arithmetic .. Added test for countries arithmetic * Added columns in @mimetypes_sorted array so that tables in SquidReportRequests look better * Added checks for number of columns in SquidReportRequests.htm using XPath on the generated tables Change-Id: I9d3749a77d8cef3f8403d472bbdb56b03924ef2f --- M squids/perl/SquidReportArchive.pl M squids/t/04-regression-countries-count-arithmetic.t 2 files changed, 74 insertions(+), 13 deletions(-) Approvals: Diederik: Verified; Looks good to me, approved jenkins-bot: Verified -- To view, visit https://gerrit.wikimedia.org/r/38006 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: merged Gerrit-Change-Id: I9d3749a77d8cef3f8403d472bbdb56b03924ef2f Gerrit-PatchSet: 1 Gerrit-Project: analytics/wikistats Gerrit-Branch: master Gerrit-Owner: Stefan.petrea ste...@garage-coding.com Gerrit-Reviewer: Diederik dvanli...@wikimedia.org Gerrit-Reviewer: jenkins-bot ___ MediaWiki-commits mailing list MediaWiki-commits@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] Fix bug in report 'Requests by destination'; totals in colum... - change (analytics/wikistats[master])
Diederik has uploaded a new change for review. Change subject: Fix bug in report 'Requests by destination'; totals in column 3 and 4 were the same. This fixes the bug that stated that two columns had been swapped (that report in itself was incorrect but lead to the root cause). .. Fix bug in report 'Requests by destination'; totals in column 3 and 4 were the same. This fixes the bug that stated that two columns had been swapped (that report in itself was incorrect but lead to the root cause). Change-Id: Iaf91b329b90b343d40de440d195db03af372b68e --- M squids/perl/SquidReportArchive.pl 1 file changed, 1 insertion(+), 1 deletion(-) git pull ssh://gerrit.wikimedia.org:29418/analytics/wikistats refs/changes/63/37663/1 -- To view, visit https://gerrit.wikimedia.org/r/37663 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: newchange Gerrit-Change-Id: Iaf91b329b90b343d40de440d195db03af372b68e Gerrit-PatchSet: 1 Gerrit-Project: analytics/wikistats Gerrit-Branch: master Gerrit-Owner: Diederik dvanli...@wikimedia.org ___ MediaWiki-commits mailing list MediaWiki-commits@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] Fix bug in report 'Requests by destination'; totals in colum... - change (analytics/wikistats[master])
Diederik has submitted this change and it was merged. Change subject: Fix bug in report 'Requests by destination'; totals in column 3 and 4 were the same. This fixes the bug that stated that two columns had been swapped (that report in itself was incorrect but lead to the root cause). .. Fix bug in report 'Requests by destination'; totals in column 3 and 4 were the same. This fixes the bug that stated that two columns had been swapped (that report in itself was incorrect but lead to the root cause). Change-Id: Iaf91b329b90b343d40de440d195db03af372b68e --- M squids/perl/SquidReportArchive.pl 1 file changed, 1 insertion(+), 1 deletion(-) Approvals: Diederik: Verified; Looks good to me, approved -- To view, visit https://gerrit.wikimedia.org/r/37663 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: merged Gerrit-Change-Id: Iaf91b329b90b343d40de440d195db03af372b68e Gerrit-PatchSet: 1 Gerrit-Project: analytics/wikistats Gerrit-Branch: master Gerrit-Owner: Diederik dvanli...@wikimedia.org Gerrit-Reviewer: Diederik dvanli...@wikimedia.org Gerrit-Reviewer: Stefan.petrea ste...@garage-coding.com ___ MediaWiki-commits mailing list MediaWiki-commits@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] Fixed blog.wikimedia.org title - change (analytics/udp-filters[master])
Diederik has submitted this change and it was merged. Change subject: Fixed blog.wikimedia.org title .. Fixed blog.wikimedia.org title * Fixed blog title for collector output * Added test and test input file Change-Id: I830619c1055bf22ce8860a8d9aa9ea0d4e07b03d --- A example.blog.wikimedia.log M run.sh M src/collector-output.c M src/collector-output.h M src/udp-filter.c 5 files changed, 52 insertions(+), 10 deletions(-) Approvals: Diederik: Verified; Looks good to me, approved jenkins-bot: Verified -- To view, visit https://gerrit.wikimedia.org/r/36398 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: merged Gerrit-Change-Id: I830619c1055bf22ce8860a8d9aa9ea0d4e07b03d Gerrit-PatchSet: 2 Gerrit-Project: analytics/udp-filters Gerrit-Branch: master Gerrit-Owner: Stefan.petrea ste...@garage-coding.com Gerrit-Reviewer: Diederik dvanli...@wikimedia.org Gerrit-Reviewer: Stefan.petrea ste...@garage-coding.com Gerrit-Reviewer: jenkins-bot ___ MediaWiki-commits mailing list MediaWiki-commits@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] Fixed segfault found during deployment - change (analytics/webstatscollector[master])
Diederik has submitted this change and it was merged. Change subject: Fixed segfault found during deployment .. Fixed segfault found during deployment * Fixed segfault caused by non-allocated pointer * Returned to input format expected from log2udp (with a seq field prepended to the udp-filter output) * Added more debugging messages * Extended buffers for collector * Added comments to mention correlation between constants for * buffer sizes in collector and in udp-filters Change-Id: I5ac2b74df5a999b7ae23e2e2608fe4cd9034f676 Change-Id: I65ad05519f8e59ef5c4f311aceedd4b5bda41506 --- M Makefile.am A debianize D debianize.bak D debianize.sh D git2deblogs D prepare-release M src/collector.c 7 files changed, 77 insertions(+), 142 deletions(-) Approvals: Diederik: Verified; Looks good to me, approved -- To view, visit https://gerrit.wikimedia.org/r/36234 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: merged Gerrit-Change-Id: I65ad05519f8e59ef5c4f311aceedd4b5bda41506 Gerrit-PatchSet: 1 Gerrit-Project: analytics/webstatscollector Gerrit-Branch: master Gerrit-Owner: Stefan.petrea ste...@garage-coding.com Gerrit-Reviewer: Diederik dvanli...@wikimedia.org ___ MediaWiki-commits mailing list MediaWiki-commits@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] Fixed bugs on deployment, merged - change (analytics/udp-filters[master])
Diederik has submitted this change and it was merged. Change subject: Fixed bugs on deployment, merged .. Fixed bugs on deployment, merged * Added wikivoyage acronym .wv * Added wikidata acronym .wd * Added test file for wikivoyage and wikidata * Included fixes from deployment * Added missing sample log files required inside tests * Added tests in run.sh for wikivoyage and wikidata Change-Id: Ib674869d303f3abf01bbcce2b1769e11992b2a66 Change-Id: Ic818fd79f3a308760a1b6aeb61bf112faf842c3f --- M Makefile.am M configure M debian/changelog M debianize A example.collector2.log A example.wikivoyage.wikidata.log R git2deblogs.pl M run.sh M src/collector-output.c M src/collector-output.h M src/udp-filter.c 11 files changed, 140 insertions(+), 38 deletions(-) Approvals: Diederik: Verified; Looks good to me, approved jenkins-bot: Verified -- To view, visit https://gerrit.wikimedia.org/r/35188 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: merged Gerrit-Change-Id: Ic818fd79f3a308760a1b6aeb61bf112faf842c3f Gerrit-PatchSet: 2 Gerrit-Project: analytics/udp-filters Gerrit-Branch: master Gerrit-Owner: Stefan.petrea ste...@garage-coding.com Gerrit-Reviewer: Diederik dvanli...@wikimedia.org Gerrit-Reviewer: jenkins-bot ___ MediaWiki-commits mailing list MediaWiki-commits@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] Added conf for wikistats only for editors - change (analytics/wikistats[master])
Diederik has submitted this change and it was merged. Change subject: Added conf for wikistats only for editors .. Added conf for wikistats only for editors * added parameter -p to SquidCountArchive.pl and SquidReportArchive.pl * removed logic for computing job_runs_on_production_server from EzLib.pm , this will be decided by the presence of the -p parameter * Added directory structure for editors reports. Added meta symlink for csv_editors/meta pointing to csv/meta * Removed csv_editors, reports_editors, logs_editors from .gitignore because they contain a lot of data (as suggested by Erik) Added -p switch to bash scripts * Squashed with 2 previous commits # The first commit's message is: first changes for wikivoyage # This is the 2nd commit message: fix retrieval of dump type and details Change-Id: Iae65c489f0713ba3264d05a2a2db42b2230abc3b --- M dumps/bash/count.sh A dumps/perl/QD_tools/readme.txt M dumps/perl/WikiCountsArguments.pm M dumps/perl/WikiCountsCollectEdits.pl M dumps/perl/WikiCountsInput.pm M dumps/perl/WikiCountsJobProgress.pl M dumps/perl/WikiReportsInput.pm M squids/.gitignore M squids/bash/SquidCountArchive.sh M squids/bash/SquidReportArchive.sh A squids/conf-editors/SquidCountArchiveConfig-editors.pm A squids/conf-editors/SquidReportArchiveConfig-editors.pm A squids/csv_editors/meta A squids/logs_editors/not_empty M squids/perl/EzLib.pm M squids/perl/SquidCountArchive.pl M squids/perl/SquidReportArchive.pl A squids/reports_editors/not_empty 18 files changed, 116 insertions(+), 35 deletions(-) Approvals: Diederik: Verified; Looks good to me, approved -- To view, visit https://gerrit.wikimedia.org/r/33858 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: merged Gerrit-Change-Id: Iae65c489f0713ba3264d05a2a2db42b2230abc3b Gerrit-PatchSet: 5 Gerrit-Project: analytics/wikistats Gerrit-Branch: master Gerrit-Owner: Stefan.petrea ste...@garage-coding.com Gerrit-Reviewer: Diederik dvanli...@wikimedia.org Gerrit-Reviewer: Erik Zachte ezac...@wikimedia.org Gerrit-Reviewer: Stefan.petrea ste...@garage-coding.com ___ MediaWiki-commits mailing list MediaWiki-commits@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] Fixed debianization. - change (analytics/udp-filters[master])
Diederik has submitted this change and it was merged. Change subject: Fixed debianization. .. Fixed debianization. * Added more checks to debianize.sh (in wmf-analytics/debianize) * Modified Source and Package fields in debian/control to confirm with package name from configure.ac * Modified configure.ac to package name udp-filters * Modified paths where all files are put inside debian/rules * Fixed manual page * Put placeholders instead of version numbers inside configure.ac (these are taken care of by debianize.sh) * Changed to symlinks of relative paths to submodule debianize/ * Reformatted debian/controls so it passes lintian checks * Modifieddebian/copyright to conform with lintian * Fixed --enable-debug in configure.ac and simplified code. Previously there were 2 places where the debug switches for gcc were set. Currently there is only one place. That is in configure.ac. Now in order to compile with debug symbols, one needs to ./configure --enable-debug * Added parameters for git2deblogs to override maintainer and * name(which are normally taken from the git repo). Also ran this * command to regenerate the debian/changelog ./git2deblogs --generate --force-maintainer-name=Diederik van Liere --force-maintainer-email=dvanli...@wikimedia.org Change-Id: I2e57a6876552a61974ebe225ba51bab5e29006bf --- D ChangeLog A ChangeLog M Makefile.am M compile.sh M configure M configure.ac D debian/changelog A debian/changelog M debian/control M debian/copyright M debian/rules D debian/udp-filter.files R debian/udp-filters.dirs A debian/udp-filters.files M debianize M docs/help2man.include M git2deblogs A prepare-release M src/udp-filter.c 19 files changed, 294 insertions(+), 312 deletions(-) Approvals: Diederik: Verified; Looks good to me, approved -- To view, visit https://gerrit.wikimedia.org/r/32137 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: merged Gerrit-Change-Id: I2e57a6876552a61974ebe225ba51bab5e29006bf Gerrit-PatchSet: 4 Gerrit-Project: analytics/udp-filters Gerrit-Branch: master Gerrit-Owner: Stefan.petrea ste...@garage-coding.com Gerrit-Reviewer: Diederik dvanli...@wikimedia.org Gerrit-Reviewer: Stefan.petrea ste...@garage-coding.com ___ MediaWiki-commits mailing list MediaWiki-commits@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] Added script to geocode all squid .gz files - change (analytics/wikistats[master])
Diederik has submitted this change and it was merged. Change subject: Added script to geocode all squid .gz files .. Added script to geocode all squid .gz files Change-Id: I794a2dd5228410fa4c837ec219048f50914d00aa --- A squids/bash/geocode_logs.sh 1 file changed, 17 insertions(+), 0 deletions(-) Approvals: Diederik: Verified; Looks good to me, approved -- To view, visit https://gerrit.wikimedia.org/r/31877 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: merged Gerrit-Change-Id: I794a2dd5228410fa4c837ec219048f50914d00aa Gerrit-PatchSet: 1 Gerrit-Project: analytics/wikistats Gerrit-Branch: master Gerrit-Owner: Stefan.petrea ste...@garage-coding.com Gerrit-Reviewer: Diederik dvanli...@wikimedia.org ___ MediaWiki-commits mailing list MediaWiki-commits@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] Added git sumodule debianize - change (analytics/udp-filters[master])
Diederik has submitted this change and it was merged. Change subject: Added git sumodule debianize .. Added git sumodule debianize Added git submodule debianize Made symlinks to scripts Change-Id: Ic330991716c29b3543296fb128f3bd341d613c6b --- A .gitmodules A debianize A debianize.sh A git2deblogs 4 files changed, 5 insertions(+), 0 deletions(-) Approvals: Diederik: Verified; Looks good to me, approved -- To view, visit https://gerrit.wikimedia.org/r/31884 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: merged Gerrit-Change-Id: Ic330991716c29b3543296fb128f3bd341d613c6b Gerrit-PatchSet: 1 Gerrit-Project: analytics/udp-filters Gerrit-Branch: master Gerrit-Owner: Stefan.petrea ste...@garage-coding.com Gerrit-Reviewer: Diederik dvanli...@wikimedia.org ___ MediaWiki-commits mailing list MediaWiki-commits@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] Temporary branch with LF fixed - change (analytics/wikistats[master])
Diederik has submitted this change and it was merged. Change subject: Temporary branch with LF fixed .. Temporary branch with LF fixed Also added .gitattributes Added .gitreview Change-Id: Ied07dad1cad79161eca34191764ee4f3ea9b2f6f --- A .gitreview A squids/.gitattributes M squids/.gitignore M squids/perl/EzLib.pm M squids/perl/SquidCountArchive.pl M squids/perl/SquidCountArchiveConfig.pm M squids/perl/SquidCountArchiveProcessLogRecord.pm M squids/perl/SquidCountArchiveReadInput.pm M squids/perl/SquidCountArchiveWriteOutput.pm M squids/perl/SquidCountryScan.pl M squids/perl/SquidLoadScan.pl M squids/perl/SquidReportArchive.pl M squids/perl/SquidReportArchiveConfig.pm M squids/perl/SquidTraceUniqueImages.pl 14 files changed, 14,376 insertions(+), 14,366 deletions(-) Approvals: Diederik: Verified; Looks good to me, approved -- To view, visit https://gerrit.wikimedia.org/r/29870 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: merged Gerrit-Change-Id: Ied07dad1cad79161eca34191764ee4f3ea9b2f6f Gerrit-PatchSet: 1 Gerrit-Project: analytics/wikistats Gerrit-Branch: master Gerrit-Owner: Stefan.petrea ste...@garage-coding.com Gerrit-Reviewer: Diederik dvanli...@wikimedia.org ___ MediaWiki-commits mailing list MediaWiki-commits@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] Making geoip field last - change (analytics/udp-filters[master])
Diederik has submitted this change and it was merged. Change subject: Making geoip field last .. Making geoip field last Change-Id: Iaad7699049bccfb27b5421e97a64f63ccd9f6d09 --- M src/udp-filter.c 1 file changed, 30 insertions(+), 5 deletions(-) Approvals: Diederik: Verified; Looks good to me, approved -- To view, visit https://gerrit.wikimedia.org/r/29469 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: merged Gerrit-Change-Id: Iaad7699049bccfb27b5421e97a64f63ccd9f6d09 Gerrit-PatchSet: 1 Gerrit-Project: analytics/udp-filters Gerrit-Branch: master Gerrit-Owner: Stefan.petrea ste...@garage-coding.com Gerrit-Reviewer: Diederik dvanli...@wikimedia.org ___ MediaWiki-commits mailing list MediaWiki-commits@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
