Milimetric has submitted this change and it was merged. Change subject: Fabric deployment setup for wikimetrics ......................................................................
Fabric deployment setup for wikimetrics Bug: T122228 Change-Id: I71785fda814e7fe3dcfe623336a135ccb321db3f --- M .gitignore A .gitmodules A README.md A config/db_config.yaml A config/queue_config.yaml A config/web_config.yaml A fabfile.py A secrets/private A secrets/public/test/db_secrets.yaml A secrets/public/test/web_secrets.yaml 10 files changed, 428 insertions(+), 0 deletions(-) Approvals: Milimetric: Verified; Looks good to me, approved diff --git a/.gitignore b/.gitignore index 0d20b64..7b064cb 100644 --- a/.gitignore +++ b/.gitignore @@ -1 +1,2 @@ *.pyc +secrets/private diff --git a/.gitmodules b/.gitmodules new file mode 100644 index 0000000..269ff4d --- /dev/null +++ b/.gitmodules @@ -0,0 +1,3 @@ +[submodule "secrets/private"] + path = secrets/private + url = ssh://gerrit.wikimedia.org:29418/secrets/wikimetrics diff --git a/README.md b/README.md new file mode 100644 index 0000000..d010cad --- /dev/null +++ b/README.md @@ -0,0 +1,75 @@ +Wikimetrics Deploy +=================== + +wikimetrics-deploy contains the fabric scripts and configuration info to help deploy [wikimetrics](https://github.com/wikimedia/analytics-wikimetrics). + +## Configuration setup + +The config folder contains templatized yaml configs for the wikimetrics web server, queue and db. + +The secrets folder is set up with a private folder which is a git submodule that contains the production and staging secrets (db passwords, oauth creds) and clones only if you have access to it; and a public folder that contains a test folder as an example. + +When initializing the server or deploying, fabric will substitute the config templates with the relevant secret keys and upload the final files to the destination config directory. + +The STAGES global variable in the fabfile defines the staging and production enviroments, and the corresponding hosts and other config information. + +The DB is setup locally for staging, and on wmf labsdb for production (so that we don't have to manage backups) + +## How to deploy + +``` +# Clone the wikimetrics-deploy repo from gerrit + +git clone ssh://[email protected]:29418/analytics/wikimetrics-deploy + +# Look at the available fabric tasks +cd wikimetrics-deploy +wikimetrics-deploy [master] ⚡ fab -list +Available commands: + + deploy Deploys updated code to the web server + initialize_server Setup an initial deployment on a fresh host. + production + restart_wikimetrics Restarts the wikimetrics queue, scheduler and web server + staging + +# Initialize staging server - Do this only once +# (Probably never have to do this when deploying to an existing host) +fab staging initialize_server + +# Or for production server +fab production initialize_server + +# To deploy to staging +fab staging deploy + +# Or to production +fab production deploy + +# Example of initialize_server with only user specified output +# (You can see all of the debug output if you do not use the --hide and --show options) + +wikimetrics-deploy [master] ⚡ fab staging initialize_server --hide everything --show user +Setting up the staging server +Updating wikimetrics-deploy repo +Updating wikimetrics source repo +Uploading config files to remote host(s) +Upgrading requirements +Setting up database and access +Bringing database uptodate + +Done. +Disconnecting from [email protected]... done. + +# To restart wikimetrics +wikimetrics-deploy [master] ⚡ fab staging restart_wikimetrics +[wikimetrics-staging.wikimetrics.eqiad.wmflabs] Executing task 'restart_wikimetrics' +Restarting queue, scheduler and web server +[wikimetrics-staging.wikimetrics.eqiad.wmflabs] sudo: service uwsgi-wikimetrics-web restart +[wikimetrics-staging.wikimetrics.eqiad.wmflabs] sudo: service wikimetrics-queue restart +[wikimetrics-staging.wikimetrics.eqiad.wmflabs] sudo: service wikimetrics-scheduler restart + +Done. +Disconnecting from [email protected]... done. + +``` \ No newline at end of file diff --git a/config/db_config.yaml b/config/db_config.yaml new file mode 100644 index 0000000..3f01017 --- /dev/null +++ b/config/db_config.yaml @@ -0,0 +1,18 @@ +DB_USER_WIKIMETRICS : $DB_USER_WIKIMETRICS +DB_PASSWORD_WIKIMETRICS : $DB_PASSWORD_WIKIMETRICS +DB_NAME_WIKIMETRICS : $DB_NAME_WIKIMETRICS +DB_HOST_WIKIMETRICS : $DB_HOST_WIKIMETRICS +DB_NAMES_TESTING : ['wikimetrics_testing', 'wiki_testing', 'wiki2_testing', 'centralauth_testing'] +DB_USER_LABSDB : $DB_USER_LABSDB +DB_PASSWORD_LABSDB : $DB_PASSWORD_LABSDB +SQL_ECHO : False +WIKIMETRICS_ENGINE_URL : 'mysql://$DB_USER_WIKIMETRICS:$DB_PASSWORD_WIKIMETRICS@$DB_HOST_WIKIMETRICS/$DB_NAME_WIKIMETRICS' +WIKIMETRICS_POOL_SIZE : 100 +MEDIAWIKI_ENGINE_URL_TEMPLATE : 'mysql://$DB_USER_LABSDB:[email protected]/{0}_p' +MEDIAWIKI_POOL_SIZE : 200 +CENTRALAUTH_ENGINE_URL : 'mysql://$DB_USER_LABSDB:[email protected]/centralauth_p' +REPLICATION_LAG_MW_PROJECTS : [ 'enwiki', 'eowiki', 'elwiki', 'commonswiki', 'dewiki', 'frwiki', 'eswiki' ] +REPLICATION_LAG_THRESHOLD : 3 # (measured in hours) +DEBUG : False +REVISION_TABLENAME : revision_userindex +ARCHIVE_TABLENAME : archive_userindex diff --git a/config/queue_config.yaml b/config/queue_config.yaml new file mode 100644 index 0000000..8df4e84 --- /dev/null +++ b/config/queue_config.yaml @@ -0,0 +1,18 @@ +BROKER_URL : redis://localhost:6379/0 +CELERY_RESULT_BACKEND : redis://localhost:6379/0 +CELERY_TASK_RESULT_EXPIRES : 2592000 +CELERY_DISABLE_RATE_LIMITS : True +CELERY_STORE_ERRORS_EVEN_IF_IGNORED : True +CELERYD_CONCURRENCY : 24 +CELERYD_TASK_TIME_LIMIT : 3630 +CELERYD_TASK_SOFT_TIME_LIMIT : 3600 +DEBUG : False +LOG_LEVEL : INFO +MAX_INSTANCES_PER_RECURRENT_REPORT : 365 +CELERY_BEAT_DATAFILE : /var/run/wikimetrics/celerybeat_scheduled_tasks +CELERY_BEAT_PIDFILE : /var/run/wikimetrics/celerybeat.pid +CELERYBEAT_SCHEDULE : + 'update-daily-recurring-reports': + 'task' : 'wikimetrics.schedules.daily.recurring_reports' + # The schedule can be set to 'daily' for a crontab-like daily recurrence + 'schedule' : daily diff --git a/config/web_config.yaml b/config/web_config.yaml new file mode 100644 index 0000000..00897be --- /dev/null +++ b/config/web_config.yaml @@ -0,0 +1,30 @@ +SECRET_KEY : $SECRET_KEY +DEBUG : False + +GOOGLE_CLIENT_SECRET : 'i1T3NJl3NKNu-BnEWQB4QAF6' +GOOGLE_BASE_URL : 'https://www.google.com/accounts/' +GOOGLE_AUTH_URI : 'https://accounts.google.com/o/oauth2/auth' +GOOGLE_TOKEN_URI : 'https://accounts.google.com/o/oauth2/token' +GOOGLE_CLIENT_EMAIL : $GOOGLE_CLIENT_EMAIL +GOOGLE_CLIENT_ID : $GOOGLE_CLIENT_ID +GOOGLE_AUTH_PROVIDER_X509_CERT_URL : 'https://www.googleapis.com/oauth2/v1/certs' +GOOGLE_AUTH_SCOPE : 'https://www.googleapis.com/auth/userinfo.email' +GOOGLE_USERINFO_URI : 'https://www.googleapis.com/oauth2/v1/userinfo' +GOOGLE_REDIRECT_URI : 'https://metrics.wmflabs.org/auth/google' +GOOGLE_JAVASCRIPT_ORIGIN : 'https://metrics.wmflabs.org' + +META_MW_BASE_URL : 'https://meta.wikimedia.org/' +META_MW_BASE_INDEX : 'w/index.php' +META_MW_TOKEN_URI : 'w/index.php?title=Special:OAuth/token' +META_MW_AUTH_URI : 'wiki/Special:OAuth/authorize' +META_MW_USERINFO_URI : 'w/api.php?action=query&meta=userinfo&format=json' +META_MW_IDENTIFY_URI : 'w/index.php?title=Special:OAuth/identify&format=json' +META_MW_REDIRECT_URI : 'https://metrics.wmflabs.org/auth/meta_mw' +META_MW_CONSUMER_KEY : $META_MW_CONSUMER_KEY +META_MW_CLIENT_SECRET : $META_MW_CLIENT_SECRET + +SERVER_HOST : '0.0.0.0' +SERVER_PORT : 80 + +PUBLIC_PATH : /srv/var/wikimetrics/public +test: test diff --git a/fabfile.py b/fabfile.py new file mode 100644 index 0000000..71ba80e --- /dev/null +++ b/fabfile.py @@ -0,0 +1,271 @@ +from fabric.api import cd, env, put, require, shell_env, sudo, task, run +from functools import wraps +from string import Template + +import os +import StringIO +import yaml + +env.use_ssh_config = True +env.shell = '/bin/bash -c' + +STAGES = { + 'staging': { + 'hosts': ['wikimetrics-staging.wikimetrics.eqiad.wmflabs'], + 'secrets_dir': 'secrets/private/staging', + 'source_branch': 'master', + 'deploy_branch': 'master', + 'debug': True, + }, + 'production': { + 'hosts': ['wikimetrics-01.wikimetrics.eqiad.wmflabs'], + 'secrets_dir': 'secrets/private/production', + 'source_branch': 'master', + 'deploy_branch': 'master', + 'debug': False, + }, +} + +SOURCE_DIR = '/srv/wikimetrics/src' +CONFIG_DIR = '/srv/wikimetrics/config' +VENV_DIR = '/srv/wikimetrics/venv' +LOCAL_CONFIG_DIR = 'config' + +DB_CONFIG_FILE = 'db_config.yaml' +QUEUE_CONFIG_FILE = 'queue_config.yaml' +WEB_CONFIG_FILE = 'web_config.yaml' + +DB_SECRETS_FILE = 'db_secrets.yaml' +WEB_SECRETS_FILE = 'web_secrets.yaml' + + +def sr(*cmd): + """ + Sudo Run - Wraps a given command around sudo and runs it as the + wikimetrics user + """ + with shell_env(HOME='/srv/wikimetrics'): + return sudo(' '.join(cmd), user='wikimetrics') + + +def set_stage(stage='staging'): + """ + Sets the stage and populate the environment with the necessary + config. Doing this allows accessing from anywhere stage related + details by simply doing something like env.source_dir etc + + It also uses the values defined in the secrets directories + to substitute the templatized config files for the db and web configs, + and loads the final db, queue and web configs as strings into the + environment + """ + env.stage = stage + for option, value in STAGES[env.stage].items(): + setattr(env, option, value) + + secrets_dir = STAGES[env.stage]['secrets_dir'] + + # Load DB config into environment + with open(secrets_dir + '/' + DB_SECRETS_FILE) as secrets, \ + open(LOCAL_CONFIG_DIR + '/' + DB_CONFIG_FILE) as config: + db_config_template = Template(config.read()) + db_secrets = yaml.load(secrets) + setattr(env, 'db_config', db_config_template.substitute(db_secrets)) + + # Load Web config into environment + with open(secrets_dir + '/' + WEB_SECRETS_FILE) as secrets, \ + open(LOCAL_CONFIG_DIR + '/' + WEB_CONFIG_FILE) as config: + web_config_template = Template(config.read()) + web_secrets = yaml.load(secrets) + setattr(env, 'web_config', web_config_template.substitute(web_secrets)) + + # Load Queue config into environment + with open(LOCAL_CONFIG_DIR + '/' + QUEUE_CONFIG_FILE) as config: + setattr(env, 'queue_config', config.read()) + + +def ensure_stage(fn): + """ + Decorator to ensure the stage is set + """ + @wraps(fn) + def wrapper(*args, **kwargs): + # The require operation will abort if the key stage + # is not set in the environment + require('stage', provided_by=(staging, production,)) + return fn(*args, **kwargs) + return wrapper + + +@task +def production(): + set_stage('production') + + +@task +def staging(): + set_stage('staging') + + +@task +@ensure_stage +def initialize_server(): + """ + Setup an initial deployment on a fresh host. + """ + print 'Setting up the ' + env.stage + ' server' + # Sets up a virtualenv directory + sr('mkdir', '-p', VENV_DIR) + sr('virtualenv', '--python', 'python2', VENV_DIR) + + # Updates current version of wikimetrics-deploy + update_deploy_repo() + + # Updates current version of wikimetrics source + update_source_repo() + + # Uploads the db and oauth creds to the server + upload_config() + + # Updates the virtualenv with new wikimetrics code + upgrade_wikimetrics() + + # Initialize DB + setup_db() + + # Update DB + update_db() + + +@task +@ensure_stage +def deploy(): + """ + Deploys updated code to the web server + """ + print 'Deploying to ' + env.stage + + # Updates current version of wikimetrics-deploy + update_deploy_repo() + + # Updates current version of wikimetrics source + update_source_repo() + + # Updates the virtualenv with new wikimetrics code + upgrade_wikimetrics() + + # Update DB + update_db() + + # Restart wikimetrics queue, scheduler and web services + restart_wikimetrics() + + +@ensure_stage +def update_source_repo(): + """ + Update the wikimetrics source repo + """ + print 'Updating wikimetrics source repo' + with cd(SOURCE_DIR): + sr('git', 'fetch', 'origin', env.source_branch) + sr('git', 'reset', '--hard', 'FETCH_HEAD') + + +@ensure_stage +def update_deploy_repo(): + """ + Updates the deployment repo + """ + print 'Updating wikimetrics-deploy repo' + with cd(CONFIG_DIR): + sr('git', 'fetch', 'origin', env.deploy_branch) + sr('git', 'reset', '--hard', 'FETCH_HEAD') + + +def upload_file(config, dest): + """ + Converts config file contents from string to file buffer using StringIO + and uploads to remote server + """ + buffer = StringIO.StringIO() + buffer.write(config) + put(buffer, dest, use_sudo=True) + buffer.close() + + +@ensure_stage +def upload_config(): + """ + Upload the queue, web and db configs to the remote host + """ + print 'Uploading config files to remote host(s)' + upload_file(env.web_config, CONFIG_DIR + '/' + WEB_CONFIG_FILE) + upload_file(env.db_config, CONFIG_DIR + '/' + DB_CONFIG_FILE) + upload_file(env.queue_config, CONFIG_DIR + '/' + QUEUE_CONFIG_FILE) + + +@ensure_stage +def upgrade_wikimetrics(): + """ + Installs upgraded versions of requirements (if applicable) + """ + print 'Upgrading requirements' + with cd(VENV_DIR): + sr(VENV_DIR + '/bin/pip', 'install', '--upgrade', '-r', + os.path.join(CONFIG_DIR, 'requirements.txt')) + + +def create_db_and_user(db_name, db_config): + """ + Creates the database db_name and grants access to the + user, as defined by the db_config, on the given remote mysql server + """ + db_user = db_config['DB_USER_WIKIMETRICS'] + db_password = db_config['DB_PASSWORD_WIKIMETRICS'] + db_host = db_config['DB_HOST_WIKIMETRICS'] + run("mysql -u root -h {0} -e ".format(db_host) + + "'CREATE DATABASE IF NOT EXISTS {1}'".format(db_host, db_name)) + run("mysql -u root -h {0} -e ".format(db_host) + + "'GRANT ALL ON `{0}`.* TO \"{1}\"@\"{2}\" identified by \"{3}\";'" + .format(db_name, db_user, db_host, db_password)) + + +@ensure_stage +def setup_db(): + """ + Creates the database and grants access to the wikimetrics user + """ + print 'Setting up database and access' + # Convert db config to yaml + db_config = yaml.load(env.db_config) + + # Setup wikimetrics DB + create_db_and_user(db_config['DB_NAME_WIKIMETRICS'], db_config) + + if env.debug: + # Setup Testing DBs + for db_name in db_config['DB_NAMES_TESTING']: + create_db_and_user(db_name, db_config) + + +@ensure_stage +def update_db(): + """ + Creates and updates all tables by running migrations + """ + print 'Bringing database uptodate by running migrations' + with cd(SOURCE_DIR): + run(VENV_DIR + '/bin/alembic upgrade head') + + +@task +@ensure_stage +def restart_wikimetrics(): + """ + Restarts the wikimetrics queue, scheduler and web server + """ + print 'Restarting queue, scheduler and web server' + sudo('service uwsgi-wikimetrics-web restart') + sudo('service wikimetrics-queue restart') + sudo('service wikimetrics-scheduler restart') diff --git a/secrets/private b/secrets/private new file mode 160000 index 0000000..f38891c --- /dev/null +++ b/secrets/private +Subproject commit f38891c59c73364fd5ce46278eade0c5894cd2aa diff --git a/secrets/public/test/db_secrets.yaml b/secrets/public/test/db_secrets.yaml new file mode 100644 index 0000000..a379585 --- /dev/null +++ b/secrets/public/test/db_secrets.yaml @@ -0,0 +1,6 @@ +DB_USER_WIKIMETRICS : wikimetrics +DB_PASSWORD_WIKIMETRICS : wikimetrics +DB_USER_LABSDB : wikimetrics +DB_PASSWORD_LABSDB : wikimetrics +DB_HOST_WIKIMETRICS : localhost +DB_NAME_WIKIMETRICS : wikimetrics diff --git a/secrets/public/test/web_secrets.yaml b/secrets/public/test/web_secrets.yaml new file mode 100644 index 0000000..98b97a8 --- /dev/null +++ b/secrets/public/test/web_secrets.yaml @@ -0,0 +1,6 @@ +SECRET_KEY : 'sw3lo(*A98ijwl3i3n&&JEK MSL IESKU* WSkl#iIREJSslsijsle ssie' +GOOGLE_CLIENT_SECRET : 'zKv0Qg7Zr6L3Q3CaWnIuVX4B' +GOOGLE_CLIENT_EMAIL : '[email protected]' +GOOGLE_CLIENT_ID : '133082872359.apps.googleusercontent.com' +META_MW_CONSUMER_KEY : 'bad4e459823278bdffb5ecf0a206112d' +META_MW_CLIENT_SECRET : 'e312699be56f1d157657727a87ce3776e172501a' -- To view, visit https://gerrit.wikimedia.org/r/261579 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: merged Gerrit-Change-Id: I71785fda814e7fe3dcfe623336a135ccb321db3f Gerrit-PatchSet: 8 Gerrit-Project: analytics/wikimetrics-deploy Gerrit-Branch: master Gerrit-Owner: Madhuvishy <[email protected]> Gerrit-Reviewer: Elukey <[email protected]> Gerrit-Reviewer: Madhuvishy <[email protected]> Gerrit-Reviewer: Milimetric <[email protected]> Gerrit-Reviewer: Nuria <[email protected]> Gerrit-Reviewer: Yuvipanda <[email protected]> _______________________________________________ MediaWiki-commits mailing list [email protected] https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
