jenkins-bot has submitted this change and it was merged. ( https://gerrit.wikimedia.org/r/396407 )
Change subject: Updating Portal documentation ...................................................................... Updating Portal documentation Updating portal repo readme and contributing files, as well as providing extensive technical documentation in a new 'docs' directory. Bug: T182119 Change-Id: If4f8f867d16d5bcca04d04fd9423edba2a5295cc --- M CONTRIBUTING.md M README.md A docs/README.md A docs/architecture/css.md A docs/architecture/data.md A docs/architecture/html.md A docs/architecture/images.md A docs/architecture/javascript.md A docs/architecture/l10n.md A docs/development/deploy.md A docs/development/getting_started.md A docs/development/gulp.md A docs/development/prod.md A docs/development/sister_portals.md A docs/portals-dataflow.jpg 15 files changed, 577 insertions(+), 86 deletions(-) Approvals: Debt: Looks good to me, approved jenkins-bot: Verified Phuedx: Looks good to me, but someone else must approve diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 59eb464..23f0b2e 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -1,55 +1,43 @@ -# Wikipedia.org Portal Development +# Contributing + +> READ: [Technical Documentation](./docs/README.md) These steps outline the wikipedia.org portal development process. Steps for updating all other Wikimedia Foundation portals are described in the [Updating Other Portal Pages](#updating-other-portal-pages) section. The wikipedia.org portal page utilizes several pre/post processors to generate an optimized HTML page. -### Installation +## Quick Start + +> NOTE: You'll need [Gerrit access](https://www.mediawiki.org/wiki/Developer_access) to contribute code to this repository. Prerequisites: -- node -- npm -`npm install` and grab a coffee (that'll take a few minutes). The portal page has numerous dev dependencies, all of which are listed in the `package.json` file. Some of the main dev dependencies include the following: +- Node +- NPM - - [Gulp](http://gulpjs.com/) - task runner - - [PostCSS](http://postcss.org/) & [cssnext](http://cssnext.io/) - CSS pre-processors - - [Handlebars](http://handlebarsjs.com/) - html templates - - [Imagemin](https://www.npmjs.com/package/gulp-imagemin) & [sprity](https://www.npmjs.com/package/sprity) - image optimization and CSS background sprite generator - - [JSHint](http://jshint.com/) & [JSCS](http://jscs.info/) - JS code style validators - - [PhantomJS](http://phantomjs.org/) & [CasperJS](http://casperjs.org/) - for running tests +From the root of the repo, run: -All these tools and more, are executed through gulp tasks. +``` +npm install # install dependancies +gulp update-stats # update portal stats +gulp watch --portal=wikipedia.org # watch dev directory for changes +python -m SimpleHTTPServer 8080 # start a dev server +``` -Running `gulp help` will output a list of available gulp tasks. The output might look something like this. +Visit `http://localhost:8080/dev/wikipedia.org` to see the changes in the `dev` directory. +Running `gulp help` will output a list of available gulp tasks. -```` -+-------------------------------------------------------------------------------------------------+ -| ===== USAGE ===== | -+-------------------------------------------------------------------------------------------------+ -| GLOBAL TASKS : | -| | -| gulp lint - run jslint on all JS files | -| gulp update-stats - update file containing projects stats | -+-------------------------------------------------------------------------------------------------+ -| PORTAL-SPECIFIC TASKS : | -| | -| gulp lint --portal wikipedia.org - run jslint on JS files on portal JS files | -| gulp inline-assets --portal wikipedia.org - build inline CSS and JS assets | -| gulp optimize-images --portal wikipedia.org - run imagemin on image directory | -| gulp watch --portal wikipedia.org - watches dev directory and generates an index.html | -| file in it without inlined/minified assets | -| gulp --portal wikipedia.org - run all of the above on the specified portal page | -| | -| gulp fetch-meta --portal wikipedia.org - overwrite the portal page with source from Meta | -+-------------------------------------------------------------------------------------------------+ -```` +*See the [Getting Started](development/getting_started.md) section of the technical documentation for more details.* +## Coding Style -### Directory Structure +1. [Mediawiki Javascript coding conventions](https://www.mediawiki.org/wiki/Manual:Coding_conventions/JavaScript) - linted by [eslint-config-wikimedia](https://www.npmjs.com/package/eslint-config-wikimedia) +2. [Mediawiki CSS coding conventions](https://www.mediawiki.org/wiki/Manual:Coding_conventions/CSS) - linted by [stylelint-config-wikimedia](https://www.npmjs.com/package/stylelint-config-wikimedia) -The directory structure is divided into a development ('dev') directory and a production ('prod') directory. `prod` should not be edited directly, as its content will be overridden when running the build command `gulp --portal wikipedia.org`. +## Directory Structure + +The directory structure is divided into a development ('dev') directory and a production ('prod') directory. `prod` should not be edited directly, as its content will be overridden when running the build command `gulp --portal=wikipedia.org`. ```` |— package.json @@ -74,38 +62,20 @@ ```` -#### Starting Development +## Build -Afer running `npm install`, you will probably run `gulp watch --portal wikipedia.org` during most of your time coding. This task watches for changes in `dev/wikipedia.org/` and generate an `index.html` file at `dev/wikipedia.org/index.html`. This file contains un-minified JS & CSS assets, making it easy to debug. CSS development is done using the PostCSS preprocessor. Consult the README file at `dev/wikipedia.org/assets/postcss/README.txt` for more details on PostCSS. Javascript development does not require any pre-processing, but must conform to [WMF coding conventions](https://www.mediawiki.org/wiki/Manual:Coding_conventions/JavaScript) and pass linting with JSHint. +`gulp --portal=wikipedia.org` will generate the production version of the page and place it in `/prod/wikipedia.org/`. The production version contains JS and image assets that have been combined, uglified, minified, compressed and suffixed with a cache-busting file-name. **The contents of the `prod` directory should not be edited directly**. + +*See the [Production Builds](development/prod.md) section of the technical documentation for more details.* -### Article-count & Wiki Ranking Files +## Updating Translations +Translations are edited through translatewiki.net. To edit a translation, [visit this page](https://translatewiki.net/w/i.php?title=Special:Translate&filter=&group=wikimedia-portals&optional=1&task=custom), select the language you wish to translate to, and proceed to edit a translation. The translations will be merged into the repository automatically by a bot (l10n-bot). -The command `gulp update-stats` updates the article-counts and rankings for the wikipedia.org portal page. To see these changes actually reflected on the page, run `gulp watch --portal wikipedia.org`. This will generate a new development version of the page in the `/dev` directory. Then run `gulp --portal wikipedia.org` to build a new production version of the page with the new article counts and rankings. - -The following files are responsible for populating the data on the portal page. - -```` -|— site-defs.json static file copied from https://meta.wikimedia.org/wiki/Module:Project_portal/wikis -|— site-stats.js generates the site-stats.json file -|— site-stats.json contains per-wiki pageviews and article counts -|— stats.js combines site-stats.json & site-defs.json into an exported Stats variable -|— dev/ - |— wikipedia.org/ - |— other-projects.json text for sister projects. - |— other-languages.json text for 'other languages' section on portal. - |— controller.js Handlebars helper that organizes wikis by ranking. -```` - -### Build - -`gulp --portal wikipedia.org` will generate the production version of the page and place it in `/prod/wikipedia.org/`. The production version contains JS and image assets that have been combined, uglified, minified, compressed and suffixed with a cache-busting file-name. The contents of `prod` should not be edited directly. - -### Testing - -`npm test` will run a basic Javascript linting test on all the JS code in the repo. - -Integration tests have been created to verify that event-logging used by the WMF for analytics and A/B testing purposes is functional. Tests are located in the `/tests` folder and are run through PhantomJS and Casper.js. More documentation on these tests, including how to run them and what they test, are found in the comments in each file. +*See [the l10n section](./docs/architecture/l10n.md) of the technical documentation for more details.* ## Updating Other Portal Pages -All other WMF project portals are still updated through their respective wiki pages on meta.wikimedia.org, (e.g [wiktionary portal](https://meta.wikimedia.org/wiki/Www.wiktionary.org_template) ). When these pages are updated through the wiki, they must then be copied into this repository in order to be deployed. They are copied using the gulp command `gulp fetch-meta --portal wiktionary.org` to copy a single portal, or `gulp fetch-meta --portal all` to copy the newest versions of all the portal pages (except for wikipedia.org) into the repo. +All other WMF project portals are still updated through their respective wiki templates on meta.wikimedia.org, (e.g [wiktionary portal](https://meta.wikimedia.org/wiki/Www.wiktionary.org_template) ). When these pages are updated through the wiki, they must then be copied into this repository in order to be deployed. They are copied using the gulp command `gulp fetch-meta --portal=wiktionary.org` to copy a single portal, or `gulp fetch-meta --portal=all` to copy the newest versions of all the portal pages (except for wikipedia.org) into the repo. + +*See the [Sister Project Portals](development/sister_portals.md) section of the technical documentation for more details.* + diff --git a/README.md b/README.md index dca6e46..b55099f 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,6 @@ # WMF Portal Pages -This repo houses the portal pages for the following Wikimedia Foundation projects: +This repo houses the portal pages for the following Wikimedia Foundation projects: - www.wikibooks.org - www.wikimedia.org @@ -11,27 +11,14 @@ - www.wikivoyage.org - www.wiktionary.org -### Project Overview +## Overview -All portal pages except wikipedia.org are generated by using [these templates](https://meta.wikimedia.org/wiki/Project_portals) on meta-wiki. These pages are then pulled into this repo for deployment. +www.wikipedia.org is built using a modern (circa 2015) front-end development stack which includes HTML templates, CSS preprocessors and build tooling to generate a static, optimized HTML page. Sister project portals (e.g. www.wiktionary.org) are generated using [these Lua templates](https://meta.wikimedia.org/wiki/Project_portals) on meta-wiki. After these templates are updated, the pages need to be pulled into this repository for deployment. -Wikipedia.org on the other hand, is not updated through meta-wiki. To update or edit the wikipedia.org portal, please read the `CONTRIBUTING.md` file located at the root of this repo. +## Getting Started -### Wikipedia.org +- [CONTRIBUTING.md](./CONTRIBUTING.md) contains a brief overview and quickstart guide for contributing code to this repository. +- [Technical Documentation](./docs/README.md) contains a detailed description of all the various aspects of development for this project, including getting started through to deployment. -This portal requires a build process that depends on many tools, including the following: - - - Gulp - - PostCSS & cssnext - - Handlebars templates - - Imagemin & sprity - - JSHint & JSCS - - PhantomJS & CasperJS (for testing) - -`npm install` to install all dependencies. - -`gulp help` to view Gulp documentation. - -`npm test` to run JSHint. - -Please read `CONTRIBUTING.md` for more info on how the wikipedia.org portal is built. +## License +Copyright (c) 2016 Wikimedia Foundation \ No newline at end of file diff --git a/docs/README.md b/docs/README.md new file mode 100644 index 0000000..6d09ff5 --- /dev/null +++ b/docs/README.md @@ -0,0 +1,31 @@ +# Wikipedia.org technical Documentation + +**TOC** + +- **Overview** +- Architecture of www.wikipedia.org + - [Data sources](architecture/data.md) + - [l10n](architecture/l10n.md) + - [HTML](architecture/html.md) + - [CSS](architecture/css.md) + - [Images](architecture/images.md) + - [JS](architecture/javascript.md) +- Development Process + - [Getting Started](development/getting_started.md) + - [Gulp Tasks](development/gulp.md) + - [Production Builds](development/prod.md) + - [Deployment](development/deploy.md) + - [Sister Project Portals](development/sister_portals.md) + +--- +## Overview + +> NOTE: This documentation is best read online through the repository clone on [diffusion](https://phabricator.wikimedia.org/diffusion/WPOR/browse/master/docs) or [github](https://github.com/wikimedia/portals/tree/master/docs) + +The wikipedia.org page is a static HTML/CSS/JS page. However, it uses modern front-end development stack that includes templates, preprocessors and build-tools to achieve an optimized and performant output. We use [Handlebars](http://handlebarsjs.com/) for HTML templates, [PostCSS](http://postcss.org/) as a CSS pre-processor, and [Gulp](http://gulpjs.com/) as build tool to stitch everything together. + +Templates and source files are located in the `dev` folder. The source files are populated with data from the `data` folder and processed using Gulp. During devlopment, the compiled but un-optimized assets remain in the dev folder. During a production build, these assets are optimized and copied to the `prod` folder. + +This diagram represents the flow of data and rough order of operation in the portals repository. The data is collected from multiple sources, processed and passed to the templates for rendering. + +[Portals repository dataflow diagram](portals-dataflow.jpg) \ No newline at end of file diff --git a/docs/architecture/css.md b/docs/architecture/css.md new file mode 100644 index 0000000..162c181 --- /dev/null +++ b/docs/architecture/css.md @@ -0,0 +1,39 @@ +# Wikipedia.org technical Documentation +**TOC** + +- [Overview](../README.md) +- Architecture of www.wikipedia.org + - [Data sources](data.md) + - [l10n](l10n.md) + - [HTML](html.md) + - **CSS** + - [Images](images.md) + - [JS](javascript.md) +- Development Process + - [Getting Started](../development/getting_started.md) + - [Gulp Tasks](../development/gulp.md) + - [Production Builds](../development/prod.md) + - [Deployment](../development/deploy.md) + - [Sister Project Portals](../development/sister_portals.md) + +--- +## CSS +CSS is processed using [PostCSS](http://postcss.org/). This processor provides features such as auto-prefixing and future friendly CSS capabilities like CSS custom-properties (variables). CSS is also linted with Stylelint, using [wikimedia-stylelint-config](https://github.com/wikimedia/stylelint-config-wikimedia). CSS is divided into a ‘main’ `style.css` file and 'partials', which are CSS files starting with an underscore that get imported into the main CSS file. Imports are done using [postcss-import](https://github.com/postcss/postcss-import). This plugin allows us to create a “SASS” style CSS structure, with many partials and one main file. It also allows us to import styles from NPM modules, such as the variables defined in [wikimedia-ui-base.css](https://phabricator.wikimedia.org/source/wikimedia-ui-base/browse/master/wikimedia-ui-base.css) . + +Processed CSS is outputted to the `dev/wikipedia.org/assets/css` folder for development. For production, the CSS is minified and **inlined** into the `index.html` file to reduce HTTP requests. A special CSS related file, `sprite-template.mustache` is also defined in `dev/wikipedia.org/assets/css`. This file is the template for the SVG image sprite, which uses CSS backgrounds to place images on the wikipedia.org page. The sprite template is compiled into a `sprite.css` file and outputted to the same directory. It is also minified and inlined for production purposes. + +**CSS directory overview** + +``` +|- dev/ + |- wikipedia.org/ + |- assets + |- css + |- sprite-template.mustache <- sprite CSS template + |- sprite.css <- compiled sprite CSS template + |- style.css <- compiled main PostCSS file + |- postcss + |- style.css <- main postCSS file + |- _footer.css <- partial postCSS files + |- etc... +``` diff --git a/docs/architecture/data.md b/docs/architecture/data.md new file mode 100644 index 0000000..3d3b6b4 --- /dev/null +++ b/docs/architecture/data.md @@ -0,0 +1,42 @@ +# Wikipedia.org technical Documentation +**TOC** + +- [Overview](../README.md) +- Architecture of www.wikipedia.org + - **Data sources** + - [l10n](l10n.md) + - [HTML](html.md) + - [CSS](css.md) + - [Images](images.md) + - [JS](javascript.md) +- Development Process + - [Getting Started](../development/getting_started.md) + - [Gulp Tasks](../development/gulp.md) + - [Production Builds](../development/prod.md) + - [Deployment](../development/deploy.md) + - [Sister Project Portals](../development/sister_portals.md) + +--- +## Data sources + +In order to populate the wikipedia.org page with up-to-date article counts and translation strings, we use three sources of data: + +1. **Page-count data** retrieved from Wikimedia Labs database replicas, by [page-counts.json](https://github.com/MaxSem/pagecounts), a tool hosted on Wikimedia toolForge that retrieves article counts of all Wikimedia projects, and outputs them in JSON format. +2. **Page-View data** retrieved from the [Wikimedia data dumps](https://dumps.wikimedia.org/other/pageviews/) by the `site-stats.js` script in the `data` folder. This script pulls down the hourly page-view data (formatted as CSV) for each project and places them in the `cache` folder. The script then parses the data, which is used to order the top-ten links around the globe. +3. **l10n** strings retrieved from translatewiki, which get pushed to the repo by an automated bot and placed in the `l10n` folder at the root of the repo. + +The scripts in the `data` directory assemble the page-counts, view-counts and translations for each project, and write the resulting file `site-stats.json`, which contains all three of these data sources, to the `data` directory. This file, along with the methods in `stats.js`, provide the data that can be consumed by the HTML templates. + +**Data directory overview** + +``` +data/ + |- site-stats.json <- file with combined page-counts & page-views + |- site-stats.js <- pulls down page-view data & generates site-stats.json + |- stats.js <- methods for retrieving site-stats ranges and l10n + |- l10n-overrides.json <- additional non-standard l10n data +l10n/ <- l10n strings from translatewiki + |- en.json + |- fr.json + |- etc... +``` diff --git a/docs/architecture/html.md b/docs/architecture/html.md new file mode 100644 index 0000000..5822ae5 --- /dev/null +++ b/docs/architecture/html.md @@ -0,0 +1,38 @@ +# Wikipedia.org technical Documentation +**TOC** + +- [Overview](../README.md) +- Architecture of www.wikipedia.org + - [Data sources](data.md) + - [l10n](l10n.md) + - **HTML** + - [CSS](css.md) + - [Images](images.md) + - [JS](javascript.md) +- Development Process + - [Getting Started](../development/getting_started.md) + - [Gulp Tasks](../development/gulp.md) + - [Production Builds](../development/prod.md) + - [Deployment](../development/deploy.md) + - [Sister Project Portals](../development/sister_portals.md) + +--- +## HTML templates +[Handlebars](http://handlebarsjs.com/) is used as the HTML template language. The reason that Handlebars was chosen for this purpose is that it provides the ability to use conditionals in templates, as well the ability to define custom helper functions, which are located in the root of the repo in the `hbs-helpers.global.js` file. + +Handlebars templates are compiled to HTML in a Gulp task using the [gulp-compile-handlebars](https://www.npmjs.com/package/gulp-compile-handlebars) plugin. This plugin uses data defined in the `controller.js` file to populate the templates. The templates consist of a main template, `index.handlebars`, and 'partial' templates, which get included in the main template. `index.handlebars` gets compiled into `index.html` and outputted into the dev directory during development, and optimizes and copied to the `prod` directory for production. + +**Template directory overview** + +``` +|- dev/ + |- wikipedia.org/ + |- index.handlebars <- main handlebars template, combines partials + |- index.html <- compiles handlebars template for dev + |- controller.js <- retrieves template data & outputs l10n files + |- templates/ <- template partials + |- header.handlebars + |- footer.handlebars + |- etc... +|- hbs-helpers.global.js <- handlebars helper functions +``` diff --git a/docs/architecture/images.md b/docs/architecture/images.md new file mode 100644 index 0000000..39a1b89 --- /dev/null +++ b/docs/architecture/images.md @@ -0,0 +1,48 @@ +# Wikipedia.org technical Documentation +**TOC** + +- [Overview](../README.md) +- Architecture of www.wikipedia.org + - [Data sources](data.md) + - [l10n](l10n.md) + - [HTML](html.md) + - [CSS](css.md) + - **Images** + - [JS](javascript.md) +- Development Process + - [Getting Started](../development/getting_started.md) + - [Gulp Tasks](../development/gulp.md) + - [Production Builds](../development/prod.md) + - [Deployment](../development/deploy.md) + - [Sister Project Portals](../development/sister_portals.md) + +--- +## Images +Images for www.wikipedia.org are placed in the `dev/wikipedia.org/assets/img` directory. SVG format is preferred for images because it is a resolution independent (vector) format, which can produce smaller file-sizes in most cases. When adding a new image, the choice of whether it should be SVG or PNG should be made based on which format produces the smaller file-size (after optimization and GZIP-ing). + +SVG images are combined into a single “sprite” image using [gulp-svg-sprite](https://github.com/jkphl/gulp-svg-sprite) and placed on the page using CSS backgrounds. A special CSS template file called `sprite-template.mustache ` is used to generate the CSS for the SVG sprite. The sizes of the SVG sprites are defined by the width and height attributes of the SVG files. + +Any SVG file placed in the `dev/wikipedia.org/assets/img/sprite_assets/` directory will be combined into a single file, named `sprite-{cache-busting-suffix}.svg` . Any newly added SVG files should follow the proposed [SVG markup guidelines](https://phabricator.wikimedia.org/T178867) which includes running them through SVGO with the `--pretty` option. A PNG fallback for the SVG sprite is is also created via a Gulp task using [gulp-svg2png](https://www.npmjs.com/package/gulp-svg2png). + +A few images on www.wikipedia.org are *not* in SVG format due to their complexity and the resulting large SVG size, which is larger than the PNG version. These image are the Wikipedia globe logo and the Wikinews logo. These two images are placed in the `dev/wikipedia.org/assets/img` directory at 1x, 1.5x and 2x sizes. These sizes match the sizes of images in MediaWiki content and skins. These images along with the SVG sprite fallback image are optimized with [pngquant](https://pngquant.org/) via a Gulp task for production usage. + +**Image directory structure** + +``` +|- dev/ + |- wikipedia.org/ + |- assets + |- css + |- sprite-template.mustache <- sprite CSS template + |- sprite.css <- compiled sprite CSS template + |- img <- main image directory + |- sprite_assets/ <- SVG sprite source files + |- Wiktionary-logo_sister.svg + |- Wikisource-logo_sister.svg + |- etc... + |- sprite-{cache-buster}.svg <- compiled SVG sprite + |- sprite-{cache-buster}.png <- compiled sprite fallback + |- Wikipedia-logo.png <- remaining raster images + |- Wikinews-logo.png + |- etc... +``` diff --git a/docs/architecture/javascript.md b/docs/architecture/javascript.md new file mode 100644 index 0000000..ed67653 --- /dev/null +++ b/docs/architecture/javascript.md @@ -0,0 +1,54 @@ +# Wikipedia.org technical Documentation +**TOC** + +- [Overview](../README.md) +- Architecture of www.wikipedia.org + - [Data sources](data.md) + - [l10n](l10n.md) + - [HTML](html.md) + - [CSS](css.md) + - [Images](images.md) + - **JS** +- Development Process + - [Getting Started](../development/getting_started.md) + - [Gulp Tasks](../development/gulp.md) + - [Production Builds](../development/prod.md) + - [Deployment](../development/deploy.md) + - [Sister Project Portals](../development/sister_portals.md) + +--- + +## Javascript +> Javascript is linted with [eslint](https://eslint.org/) following [eslint-config-wikimedia](https://www.npmjs.com/package/eslint-config-wikimedia) and should follow the [MediaWiki Javascript coding conventions](https://www.mediawiki.org/wiki/Manual:Coding_conventions/JavaScript). + +The main javascript features of www.wikipedia.org include: + +- client-side translation +- search-as-you-type suggestions with thumbnails and descriptions +- expanding/collapsing the wiki languages list +- event-logging +- And more! + +All these features and more live in the `dev/wikipedia.org/assets/js` directory. JS features are separated into individual files, with each file being encapsulated in an [IIFE](https://en.wikipedia.org/wiki/Immediately-invoked_function_expression). If a features needs to be shared across different files, it is exposed as a global variable, assigned to an IIFE (see `wm-test.js` as an example). This lets us follow a [revealing module pattern](https://addyosmani.com/resources/essentialjsdesignpatterns/book/#revealingmodulepatternjavascript) for separate features. + +There is no ES6 style module-loading on www.wikipedia.org. The javascript files are included in the order they are needed, before the closing body tag in the `index.handlebars` file. The script tags are placed between special comment tags starting with `<!-- build:js` . For a production build, the files between these comments are concatenated using [gulp-useref](https://www.npmjs.com/package/gulp-useref), given a cache-busing suffix with [gulp-rev](https://github.com/sindresorhus/gulp-rev) and minified with [gulp-uglify](https://www.npmjs.com/package/gulp-uglify). The minified JS file is then placed in the `prod` directory. + +**Javascript directory structure** + +``` +|- dev/ + |- wikipedia.org/ + |- index.handlebars <- js files are included here before the ending body tag + |- index.html <- compiled html includes links to js source files for dev purposes + |- assets + |- js <- source javascript files + |- polyfills.js + |- wm-test.js + |- etc... +|- prod/ + |- wikipedia.org/ + |- assets/ + |- js + |- index-47f5f07682.js <- concatenated & minified JS file for production +``` + diff --git a/docs/architecture/l10n.md b/docs/architecture/l10n.md new file mode 100644 index 0000000..e24a46c --- /dev/null +++ b/docs/architecture/l10n.md @@ -0,0 +1,49 @@ + +# Wikipedia.org technical Documentation +**TOC** + +- [Overview](../README.md) +- Architecture of www.wikipedia.org + - [Data sources](data.md) + - **l10n** + - [HTML](html.md) + - [CSS](css.md) + - [Images](images.md) + - [JS](javascript.md) +- Development Process + - [Getting Started](../development/getting_started.md) + - [Gulp Tasks](../development/gulp.md) + - [Production Builds](../development/prod.md) + - [Deployment](../development/deploy.md) + - [Sister Project Portals](../development/sister_portals.md) + +--- +## L10n +Localization strings for wikipedia.org are provided by translatewiki. Translations from translatewiki are submitted and automatically merged into the repository by a bot (l10n-bot). The translation strings are formatted in JSON and placed in the `l10n` directory, with one file per language. + +> **DO NOT EDIT THE L10N FOLDER MANUALLY, CHANGES WILL BE OVERRIDDEN BY THE L10N-BOT.** + +## Javascript translations +The translation mechanism on www.wikipedia.org is handled by javascript in the browser. By default, the page is server-rendered in english, and when a different browser language is detected, the page is then translated by calling an AJAX request for the appropriate translation, and the new translation strings are replaced in the DOM. The translation is then stored in localStorage for subsequent requests. + +The javascript-based translation mechanism is a short-coming of the current architecture (server-rendering localized pages would be ideal) but late l10n is better than no l10n. The l10n strings with all the necessary info (l10n+stats for a given language wiki) are outputted into the dev directory as json files by the `/dev/wikipedia.org/controller.js` file. The JSON translation files are then available to be fetched by the browser. The l10n files are suffixed with a cache-busting hash to ensure the browser fetches the latest version. For production builds, these files are copied to the `prod` directory. + +**l10n directory structure** + +``` +l10n/ + |- en.json <- source translation files + |- fr.json + |- etc... +dev/ + |- wikipedia.org/ + |- assets/ + |- l10n/ + |- en-03ecca8f.json <- compiled l10n files + |- fr-03ecca8f.json + |- etc... + |- controller.js <- generates compiled l10n files +``` + +## Editing translations +Translations are edited through translatewiki.net. To edit a translation, [visit this page](https://translatewiki.net/w/i.php?title=Special:Translate&filter=&group=wikimedia-portals&optional=1&task=custom), select the language you wish to translate to, and proceed to edit a translation. The translations will be merged into the repository automatically by a bot. \ No newline at end of file diff --git a/docs/development/deploy.md b/docs/development/deploy.md new file mode 100644 index 0000000..ebc7f43 --- /dev/null +++ b/docs/development/deploy.md @@ -0,0 +1,57 @@ +# Wikipedia.org technical Documentation +**TOC** + +- [Overview](../README.md) +- Architecture of www.wikipedia.org + - [Data sources](../architecture/data.md) + - [l10n](../architecture/l10n.md) + - [HTML](../architecture/html.md) + - [CSS](../architecture/css.md) + - [Images](../architecture/images.md) + - [JS](../architecture/javascript.md) +- Development Process + - [Getting Started](getting_started.md) + - [Gulp Tasks](gulp.md) + - [Production Builds](prod.md) + - **Deployment** + - [Sister Project Portals](sister_portals.md) + +--- + +## Creating a deployment patch + +The Wikimedia Portals are placed in a submodule in the `mediawiki-config` repository. To Deploy a new build of the Wikimedia portals, a new patch that updates the submodule must be submitted to `mediawiki-config`. + +From the `mediawiki-config` repository, run `git submodule init portals` (on first run), then `git submodule update --remote portals` to bump the portals to the head of the master branch. Then, create a commit with the message + +``` +Bumping portals to master + +Bug: T128546 +``` + +and push the commit to Gerrit. The commit should be merged just before a deployment is about to take place. + +## Deploying the Wikimedia portals +From the deployment server (tin), run + +``` +cd /srv/mediawiki-staging && git pull && git submodule update portals +``` + +To test the portals on mwdebug, login to a debug server (e.g. mwdebug1002) and run + +``` +scap pull +``` + +If all looks well, then on the deployment server, run + +``` +cd portals/ +./sync-portals 'Wikimedia Portals Update: [[gerrit:{{patchID}}|Bumping portals to master (T128546)]]' +``` + +Where `{{patchID}}` is the patch ID of the `mediawiki-config` patch that updates the portals submodule. + +The script `.sync-portals` runs scap and purges all the urls listed in the `urls-to-purge.txt` file at the root of the repo. \ No newline at end of file diff --git a/docs/development/getting_started.md b/docs/development/getting_started.md new file mode 100644 index 0000000..41d0255 --- /dev/null +++ b/docs/development/getting_started.md @@ -0,0 +1,68 @@ +# Wikipedia.org technical Documentation +**TOC** + +- [Overview](../README.md) +- Architecture of www.wikipedia.org + - [Data sources](../architecture/data.md) + - [l10n](../architecture/l10n.md) + - [HTML](../architecture/html.md) + - [CSS](../architecture/css.md) + - [Images](../architecture/images.md) + - [JS](../architecture/javascript.md) +- Development Process + - **Getting Started** + - [Gulp Tasks](gulp.md) + - [Production Builds](prod.md) + - [Deployment](deploy.md) + - [Sister Project Portals](sister_portals.md) + +--- + +## Getting Started + +Prerequisites: +- Node v6.11 or greater +- NPM v3.8 or greater + +To download all dependencies, run: + +``` +npm install +``` + +### Starting a development server + +**The portals repo does not provide a developement server**. To view changes you make while developing, you will need to run an HTTP server pointing to the root of this repository. This can be as easy as running one of the following commands from the root directory of the portals repository. + +Starting a python simple server: + +``` +python -m SimpleHTTPServer 8080 +``` + +Or starting a PHP server: + +``` +php -S 0.0.0.0:8080 +``` + +Then, you should be able to access the development directory at `http://localhost:8080/dev/wikipedia.org` and the production directory at `http://localhost:8080/prod/wikipedia.org`. + +### Watching for changes + +To watch for changes while you're developing, run: + +``` +gulp watch --portal=wikipedia.org +``` + +This will watch for changes in `dev/wikipedia.org` and recompile the HTML/CSS/JS and images if they change. The changes will be visible when then browser is reloaded. + +### Updating Stats +To download the latest stats (article-counts and page-views) for the portal pages, run: + +``` +gulp update-stats +``` + +After that task completes, the next time you run the watch command the wikipedia.org dev page will be updated with the latest stats. \ No newline at end of file diff --git a/docs/development/gulp.md b/docs/development/gulp.md new file mode 100644 index 0000000..1827438 --- /dev/null +++ b/docs/development/gulp.md @@ -0,0 +1,43 @@ +# Wikipedia.org technical Documentation +**TOC** + +- [Overview](../README.md) +- Architecture of www.wikipedia.org + - [Data sources](../architecture/data.md) + - [l10n](../architecture/l10n.md) + - [HTML](../architecture/html.md) + - [CSS](../architecture/css.md) + - [Images](../architecture/images.md) + - [JS](../architecture/javascript.md) +- Development Process + - [Getting Started](getting_started.md) + - **Gulp Tasks** + - [Production Builds](prod.md) + - [Deployment](deploy.md) + - [Sister Project Portals](sister_portals.md) + +--- + +## Running Gulp Tasks +Gulp is a dependancy of this repository, but it's also common to install Gulp globally with `npm install -g gulp`. + +The Gulp tasks are designed to accommodate building all portal pages, not just the wikipedia.org portal (however, currently only the wikipedia.org portal is setup to build with the Gulp tasks). See the [Sister Project Portals](sister_portals.md) section for details. + +The Gulp tasks take a `--portal` parameter to specify which portal directory the task should be run on. Until other project portals are brought into the Gulp workflow, this parameter will usually be `--portal=wikipedia.org`. + +**The major Gulp tasks include:** + +- `lint` - Lints JS and CSS. +- `compile-handlebars` - Compiles Handlebars templates. +- `svgSprite` - Creates the SVG sprite and it's PNG fallback. +- `postcss` - Compiles PostCSS. +- `inline-assets` - Inlines the CSS into the index.html file (for production build). +- `clean-prod-js` - Removes JS files from `prod` directory (for production build). +- `concat-minify-js` - Concatenates and minifies JS (for production build). +- `minify-html` - Cleans index.html (for production build). +- `copy-images` - Copies images to `prod` directory (for production build). +- `copy-translation-files` - Copies translation files from `dev/wikipedia.org/assets/l10n` to `prod` (for production build). +- `update-urls-to-purge` - Updates `urls-to-purge.txt` file (for deployment). +- `fetch-meta` - downloads the latest versions of the sister project portals into the `prod` directory. + +The command `gulp help` can be run to print out a list of the major gulp tasks on the command line. \ No newline at end of file diff --git a/docs/development/prod.md b/docs/development/prod.md new file mode 100644 index 0000000..9dff12f --- /dev/null +++ b/docs/development/prod.md @@ -0,0 +1,37 @@ +# Wikipedia.org technical Documentation +**TOC** + +- [Overview](../README.md) +- Architecture of www.wikipedia.org + - [Data sources](../architecture/data.md) + - [l10n](../architecture/l10n.md) + - [HTML](../architecture/html.md) + - [CSS](../architecture/css.md) + - [Images](../architecture/images.md) + - [JS](../architecture/javascript.md) +- Development Process + - [Getting Started](getting_started.md) + - [Gulp Tasks](gulp.md) + - **Production Builds** + - [Deployment](deploy.md) + - [Sister Project Portals](sister_portals.md) + +--- + +## Creating a Production Build + +A build of the Wikimedia portals is created automatically each week by a Jenkins Job in CI that runs every mondays at 9:30 A.M. UTC. +This build is run by the `PortalsBuilder` user, whose patch history can be [viewed here](https://gerrit.wikimedia.org/r/#/q/owner:releng%2540lists.wikimedia.org). +A changelog of each build is also available and can be [viewed here](https://integration.wikimedia.org/ci/job/wikimedia-portals-build/changes). + +It is also possible to manually create a new build in the event that an error must be immediately fixed. +To run a manual build, first update the stats, then run the following commands: + +1. `gulp update-stats` Updates stats. +2. `gulp --portal=wikipedia.org` Runs the build of wikipedia.org. +3. `gulp fetch-meta --portal=all` Fetch updates for the remaining project portals +4. Commit the changes to Gerrit. + +These Gulp task have also been combined into a single NPM script: `npm run build-all-portals`. + +Running a build produces a high amount of changes in Git because each l10n file is appended with a cache-busting hash at the end of the file name. When a single translation changes, all the translations get updated with a new cache-busting hash, and therefore, all the l10n file names get changed as well. For this reason, it is typically not necessary to commit a new build of the portals for each patch submitted to Gerrit. \ No newline at end of file diff --git a/docs/development/sister_portals.md b/docs/development/sister_portals.md new file mode 100644 index 0000000..f500a81 --- /dev/null +++ b/docs/development/sister_portals.md @@ -0,0 +1,28 @@ +# Wikipedia.org technical Documentation +**TOC** + +- [Overview](../README.md) +- Architecture of www.wikipedia.org + - [Data sources](../architecture/data.md) + - [l10n](../architecture/l10n.md) + - [HTML](../architecture/html.md) + - [CSS](../architecture/css.md) + - [Images](../architecture/images.md) + - [JS](../architecture/javascript.md) +- Development Process + - [Getting Started](getting_started.md) + - [Gulp Tasks](gulp.md) + - [Production Builds](prod.md) + - [Deployment](deploy.md) + - **Sister Project Portals** + +--- + +## Updating sister project portals + +The portals for sister projects are not updated in the same way as the wikipedia.org portal. These portal pages are still updated through Lua templates on mediawiki.org. They are however, deployed to production in the same way as the wikipedia.org portal (see the [deployment](deploy.md) section for details). This sub optimal arrangement was settled upon due to the resource constraints at the Wikimedia Foundation, where efforts were focused on updating the wikipedia.org portal only. + +### Sister project templates +[The Project portals](https://meta.wikimedia.org/wiki/Project_portals) page on metawiki has links to the templates used to update the sister project portals. The main template used to generate these portals is [Module:Project_portal](https://meta.wikimedia.org/wiki/Module:Project_portal), which takes the list of wikis from [Module:Project_portal/wikis](https://meta.wikimedia.org/wiki/Module:Project_portal/wikis) and stats for those wikis from [Module:Project_portal/views](https://meta.wikimedia.org/wiki/Module:Project_portal/views). + +Administrator access is required to update these templates. Once an administrator has made updates to these portals, they are pulled into the repository with the gulp task `gulp fetch-meta --portals=all`. They can then be committed to Gerrit for deployment. \ No newline at end of file diff --git a/docs/portals-dataflow.jpg b/docs/portals-dataflow.jpg new file mode 100644 index 0000000..338f7a4 --- /dev/null +++ b/docs/portals-dataflow.jpg Binary files differ -- To view, visit https://gerrit.wikimedia.org/r/396407 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: merged Gerrit-Change-Id: If4f8f867d16d5bcca04d04fd9423edba2a5295cc Gerrit-PatchSet: 2 Gerrit-Project: wikimedia/portals Gerrit-Branch: master Gerrit-Owner: Jdrewniak <[email protected]> Gerrit-Reviewer: Debt <[email protected]> Gerrit-Reviewer: Jdrewniak <[email protected]> Gerrit-Reviewer: Mxn <[email protected]> Gerrit-Reviewer: Phuedx <[email protected]> Gerrit-Reviewer: VolkerE <[email protected]> Gerrit-Reviewer: jenkins-bot <> _______________________________________________ MediaWiki-commits mailing list [email protected] https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
