[MediaWiki-commits] [Gerrit] mediawiki...TemplateData[master]: Extend 'format' spec to include custom format strings.
Cscott has uploaded a new change for review.
https://gerrit.wikimedia.org/r/306975
Change subject: Extend 'format' spec to include custom format strings.
..
Extend 'format' spec to include custom format strings.
Thanks to Thiemo Mättig for the suggestion and specification at
Wikimania 2016 in Esino Lario.
This is an extended version of Thiemo's original specification. This
version also allows specification of "own line" properties for
templates; that is, whether the template should be preceded/followed
by a newline, as requested by James Forrester.
Bug: T138492
Bug: T135667
Change-Id: Idc6b2680330e6bf5caec2bf6fc86a705d25bc649
---
M Specification.md
M TemplateDataBlob.php
M extension.json
M i18n/en.json
M i18n/qqq.json
M modules/ext.templateDataGenerator.ui.tdDialog.js
M resources/ext.templateData.css
M tests/ext.templateData.tests.js
M tests/phpunit/TemplateDataBlobTest.php
9 files changed, 318 insertions(+), 36 deletions(-)
git pull ssh://gerrit.wikimedia.org:29418/mediawiki/extensions/TemplateData
refs/changes/75/306975/1
diff --git a/Specification.md b/Specification.md
index 3929441..d0133b9 100644
--- a/Specification.md
+++ b/Specification.md
@@ -98,25 +98,14 @@
Authors MUST ensure that the `maps` object contains only `Map` objects.
Authors MAY include a parameter in multiple `Map` objects. Authors are NOT
REQUIRED to reference each parameter in at least one `Map` object.
3.1.6 `format`
-* Value: `null` or `string` of either `'inline'` or `'block'`
+* Value: `null` or `FormatString` or `string` of either `'inline'` or `'block'`
* Default: `null`
How the template's wikitext representation SHOULD be laid out. Authors MAY
choose to use this parameter to express that a template will be better
understood by other human readers of the wikitext representation if a template
is in one form or the other.
-If the parameter is set to `'block'`, Consumers SHOULD create a wikitext
representation with a single newline after the template invocation and each
parameter value, a single space between each pipe and its subsequent parameter
key, and a space either side of the assignment separator between the parameter
key and value, like so:
+If the parameter is set to `'block'`, it MUST be interpreted as the format
string `'{{_\n| _ = _\n}}'`. If the parameter is set to `'inline'` it MUST be
interpreted as the format string `'{{_|_=_}}'`.
-```
-{{Foo
-| bar = baz
-| qux = quux
-}}
-```
-
-If the parameter is set to `'inline'`, Consumers SHOULD create a wikitext
representation with no whitespace, like so:
-
-```
-{{Foo|bar=baz|qux=quux}}
-```
+If the parameter is not null, Consumers SHOULD create a wikitext
representation corresponding to the given format string, as described in
section 3.7.
If the parameter is set to `null`, Consumers SHOULD create the same
representation as for `'inline'` format for new template transclusions, and
SHOULD attempt to use the same formatting for new parameters as for existing
ones for existing transclusions that are edited.
@@ -289,6 +278,87 @@
The key corresponds to the name of a Consumer variable that relates to the
specified parameter(s).
+### 3.7 FormatString
+* Value: `string`
+
+A format string describes how whitespace should be added to a template
instantiation in wikitext. A format string looks like `{{_|_=_}}`, with
optional whitespace and extended underscores. Formally, its structure is given
by the following grammar:
+```
+FormatString = StartFormat ParameterFormat EndFormat
+StartFormat = nl? "{{" ws* Hole ws*
+ParameterFormat = nl? "|" nl? ws* Hole ws* "=" ws* Hole
+EndFormat = nl? ws* "}}" nl?
+Hole = "_"+
+ws = " "
+nl = "\n"
+```
+
+To format a template invocation according to the format string, first split it
in three parts, corresponding to `StartFormat`, `ParameterFormat`, and
`EndFormat` in the grammar above. In the following, to "replace the `Hole`"
with some value means to first pad a non-empty value with spaces on the right
until it is at least as many characters long as the replaced underscore
sequence. Zero-length values are not padded.
+
+Begin with `StartFormat`, and replace the `Hole` with the name of the template
to create the "output string". If `StartFormat` begins with a newline and
template is already at the start of a line (the character preceding this
template invocation is a newline or the template is at the start of the
output), delete the initial newline from the output string.
+
+For each parameter, append the `ParameterFormat` to the output string after
replacing the first `Hole` with the name of the parameter and the second `Hole`
with the value of the parameter.
+
+Finally, append the `EndFormat` to the output string.
+
+Some example format strings:
+
+Inline formatting: `{{_|_=_}}`
+```
+{{Foo|bar=baz|qux=quux}}{{Bar}}
+```
+
+Block formatting: `{{_\n| _ = _\n}}`
+```
+{{Foo
+| bar = baz
+| qux = quux
+}
[MediaWiki-commits] [Gerrit] mediawiki...ocg-collection[wmf-deploy]: Merge branch 'master' into wmf-deploy
Cscott has submitted this change and it was merged. Change subject: Merge branch 'master' into wmf-deploy .. Merge branch 'master' into wmf-deploy Change-Id: I5f8f1f2a1f7da85fbac99ff7183d055b42138869 --- 0 files changed, 0 insertions(+), 0 deletions(-) Approvals: Cscott: Verified; Looks good to me, approved -- To view, visit https://gerrit.wikimedia.org/r/305393 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: merged Gerrit-Change-Id: I5f8f1f2a1f7da85fbac99ff7183d055b42138869 Gerrit-PatchSet: 1 Gerrit-Project: mediawiki/services/ocg-collection Gerrit-Branch: wmf-deploy Gerrit-Owner: Cscott Gerrit-Reviewer: Cscott ___ MediaWiki-commits mailing list MediaWiki-commits@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] mediawiki...ocg-collection[master]: Updating to latest masters
Cscott has submitted this change and it was merged. Change subject: Updating to latest masters .. Updating to latest masters Deployed new bundler (40ba27f770480cb1642ed58ef18b986ec67b0daf): * Remove rest.wikimedia.org, the server is being shut down Sept 1. (T133001) Deployed new service (51d2eb6d47dfc5225d8d5bc45444e81cc233b1c7): * Add metrics for number of items in collections. Deployed new texter (6e2c2d04f00c379ac35cac9d85af7096e767): * Add option to emit `blessed`-style markup. Change-Id: Ie7fe1923b87e29a1f386e2eab2064ffafef93d46 --- M mw-ocg-bundler M mw-ocg-service M mw-ocg-texter 3 files changed, 3 insertions(+), 3 deletions(-) Approvals: Cscott: Verified; Looks good to me, approved diff --git a/mw-ocg-bundler b/mw-ocg-bundler index e339630..40ba27f 16 --- a/mw-ocg-bundler +++ b/mw-ocg-bundler @@ -1 +1 @@ -Subproject commit e33963029ff99f63a54fae78515b742375d88a3c +Subproject commit 40ba27f770480cb1642ed58ef18b986ec67b0daf diff --git a/mw-ocg-service b/mw-ocg-service index ac2dff0..51d2eb6 16 --- a/mw-ocg-service +++ b/mw-ocg-service @@ -1 +1 @@ -Subproject commit ac2dff089f66fd5f6d50d7e7cd046cfd5558ac59 +Subproject commit 51d2eb6d47dfc5225d8d5bc45444e81cc233b1c7 diff --git a/mw-ocg-texter b/mw-ocg-texter index df26683..6e2 16 --- a/mw-ocg-texter +++ b/mw-ocg-texter @@ -1 +1 @@ -Subproject commit df26683c689eb282dba55b0f287fd71307c4a2e2 +Subproject commit 6e2c2d04f00c379ac35cac9d85af7096e767 -- To view, visit https://gerrit.wikimedia.org/r/305391 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: merged Gerrit-Change-Id: Ie7fe1923b87e29a1f386e2eab2064ffafef93d46 Gerrit-PatchSet: 1 Gerrit-Project: mediawiki/services/ocg-collection Gerrit-Branch: master Gerrit-Owner: Cscott Gerrit-Reviewer: Cscott ___ MediaWiki-commits mailing list MediaWiki-commits@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] mediawiki...ocg-collection[wmf-deploy]: Merge branch 'master' into wmf-deploy
Cscott has uploaded a new change for review. https://gerrit.wikimedia.org/r/305393 Change subject: Merge branch 'master' into wmf-deploy .. Merge branch 'master' into wmf-deploy Change-Id: I5f8f1f2a1f7da85fbac99ff7183d055b42138869 --- 0 files changed, 0 insertions(+), 0 deletions(-) git pull ssh://gerrit.wikimedia.org:29418/mediawiki/services/ocg-collection refs/changes/93/305393/1 -- To view, visit https://gerrit.wikimedia.org/r/305393 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I5f8f1f2a1f7da85fbac99ff7183d055b42138869 Gerrit-PatchSet: 1 Gerrit-Project: mediawiki/services/ocg-collection Gerrit-Branch: wmf-deploy Gerrit-Owner: Cscott ___ MediaWiki-commits mailing list MediaWiki-commits@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] mediawiki...ocg-collection[master]: Updating to latest masters
Cscott has uploaded a new change for review. https://gerrit.wikimedia.org/r/305391 Change subject: Updating to latest masters .. Updating to latest masters Deployed new bundler (40ba27f770480cb1642ed58ef18b986ec67b0daf): * Remove rest.wikimedia.org, the server is being shut down Sept 1. (T133001) Deployed new service (51d2eb6d47dfc5225d8d5bc45444e81cc233b1c7): * Add metrics for number of items in collections. Deployed new texter (6e2c2d04f00c379ac35cac9d85af7096e767): * Add option to emit `blessed`-style markup. Change-Id: Ie7fe1923b87e29a1f386e2eab2064ffafef93d46 --- M mw-ocg-bundler M mw-ocg-service M mw-ocg-texter 3 files changed, 3 insertions(+), 3 deletions(-) git pull ssh://gerrit.wikimedia.org:29418/mediawiki/services/ocg-collection refs/changes/91/305391/1 diff --git a/mw-ocg-bundler b/mw-ocg-bundler index e339630..40ba27f 16 --- a/mw-ocg-bundler +++ b/mw-ocg-bundler @@ -1 +1 @@ -Subproject commit e33963029ff99f63a54fae78515b742375d88a3c +Subproject commit 40ba27f770480cb1642ed58ef18b986ec67b0daf diff --git a/mw-ocg-service b/mw-ocg-service index ac2dff0..51d2eb6 16 --- a/mw-ocg-service +++ b/mw-ocg-service @@ -1 +1 @@ -Subproject commit ac2dff089f66fd5f6d50d7e7cd046cfd5558ac59 +Subproject commit 51d2eb6d47dfc5225d8d5bc45444e81cc233b1c7 diff --git a/mw-ocg-texter b/mw-ocg-texter index df26683..6e2 16 --- a/mw-ocg-texter +++ b/mw-ocg-texter @@ -1 +1 @@ -Subproject commit df26683c689eb282dba55b0f287fd71307c4a2e2 +Subproject commit 6e2c2d04f00c379ac35cac9d85af7096e767 -- To view, visit https://gerrit.wikimedia.org/r/305391 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: newchange Gerrit-Change-Id: Ie7fe1923b87e29a1f386e2eab2064ffafef93d46 Gerrit-PatchSet: 1 Gerrit-Project: mediawiki/services/ocg-collection Gerrit-Branch: master Gerrit-Owner: Cscott ___ MediaWiki-commits mailing list MediaWiki-commits@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] mediawiki...OfflineContentGenerator[master]: Add metrics for number of items in collections
Cscott has submitted this change and it was merged.
Change subject: Add metrics for number of items in collections
..
Add metrics for number of items in collections
For planning purposes, we would like to know how many requests are for
single-page vs. multi-page collections. To this end, it seems useful
to record the size of requested collections in a metric.
This patch adds a timing metric for this purpose, which gives us
percentiles in graphite. Additionally, it adds separate counters for
single page & multi-page requests, which should let us establish the
exact ratio of the two over time.
Change-Id: I0f91778f0d71bbb66b719b6b3e5dd19a4de61962
---
M lib/threads/frontend.js
1 file changed, 27 insertions(+), 9 deletions(-)
Approvals:
Cscott: Verified; Looks good to me, approved
Objections:
jenkins-bot: Fails
diff --git a/lib/threads/frontend.js b/lib/threads/frontend.js
index 25894e0..c67ba94 100644
--- a/lib/threads/frontend.js
+++ b/lib/threads/frontend.js
@@ -488,15 +488,32 @@
'Cannot begin "render" command: parameter
"metabook" not specified',
400
);
- } else {
- metabookObj = JSON.parse(metabook);
- if (!metabookObj) {
- throw new FrontendError(
- 'Cannot begin "render" command:
parameter "metabook" not valid JSON',
- 400
- );
- }
- collectionId = createCollectionId(metabookObj, writer);
+ }
+ metabookObj = JSON.parse(metabook);
+ if (!metabookObj) {
+ throw new FrontendError(
+ 'Cannot begin "render" command: parameter
"metabook" not valid JSON',
+ 400
+ );
+ }
+ collectionId = createCollectionId(metabookObj, writer);
+
+ // Collect metrics on collection size. This is per render
request,
+ // independent of whether the job ends up being served from
cache
+ // or not. It also won't count "force render" re-requests,
which
+ // ought to have `collectionId` set.
+
+ // Use a timing metric, so that we get percentiles.
+ statsd.timing(
+ 'frontend.requests.render.itemCount',
+ metabookObj.items.length
+ );
+ // Also record single / multi page renders separately, so that
we can
+ // get a precise long-term ratio.
+ if (metabookObj.items.length === 1) {
+ statsd.increment('frontend.requests.render.singlePage');
+ } else if (metabookObj.items.length > 1) {
+ statsd.increment('frontend.requests.render.multiPage');
}
}
@@ -793,6 +810,7 @@
}
});
};
+
updateChapter(metabookObj.items);
// XXX in the future we might separately cache the bundle.zip and
--
To view, visit https://gerrit.wikimedia.org/r/304043
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: merged
Gerrit-Change-Id: I0f91778f0d71bbb66b719b6b3e5dd19a4de61962
Gerrit-PatchSet: 3
Gerrit-Project: mediawiki/extensions/Collection/OfflineContentGenerator
Gerrit-Branch: master
Gerrit-Owner: GWicke
Gerrit-Reviewer: Arlolra
Gerrit-Reviewer: Cscott
Gerrit-Reviewer: GWicke
Gerrit-Reviewer: Mobrovac
Gerrit-Reviewer: Ppchelko
Gerrit-Reviewer: Subramanya Sastry
Gerrit-Reviewer: jenkins-bot <>
___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] mediawiki...TemplateData[master]: Make cancelling edit dialog less confusing.
Cscott has uploaded a new change for review. https://gerrit.wikimedia.org/r/303850 Change subject: Make cancelling edit dialog less confusing. .. Make cancelling edit dialog less confusing. Bug: T142514 Change-Id: Id3a5dd75ec34c92d654c65f34723847251a3e52a --- M i18n/en.json 1 file changed, 1 insertion(+), 1 deletion(-) git pull ssh://gerrit.wikimedia.org:29418/mediawiki/extensions/TemplateData refs/changes/50/303850/1 diff --git a/i18n/en.json b/i18n/en.json index 0cbc992..a364a32 100644 --- a/i18n/en.json +++ b/i18n/en.json @@ -70,7 +70,7 @@ "templatedata-modal-button-importParams": "Import parameters", "templatedata-modal-button-restoreparam": "Restore parameter", "templatedata-modal-button-saveparam": "Save", - "templatedata-modal-confirmcancel": "Are you sure you want to cancel? Any changes will be lost.", + "templatedata-modal-confirmcancel": "Are you sure you want to discard your changes?", "templatedata-modal-current-language": "Current language: $1", "templatedata-modal-errormsg": "Errors found. Please make sure there are no empty or duplicate parameter names, and that the parameter name does not include \"$1\", \"$2\" or \"$3\".", "templatedata-modal-errormsg-import-noparams": "No new parameters found during import.", -- To view, visit https://gerrit.wikimedia.org/r/303850 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: newchange Gerrit-Change-Id: Id3a5dd75ec34c92d654c65f34723847251a3e52a Gerrit-PatchSet: 1 Gerrit-Project: mediawiki/extensions/TemplateData Gerrit-Branch: master Gerrit-Owner: Cscott ___ MediaWiki-commits mailing list MediaWiki-commits@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] mediawiki...TemplateData[master]: WIP: Extend 'format' spec to include format strings.
Cscott has uploaded a new change for review.
https://gerrit.wikimedia.org/r/303431
Change subject: WIP: Extend 'format' spec to include format strings.
..
WIP: Extend 'format' spec to include format strings.
TODO: validate format strings, use appropriate UI widget.
Bug: T114445
Change-Id: Ic7cfe0560efe72ff4058134bd6d3fd97b21ba5ed
---
M Specification.md
M TemplateDataBlob.php
M i18n/en.json
M modules/ext.templateDataGenerator.ui.tdDialog.js
4 files changed, 71 insertions(+), 19 deletions(-)
git pull ssh://gerrit.wikimedia.org:29418/mediawiki/extensions/TemplateData
refs/changes/31/303431/1
diff --git a/Specification.md b/Specification.md
index 3929441..c525778 100644
--- a/Specification.md
+++ b/Specification.md
@@ -98,25 +98,14 @@
Authors MUST ensure that the `maps` object contains only `Map` objects.
Authors MAY include a parameter in multiple `Map` objects. Authors are NOT
REQUIRED to reference each parameter in at least one `Map` object.
3.1.6 `format`
-* Value: `null` or `string` of either `'inline'` or `'block'`
+* Value: `null` or `FormatString` or `string` of either `'inline'` or `'block'`
* Default: `null`
How the template's wikitext representation SHOULD be laid out. Authors MAY
choose to use this parameter to express that a template will be better
understood by other human readers of the wikitext representation if a template
is in one form or the other.
-If the parameter is set to `'block'`, Consumers SHOULD create a wikitext
representation with a single newline after the template invocation and each
parameter value, a single space between each pipe and its subsequent parameter
key, and a space either side of the assignment separator between the parameter
key and value, like so:
+If the parameter is set to `'block'`, it MUST be interpreted as the format
string `'| _ = _\n'`. If the parameter is set to `'inline'` it MUST be
interpreted as the format string `'|_=_'`.
-```
-{{Foo
-| bar = baz
-| qux = quux
-}}
-```
-
-If the parameter is set to `'inline'`, Consumers SHOULD create a wikitext
representation with no whitespace, like so:
-
-```
-{{Foo|bar=baz|qux=quux}}
-```
+If the parameter is not null, Consumers SHOULD create a wikitext
representation corresponding to the given format string, as described in
section 3.7.
If the parameter is set to `null`, Consumers SHOULD create the same
representation as for `'inline'` format for new template transclusions, and
SHOULD attempt to use the same formatting for new parameters as for existing
ones for existing transclusions that are edited.
@@ -289,6 +278,67 @@
The key corresponds to the name of a Consumer variable that relates to the
specified parameter(s).
+### 3.7 FormatString
+* Value: `string`
+
+A format string describes how whitespace should be added to template
parameters in wikitext. A format string contains the characters `|_=_` (in
that order), where additional space characters may occur at the beginning of
the string, end of the string, and/or between any of the elements. Each
underscore character may additionally be replaced with a series of consecutive
underscores. In addition, a newline character can occur either at the end of
the string or after the vertical bar symbol.
+
+To format a parameter according to the format string, you would replace the
first (second) sequence of underscores with the name (value) of the parameter.
If there replaced sequence is more than a single underscore, the name or value
is padded with spaces on the right until it is at least as many characters long
as the replaced underscore sequence. (A single underscore does not create an
extra space when the parameter value is empty.)
+
+If the format string ends with a newline, an extra newline is added between
the template name and the first parameter. If there is a newline somewhere in
the format string, the closing }} is always put on an extra line, no matter
what the format string says (but two sequential newlines should never be
generated).
+
+Some example format strings:
+
+Inline formatting: `|_=_`
+```
+{{Foo|bar=baz|qux=quux}}
+```
+
+Block formatting: `| _ = _\n`
+```
+{{Foo
+| bar = baz
+| qux = quux
+}}
+```
+
+No space before the parameter name: `|_ = _\n`
+```
+{{Foo
+|bar = baz
+|qux = quux
+}}
+```
+
+Indent each parameter: ` |_ = _\n`
+```
+{{Foo
+ |bar = baz
+ |qux = quux
+}}
+```
+
+Align all parameter names to a given length: `|___ = _\n`
+```
+{{Foo
+|bar = baz
+|qux = quux
+|veryverylongparameter = bat
+}}
+```
+
+Pipe characters at the end of the previous line: `|\n_ = _`
+```
+{{Foo|
+bar = baz|
+qux = quux}}
+```
+
+Inline style with more spaces: ` | _ = _`
+```
+{{Foo | bar = baz | qux = quux}}
+```
+
## 4 Examples
### 4.1 The "Unsigned" template
diff --git a/TemplateDataBlob.php b/TemplateDataBlob.php
index b8c8078..25d1245 100644
--- a/Te
[MediaWiki-commits] [Gerrit] mediawiki...TemplateData[master]: Add new 'ownLine' property to control newlines around templates
Cscott has uploaded a new change for review.
https://gerrit.wikimedia.org/r/303432
Change subject: Add new 'ownLine' property to control newlines around templates
..
Add new 'ownLine' property to control newlines around templates
Bug: T135667
Change-Id: I7e364be295140fbc982e44903e03c7f151ddb5f1
---
M Specification.md
M TemplateDataBlob.php
M extension.json
M i18n/en.json
M i18n/qqq.json
M modules/ext.templateDataGenerator.data.js
M modules/ext.templateDataGenerator.ui.tdDialog.js
M resources/ext.templateData.css
8 files changed, 128 insertions(+), 2 deletions(-)
git pull ssh://gerrit.wikimedia.org:29418/mediawiki/extensions/TemplateData
refs/changes/32/303432/1
diff --git a/Specification.md b/Specification.md
index c525778..6acddf0 100644
--- a/Specification.md
+++ b/Specification.md
@@ -111,6 +111,17 @@
In the absence of the parameter being set, the system will supply `null` as a
fallback value.
+### 3.1.7 `ownLine`
+* Value: `string` of either `'prefer'` or `'avoid'`
+* Default: `"avoid"`
+
+Whether the template SHOULD occur on a line of its own.
+
+If the parameter is set to `prefer`, a newly-added template SHOULD appear
+on its own line in wikitext. If the parameter is set to `avoid`, a
newly-added template SHOULD appear inline (with no surrounding newlines) in
wikitext. For both `prefer` and `avoid`, edited templates should maintain
their existing formatting, whether that is on its own line or not.
+
+In the absence of the parameter being set, the system will supply `"avoid"` as
a fallback value.
+
### 3.2 Param
* Value: `Object`
diff --git a/TemplateDataBlob.php b/TemplateDataBlob.php
index 25d1245..ad0603c 100644
--- a/TemplateDataBlob.php
+++ b/TemplateDataBlob.php
@@ -62,6 +62,7 @@
$tdb->data->params = new stdClass();
$tdb->data->paramOrder = [];
$tdb->data->format = null;
+ $tdb->data->ownLine = 'avoid';
$tdb->data->sets = [];
$tdb->data->maps = new stdClass();
}
@@ -99,6 +100,7 @@
'sets',
'maps',
'format',
+ 'ownLine',
];
static $paramKeys = [
@@ -134,6 +136,11 @@
static $formats = [
'block' => "| _ = _\n",
'inline' => '|_=_',
+ ];
+
+ static $ownLine = [
+ 'prefer',
+ 'avoid',
];
static $typeCompatMap = [
@@ -178,6 +185,18 @@
}
} else {
$data->format = null;
+ }
+
+ // Root.ownLine
+ if ( isset( $data->ownLine ) ) {
+ if ( !in_array( $data->ownLine, $ownLine ) ) {
+ return Status::newFatal(
+ 'templatedata-invalid-ownline',
+ 'ownLine'
+ );
+ }
+ } else {
+ $data->ownLine = 'avoid';
}
// Root.params
@@ -732,11 +751,24 @@
. Html::element(
'span',
[ 'class' =>
'mw-templatedata-format' ],
- // Messages:
templatedata-modal-format-inline, templatedata-modal-format-block
+ // Messages:
templatedata-doc-format-inline, templatedata-doc-format-block
wfMessage(
'templatedata-doc-format-' . $data->format )->inLanguage( $lang )->text()
)
) :
'' )
+ . ( $data->ownLine !== null ?
+ Html::rawElement(
+ 'p',
+ [],
+ new OOUI\IconWidget( [ 'icon'
=> 'template-ownline-' . $data->ownLine ] )
+ . Html::element(
+ 'span',
+ [ 'class' =>
'mw-templatedata-ownline' ],
+ // Messages:
templatedata-doc-ownline-avoid, templatedata-doc-ownline-prefer
+ wfMessage(
'templatedata-doc-ownline-' . $data->ownLine )->inLanguage(
[MediaWiki-commits] [Gerrit] mediawiki/core[master]: Balancer: pass options object to flatten instead of individu...
Cscott has uploaded a new change for review.
https://gerrit.wikimedia.org/r/302981
Change subject: Balancer: pass options object to flatten instead of individual
booleans
..
Balancer: pass options object to flatten instead of individual booleans
This refactoring makes it easier to add additional options later without
having to pass them manually through the call chain.
Change-Id: I46814f17d1b338b971ab57f63c2ec75d4a6b45d5
---
M includes/tidy/Balancer.php
1 file changed, 14 insertions(+), 13 deletions(-)
git pull ssh://gerrit.wikimedia.org:29418/mediawiki/core
refs/changes/81/302981/1
diff --git a/includes/tidy/Balancer.php b/includes/tidy/Balancer.php
index b2d6ba1..b548deb 100644
--- a/includes/tidy/Balancer.php
+++ b/includes/tidy/Balancer.php
@@ -466,18 +466,19 @@
*
* @see __toString()
*/
- public function flatten( $tidyCompat = false ) {
+ public function flatten( $opts ) {
Assert::parameter( $this->parent !== null, '$this', 'must be a
child' );
Assert::parameter( $this->parent !== 'flat', '$this', 'already
flat' );
$idx = array_search( $this, $this->parent->children, true );
Assert::parameter(
$idx !== false, '$this', 'must be a child of its parent'
);
+ $tidyCompat = $opts->tidyCompat;
if ( $tidyCompat ) {
$blank = true;
foreach ( $this->children as $elt ) {
if ( !is_string( $elt ) ) {
- $elt = $elt->flatten( $tidyCompat );
+ $elt = $elt->flatten( $opts );
}
if ( $blank && preg_match( '/[^\t\n\f\r ]/',
$elt ) ) {
$blank = false;
@@ -661,9 +662,9 @@
*/
public $fosterParentMode = false;
/**
-* Tidy compatibility mode, determines behavior of body/blockquote
+* Configuration options governing flattening.
*/
- public $tidyCompat = false;
+ public $opts;
/**
* Reference to the current element
*/
@@ -673,13 +674,14 @@
* Create a new BalanceStack with a single BalanceElement on it,
* representing the root node.
*/
- public function __construct() {
+ public function __construct( $opts ) {
# always a root element on the stack
array_push(
$this->elements,
new BalanceElement( BalanceSets::HTML_NAMESPACE,
'html', [] )
);
$this->currentNode = $this->elements[0];
+ $this->opts = $opts;
}
/**
@@ -692,7 +694,7 @@
$out = '';
foreach ( $this->elements[0]->children as $elt ) {
$out .= is_string( $elt ) ? $elt :
- $elt->flatten( $this->tidyCompat );
+ $elt->flatten( $this->opts );
}
return $out;
}
@@ -719,7 +721,7 @@
) {
$this->fosterParent( $value );
} elseif (
- $this->tidyCompat && !$isComment &&
+ $this->opts->tidyCompat && !$isComment &&
$this->currentNode->isA( BalanceSets::$tidyPWrapSet )
) {
$this->insertHTMLELement( 'mw:p-wrap', [] );
@@ -970,7 +972,7 @@
$this->currentNode = null;
}
if ( !$elt->isHtmlNamed( 'mw:p-wrap' ) ) {
- $elt->flatten( $this->tidyCompat );
+ $elt->flatten( $this->opts );
}
}
@@ -1045,7 +1047,7 @@
// otherwise, it will eventually serialize when the
parent
// is serialized, we just hold onto the memory for its
// tree of objects a little longer.
- $elt->flatten( $this->tidyCompat );
+ $elt->flatten( $this->opts );
}
Assert::postcondition(
array_search( $elt, $this->elements, true ) === false,
@@ -1095,7 +1097,7 @@
$parent = $this->elements[0]; // the `html` element.
}
- if ( $this->tidyCompat ) {
+ if ( $this->opts->tidyCompat ) {
if ( is_string( $elt ) ) {
// We're fostering text: do we need a p-wrapper?
if ( $parent->isA( BalanceSets::$tidyPWrapSet )
) {
@@ -1789,8 +1791,8 @@
private $afe;
private $stack;
private $strict;
[MediaWiki-commits] [Gerrit] mediawiki...parsoid[master]: Add --check option to roundtrip-test.js.
Cscott has uploaded a new change for review.
https://gerrit.wikimedia.org/r/301881
Change subject: Add --check option to roundtrip-test.js.
..
Add --check option to roundtrip-test.js.
This exits with a non-zero exit code if a problem is found to ensure that
selser problems are caught by jenkins.
Change-Id: I8c5f6f0cfa2ffd008daded8ac802c81025daa547
---
M bin/roundtrip-test.js
M package.json
2 files changed, 16 insertions(+), 5 deletions(-)
git pull ssh://gerrit.wikimedia.org:29418/mediawiki/services/parsoid
refs/changes/81/301881/1
diff --git a/bin/roundtrip-test.js b/bin/roundtrip-test.js
index 57d9685..4dbb068 100755
--- a/bin/roundtrip-test.js
+++ b/bin/roundtrip-test.js
@@ -581,7 +581,12 @@
var profile = { time: { total: 0, start: 0 }, size: {} };
var closeFormatter = function(e, results) {
- return formatter(e, prefix, title, results, profile);
+ var failure = e || results.some(function(r) {
+ return r.selser;
+ });
+ var output = formatter(e, prefix, title, results, profile);
+ return options.check ? { output: output, exitCode: failure ? 1
: 0 } :
+ output;
};
var uri = options.parsoidURL;
@@ -717,6 +722,12 @@
boolean: false,
default: defaultContentVersion,
},
+ check: {
+ description: 'Exit with non-zero exit code if
differences found using selser',
+ boolean: true,
+ default: false,
+ alias: 'c',
+ },
};
(function() {
@@ -748,9 +759,9 @@
}).then(function() {
var formatter = Util.booleanOption(argv.xml) ?
xmlFormat : plainFormat;
return runTests(title, argv, formatter);
- }).then(function(output) {
- console.log(output);
- process.exit(0);
+ }).then(function(r) {
+ console.log(argv.check ? r.output : r);
+ process.exit(argv.check ? r.exitCode : 0);
}).done();
}());
} else if (typeof module === 'object') {
diff --git a/package.json b/package.json
index bb45879..7a4ca94 100644
--- a/package.json
+++ b/package.json
@@ -60,7 +60,7 @@
"dump-tokenizer": "node lib/wt2html/tokenizer.js",
"mocha": "mocha --opts tests/mocha/mocha.opts tests/mocha",
"parserTests": "node bin/parserTests.js --wt2html --wt2wt --html2wt
--html2html --selser --no-color --quiet --blacklist",
-"roundtrip": "node bin/roundtrip-test.js 'Barack Obama' && node
bin/roundtrip-test.js --prefix frwiki Chope && node bin/roundtrip-test.js --xml
Parkour",
+"roundtrip": "node bin/roundtrip-test.js -c 'Barack Obama' && node
bin/roundtrip-test.js -c --prefix frwiki Chope && node bin/roundtrip-test.js -c
--xml Parkour",
"toolcheck": "bin/toolcheck.sh",
"test": "npm run nsp && npm run lint && npm run parserTests && npm run
mocha",
"cover-mocha": "istanbul cover _mocha --dir ./coverage/mocha -- --opts
tests/mocha/mocha.opts tests/mocha",
--
To view, visit https://gerrit.wikimedia.org/r/301881
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: I8c5f6f0cfa2ffd008daded8ac802c81025daa547
Gerrit-PatchSet: 1
Gerrit-Project: mediawiki/services/parsoid
Gerrit-Branch: master
Gerrit-Owner: Cscott
___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] integration/config[master]: Move Parsoid roundtrip and toolcheck tests into npm scripts ...
Cscott has uploaded a new change for review. https://gerrit.wikimedia.org/r/301499 Change subject: Move Parsoid roundtrip and toolcheck tests into npm scripts in Parsoid repo .. Move Parsoid roundtrip and toolcheck tests into npm scripts in Parsoid repo This way we don't need to edit the jenkins configuration to tweak our tests and we can more easily reproduce jenkins test failures. Bug: T141481 Change-Id: I0dc262490373430d9dcd7af007ed52dce19d064b --- M jjb/parsoidsvc.yaml 1 file changed, 4 insertions(+), 9 deletions(-) git pull ssh://gerrit.wikimedia.org:29418/integration/config refs/changes/99/301499/1 diff --git a/jjb/parsoidsvc.yaml b/jjb/parsoidsvc.yaml index 142529e..37c2cf2 100644 --- a/jjb/parsoidsvc.yaml +++ b/jjb/parsoidsvc.yaml @@ -64,11 +64,8 @@ # If any of these commands fail, something is wrong set -e . npm-oid.env -cd "$NPM_SET_PATH"/bin -echo "Foo" | node parse.js --wt2html -echo "Foo" | node parse.js --wt2wt -echo "Foo" | node parse.js --html2wt -echo "Foo" | node parse.js --html2html +cd "$NPM_SET_PATH" +npm run toolcheck publishers: - castor-save @@ -94,10 +91,8 @@ # If any of these commands fail, something is wrong set -e . npm-oid.env -cd "$NPM_SET_PATH"/bin -node roundtrip-test.js "Barack Obama" -node roundtrip-test.js --prefix frwiki "Chope" -node roundtrip-test.js --xml "Parkour" +cd "$NPM_SET_PATH" +npm run roundtrip publishers: - castor-save -- To view, visit https://gerrit.wikimedia.org/r/301499 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I0dc262490373430d9dcd7af007ed52dce19d064b Gerrit-PatchSet: 1 Gerrit-Project: integration/config Gerrit-Branch: master Gerrit-Owner: Cscott ___ MediaWiki-commits mailing list MediaWiki-commits@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] mediawiki...parsoid[master]: Make `npm run roundtrip` match the current jenkins roundtrip...
Cscott has uploaded a new change for review. https://gerrit.wikimedia.org/r/301498 Change subject: Make `npm run roundtrip` match the current jenkins roundtrip test. .. Make `npm run roundtrip` match the current jenkins roundtrip test. Change-Id: I9dba7da273dee7fd54ac850d16cfde2d42ccebe0 --- M package.json 1 file changed, 1 insertion(+), 1 deletion(-) git pull ssh://gerrit.wikimedia.org:29418/mediawiki/services/parsoid refs/changes/98/301498/1 diff --git a/package.json b/package.json index 5fae47f..bb45879 100644 --- a/package.json +++ b/package.json @@ -60,7 +60,7 @@ "dump-tokenizer": "node lib/wt2html/tokenizer.js", "mocha": "mocha --opts tests/mocha/mocha.opts tests/mocha", "parserTests": "node bin/parserTests.js --wt2html --wt2wt --html2wt --html2html --selser --no-color --quiet --blacklist", -"roundtrip": "node bin/roundtrip-test.js 'Barack Obama' && node bin/roundtrip-test.js 'Parkour'", +"roundtrip": "node bin/roundtrip-test.js 'Barack Obama' && node bin/roundtrip-test.js --prefix frwiki Chope && node bin/roundtrip-test.js --xml Parkour", "toolcheck": "bin/toolcheck.sh", "test": "npm run nsp && npm run lint && npm run parserTests && npm run mocha", "cover-mocha": "istanbul cover _mocha --dir ./coverage/mocha -- --opts tests/mocha/mocha.opts tests/mocha", -- To view, visit https://gerrit.wikimedia.org/r/301498 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I9dba7da273dee7fd54ac850d16cfde2d42ccebe0 Gerrit-PatchSet: 1 Gerrit-Project: mediawiki/services/parsoid Gerrit-Branch: master Gerrit-Owner: Cscott ___ MediaWiki-commits mailing list MediaWiki-commits@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] mediawiki...parsoid[master]: Add --pboutfile option to parse.js.
Cscott has uploaded a new change for review.
https://gerrit.wikimedia.org/r/301497
Change subject: Add --pboutfile option to parse.js.
..
Add --pboutfile option to parse.js.
This lets us more easily test the --pbinfile option!
Change-Id: Ib769db76882e6164cb3ff36400724bd83fe1
---
M bin/parse.js
1 file changed, 10 insertions(+), 1 deletion(-)
git pull ssh://gerrit.wikimedia.org:29418/mediawiki/services/parsoid
refs/changes/97/301497/1
diff --git a/bin/parse.js b/bin/parse.js
index d02a53a..6242a50 100755
--- a/bin/parse.js
+++ b/bin/parse.js
@@ -123,6 +123,11 @@
'boolean': false,
'default': '',
},
+ 'pboutfile': {
+ description: 'Output pagebundle JSON to file',
+ 'boolean': false,
+ 'default': false,
+ },
'lint': {
description: 'Parse with linter enabled',
'boolean': true,
@@ -184,6 +189,10 @@
}
if (argv.wt2html || argv.html2html) {
var out;
+ if (argv.pboutfile) {
+ var pb = DU.extractPageBundle(doc);
+ fs.writeFileSync(argv.pboutfile,
JSON.stringify(pb), 'utf8');
+ }
if (argv.normalize) {
out = DU.normalizeOut(doc.body, (argv.normalize
=== 'parsoid'));
} else if (argv.document) {
@@ -217,7 +226,7 @@
env.scrubWikitext = argv.scrubWikitext;
// Sets ids on nodes and stores data-* attributes in a JSON blob
- env.pageBundle = argv.pagebundle;
+ env.pageBundle = argv.pagebundle || argv.pboutfile;
// The content version to output
if (argv.contentVersion) {
--
To view, visit https://gerrit.wikimedia.org/r/301497
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: Ib769db76882e6164cb3ff36400724bd83fe1
Gerrit-PatchSet: 1
Gerrit-Project: mediawiki/services/parsoid
Gerrit-Branch: master
Gerrit-Owner: Cscott
___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] mediawiki...parsoid[master]: Move jenkins "tool-check" script into repository; add --pbou...
Cscott has uploaded a new change for review.
https://gerrit.wikimedia.org/r/301496
Change subject: Move jenkins "tool-check" script into repository; add
--pboutfile to parse.
..
Move jenkins "tool-check" script into repository; add --pboutfile to parse.
Bug: T141481
Change-Id: I38b4b4b8e99ab0861063efc0dc4195dc8d7eb99d
---
M bin/parse.js
A bin/toolcheck.sh
M package.json
3 files changed, 47 insertions(+), 1 deletion(-)
git pull ssh://gerrit.wikimedia.org:29418/mediawiki/services/parsoid
refs/changes/96/301496/1
diff --git a/bin/parse.js b/bin/parse.js
index d02a53a..6242a50 100755
--- a/bin/parse.js
+++ b/bin/parse.js
@@ -123,6 +123,11 @@
'boolean': false,
'default': '',
},
+ 'pboutfile': {
+ description: 'Output pagebundle JSON to file',
+ 'boolean': false,
+ 'default': false,
+ },
'lint': {
description: 'Parse with linter enabled',
'boolean': true,
@@ -184,6 +189,10 @@
}
if (argv.wt2html || argv.html2html) {
var out;
+ if (argv.pboutfile) {
+ var pb = DU.extractPageBundle(doc);
+ fs.writeFileSync(argv.pboutfile,
JSON.stringify(pb), 'utf8');
+ }
if (argv.normalize) {
out = DU.normalizeOut(doc.body, (argv.normalize
=== 'parsoid'));
} else if (argv.document) {
@@ -217,7 +226,7 @@
env.scrubWikitext = argv.scrubWikitext;
// Sets ids on nodes and stores data-* attributes in a JSON blob
- env.pageBundle = argv.pagebundle;
+ env.pageBundle = argv.pagebundle || argv.pboutfile;
// The content version to output
if (argv.contentVersion) {
diff --git a/bin/toolcheck.sh b/bin/toolcheck.sh
new file mode 100755
index 000..ba59182
--- /dev/null
+++ b/bin/toolcheck.sh
@@ -0,0 +1,36 @@
+#!/bin/bash
+
+# Perform a basic crash test.
+# This used to be the
+# `parsoidsvc-{repository}-parse-tool-check-trusty`
+# test on jenkins. (T141481)
+
+BIN=$(dirname $0)
+
+# If any of these commands fail, something is wrong!
+set -ev
+
+echo "Foo" | node $BIN/parse.js --wt2html
+echo "Foo" | node $BIN/parse.js --wt2html --pagebundle
+echo "Foo" | node $BIN/parse.js --wt2wt
+echo "Foo" | node $BIN/parse.js --html2wt
+echo "Foo" | node $BIN/parse.js --html2html
+
+# Check --selser too!
+TMPWT=$(tempfile -s wt)
+TMPORIG=$(tempfile -s orig)
+TMPEDIT=$(tempfile -s edit)
+TMPPB=$(tempfile -s pb)
+# inline data-parsoid
+echo "fooboo" | tee $TMPWT | node $BIN/parse.js | tee $TMPORIG |
+sed -e "s/foo/bar/g" > $TMPEDIT
+node $BIN/parse.js --selser --oldtextfile $TMPWT --oldhtmlfile $TMPORIG <
$TMPEDIT
+
+# data-parsoid in separate files
+node $BIN/parse.js --pboutfile $TMPPB --contentVersion 2.0.0 < $TMPWT |
+tee $TMPORIG | sed -e "s/foo/bar/g" > $TMPEDIT
+node $BIN/parse.js --pbinfile $TMPPB --selser \
+--oldtextfile $TMPWT --oldhtmlfile $TMPORIG < $TMPEDIT
+
+# clean up
+/bin/rm $TMPWT $TMPORIG $TMPEDIT $TMPPB
diff --git a/package.json b/package.json
index 871c3c8..5fae47f 100644
--- a/package.json
+++ b/package.json
@@ -61,6 +61,7 @@
"mocha": "mocha --opts tests/mocha/mocha.opts tests/mocha",
"parserTests": "node bin/parserTests.js --wt2html --wt2wt --html2wt
--html2html --selser --no-color --quiet --blacklist",
"roundtrip": "node bin/roundtrip-test.js 'Barack Obama' && node
bin/roundtrip-test.js 'Parkour'",
+"toolcheck": "bin/toolcheck.sh",
"test": "npm run nsp && npm run lint && npm run parserTests && npm run
mocha",
"cover-mocha": "istanbul cover _mocha --dir ./coverage/mocha -- --opts
tests/mocha/mocha.opts tests/mocha",
"cover-parserTests": "istanbul cover bin/parserTests.js --dir
./coverage/parserTests -- --wt2html --wt2wt --html2wt --html2html --selser
--no-color --quiet --blacklist",
--
To view, visit https://gerrit.wikimedia.org/r/301496
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: I38b4b4b8e99ab0861063efc0dc4195dc8d7eb99d
Gerrit-PatchSet: 1
Gerrit-Project: mediawiki/services/parsoid
Gerrit-Branch: master
Gerrit-Owner: Cscott
___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] mediawiki...parsoid[master]: Better logging for empty JSON replies.
Cscott has uploaded a new change for review.
https://gerrit.wikimedia.org/r/301490
Change subject: Better logging for empty JSON replies.
..
Better logging for empty JSON replies.
Suggested by T141391.
Change-Id: I3f9d6f7c93b60ea51c6a277d42d58bd5fc3e379f
---
M lib/mw/ApiRequest.js
1 file changed, 9 insertions(+), 1 deletion(-)
git pull ssh://gerrit.wikimedia.org:29418/mediawiki/services/parsoid
refs/changes/90/301490/1
diff --git a/lib/mw/ApiRequest.js b/lib/mw/ApiRequest.js
index 3faf2de..3eb9c53 100644
--- a/lib/mw/ApiRequest.js
+++ b/lib/mw/ApiRequest.js
@@ -454,8 +454,16 @@
}
data = JSON.parse(body);
} catch (e) {
- error = new ParserError('Failed to parse the JSON response for
' +
+ if (!body) {
+ // This is usually due to a fatal error on the PHP
side, although
+ // it would be nice (!) if PHP would return a non-200
error code
+ // for this!
+ error = new ParserError('Empty JSON response returned
for ' +
this.reqType);
+ } else {
+ error = new ParserError('Failed to parse the JSON
response for ' +
+ this.reqType);
+ }
}
this._handleJSON(error, data);
};
--
To view, visit https://gerrit.wikimedia.org/r/301490
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: I3f9d6f7c93b60ea51c6a277d42d58bd5fc3e379f
Gerrit-PatchSet: 1
Gerrit-Project: mediawiki/services/parsoid
Gerrit-Branch: master
Gerrit-Owner: Cscott
___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] Remove rest.wikimedia.org, the server is being shut down Sep... - change (mediawiki...bundler)
Cscott has uploaded a new change for review.
https://gerrit.wikimedia.org/r/300570
Change subject: Remove rest.wikimedia.org, the server is being shut down Sept 1.
..
Remove rest.wikimedia.org, the server is being shut down Sept 1.
Also remove a lingering reference to parsoid.eqiad, which has already
been shut down.
Change-Id: I7554bb335c436d110ae1a34095709b8425d89b42
---
M bin/mw-ocg-bundler
M lib/metabook.js
2 files changed, 2 insertions(+), 3 deletions(-)
git pull
ssh://gerrit.wikimedia.org:29418/mediawiki/extensions/Collection/OfflineContentGenerator/bundler
refs/changes/70/300570/1
diff --git a/bin/mw-ocg-bundler b/bin/mw-ocg-bundler
index dc84661..066127c 100755
--- a/bin/mw-ocg-bundler
+++ b/bin/mw-ocg-bundler
@@ -28,7 +28,7 @@
.option('-a, --parsoid-api ',
'Parsoid API root')
.option('--restbase-api ',
- 'RESTbase API root
[http://rest.wikimedia.org//v1/]')
+ 'RESTbase API root [eg
https://en.wikipedia.org/api/rest_v1/]')
.option('--php-api ',
'PHP API, required for non-standard prefix', null)
.option('--api-version ',
diff --git a/lib/metabook.js b/lib/metabook.js
index d70b0f1..37367c5 100644
--- a/lib/metabook.js
+++ b/lib/metabook.js
@@ -158,8 +158,7 @@
format: 'nuwiki',
restbase1: options.restbaseApi ||
url.resolve(s.url, '/api/rest_v1/'),
- parsoid: options.parsoidApi ||
- 'http://parsoid-lb.eqiad.wikimedia.org/',
+ parsoid: options.parsoidApi,
prefix: prefix,
domain: domain,
titleurl: url.resolve(s.url, '/wiki/') + '$1', //
(Temporary)
--
To view, visit https://gerrit.wikimedia.org/r/300570
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: I7554bb335c436d110ae1a34095709b8425d89b42
Gerrit-PatchSet: 1
Gerrit-Project: mediawiki/extensions/Collection/OfflineContentGenerator/bundler
Gerrit-Branch: master
Gerrit-Owner: Cscott
___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] Don't JSON-decode the response of html/to/wikitext/ request ... - change (mediawiki...parsoid)
Cscott has uploaded a new change for review.
https://gerrit.wikimedia.org/r/300441
Change subject: Don't JSON-decode the response of html/to/wikitext/ request in
roundtrip-test.
..
Don't JSON-decode the response of html/to/wikitext/ request in roundtrip-test.
The result of this API call is wikitext, not a JSON blob. It currently
works only because the http library silently swallows the decode exception.
Change-Id: I5ef16832c5479b519d4c74943c2ca5af385371e1
---
M bin/roundtrip-test.js
1 file changed, 7 insertions(+), 1 deletion(-)
git pull ssh://gerrit.wikimedia.org:29418/mediawiki/services/parsoid
refs/changes/41/300441/1
diff --git a/bin/roundtrip-test.js b/bin/roundtrip-test.js
index dd2a627..2dd60b9 100755
--- a/bin/roundtrip-test.js
+++ b/bin/roundtrip-test.js
@@ -473,7 +473,6 @@
function parsoidPost(profile, options) {
var httpOptions = {
method: 'POST',
- json: true,
body: options.data,
};
@@ -484,11 +483,18 @@
uri += '/' + options.oldid;
}
httpOptions.body.scrub_wikitext = true;
+ // We want to encode the request but *not* decode the response.
+ httpOptions.body = JSON.stringify(httpOptions.body);
+ httpOptions.headers = {
+ 'Content-Type': 'application/json',
+ };
} else { // wt2html
uri += 'wikitext/to/pagebundle/' + options.title;
httpOptions.headers = {
Accept: apiUtils.pagebundleContentType(null,
options.contentVersion),
};
+ // setting json here encodes the request *and* decodes the
response.
+ httpOptions.json = true;
}
httpOptions.uri = uri;
--
To view, visit https://gerrit.wikimedia.org/r/300441
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: I5ef16832c5479b519d4c74943c2ca5af385371e1
Gerrit-PatchSet: 1
Gerrit-Project: mediawiki/services/parsoid
Gerrit-Branch: master
Gerrit-Owner: Cscott
___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] Add additional balancer tests (those starting with `
Cscott has uploaded a new change for review.
https://gerrit.wikimedia.org/r/299669
Change subject: Add additional balancer tests (those starting with ``)
..
Add additional balancer tests (those starting with ``)
Change-Id: Ie854cf99f7e72bcca1bb8565ace558a43dcb6379
---
M tests/phpunit/includes/tidy/BalancerTest.php
1 file changed, 25 insertions(+), 12 deletions(-)
git pull ssh://gerrit.wikimedia.org:29418/mediawiki/core
refs/changes/69/299669/1
diff --git a/tests/phpunit/includes/tidy/BalancerTest.php
b/tests/phpunit/includes/tidy/BalancerTest.php
index f2e41bd..740ddb9 100644
--- a/tests/phpunit/includes/tidy/BalancerTest.php
+++ b/tests/phpunit/includes/tidy/BalancerTest.php
@@ -48,15 +48,16 @@
// for providers, and filter out HTML constructs which
// the balancer doesn't support.
$tests = [];
- $start = '';
- $end = '';
+ $okre = "~ \A
+ (?i:)?
+
+ .*
+
+ \z ~xs";
foreach ( $json as $filename => $cases ) {
foreach ( $cases as $case ) {
$html = $case['document']['html'];
- if (
- substr( $html, 0, strlen( $start ) )
!== $start ||
- substr( $html, -strlen( $end ) ) !==
$end
- ) {
+ if ( !preg_match( $okre, $html ) ) {
// Skip tests which involve stuff in
the or
// weird doctypes.
continue;
@@ -70,6 +71,8 @@
$html = $case['document']['noQuirksBodyHtml'];
// Normalize case of SVG attributes.
$html = str_replace( 'foreignObject',
'foreignobject', $html );
+ // Normalize case of MathML attributes.
+ $html = str_replace( 'definitionURL',
'definitionurl', $html );
if (
isset(
$case['document']['props']['comment'] ) &&
@@ -83,11 +86,17 @@
// Skip tests involving
quoting.
continue;
}
- if ( stripos( $case['data'], '' ) === false
+ ) {
+ // Skip tests involving unusual
doctypes.
continue;
}
- if ( preg_match(
',|)$:', $case['data'] ) ||
- preg_match( '::', $case['data'] )
+ preg_match( '::', $case['data'] )
||
+ preg_match( ':|:',
$case['data'] )
) {
// Skip tests with funny tag or
attribute names,
// which are really tests of the HTML
tokenizer, not
@@ -127,7 +137,7 @@
continue;
}
if (
- stripos( $case['data'], 'encoding="
text/html "' ) !== false
+ preg_match( ':encoding=" text/html
"|type=" hidden":', $case['data'] )
) {
// The Sanitizer normalizes whitespace
in attribute
// values, which makes this test case
invalid.
@@ -137,9 +147,12 @@
// Skip tests with ASCII null, etc.
continue;
}
+ $data = preg_replace(
+ '~~i', '', $case['data']
+ );
$tests[] = [
$filename, # use better description?
- $case['data'],
+ $data,
$html
];
}
--
To view, visit https://gerrit.wikimedia.org/r/299669
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: Ie854cf99f7e72bcca1bb8565ace558a43dcb6379
Gerrit-PatchSet: 1
Gerrit-Project: mediawiki/core
Gerrit-Branch: master
Gerrit-Owner: Cscott
___
MediaWiki-commi
[MediaWiki-commits] [Gerrit] Support tags in Balancer. - change (mediawiki/core)
Cscott has uploaded a new change for review.
https://gerrit.wikimedia.org/r/299668
Change subject: Support tags in Balancer.
..
Support tags in Balancer.
Change-Id: I63c2fd1c343362e49cf3b5a258fc98489744ad68
---
M includes/tidy/Balancer.php
M tests/phpunit/includes/tidy/BalancerTest.php
2 files changed, 72 insertions(+), 11 deletions(-)
git pull ssh://gerrit.wikimedia.org:29418/mediawiki/core
refs/changes/68/299668/1
diff --git a/includes/tidy/Balancer.php b/includes/tidy/Balancer.php
index 37807ba..4c348b1 100644
--- a/includes/tidy/Balancer.php
+++ b/includes/tidy/Balancer.php
@@ -75,7 +75,7 @@
self::HTML_NAMESPACE => [
'html' => true, 'head' => true, 'body' => true,
'frameset' => true,
'frame' => true,
- 'plaintext' => true, 'isindex' => true, 'textarea' =>
true,
+ 'plaintext' => true, 'isindex' => true,
'xmp' => true, 'iframe' => true, 'noembed' => true,
'noscript' => true, 'script' => true,
'title' => true
@@ -89,6 +89,12 @@
'embed' => true, 'frame' => true, 'hr' => true, 'img'
=> true,
'input' => true, 'keygen' => true, 'link' => true,
'meta' => true,
'param' => true, 'source' => true, 'track' => true,
'wbr' => true
+ ]
+ ];
+
+ public static $extraLinefeedSet = [
+ self::HTML_NAMESPACE => [
+ 'pre' => true, 'textarea' => true, 'listing' => true,
]
];
@@ -513,11 +519,21 @@
}
if ( !$this->isA( BalanceSets::$emptyElementSet ) ) {
$out = "<{$this->localName}{$encAttribs}>";
+ $len = strlen( $out );
// flatten children
foreach ( $this->children as $elt ) {
$out .= "{$elt}";
}
$out .= "localName}>";
+ if (
+ $this->isA( BalanceSets::$extraLinefeedSet ) &&
+ $out[$len] === "\n"
+ ) {
+ // Double the linefeed after
pre/listing/textarea
+ // according to the HTML5 fragment
serialization algorithm.
+ $out = substr( $out, 0, $len + 1 ) .
+ substr( $out, $len );
+ }
} else {
$out = "<{$this->localName}{$encAttribs} />";
Assert::invariant(
@@ -1740,18 +1756,19 @@
* - The document is never in "quirks mode".
* - All occurrences of < and > have been entity escaped, so we
* can parse tags by simply splitting on those two characters.
+ * (This also simplifies the handling of < inside .)
* The character < must not appear inside comments.
* Similarly, all attributes have been "cleaned" and are double-quoted
* and escaped.
* - All null characters are assumed to have been removed.
- * - We don't alter linefeeds after /.
* - The following elements are disallowed: , , , ,
- * , , , , , ,
+ * , , , , ,
* , ,
[MediaWiki-commits] [Gerrit] Minor bug fixes to Balancer. - change (mediawiki/core)
Cscott has uploaded a new change for review.
https://gerrit.wikimedia.org/r/299667
Change subject: Minor bug fixes to Balancer.
..
Minor bug fixes to Balancer.
This is a follow-up to the refactor done in
5726c9ceb0644af360d37b86351b97ddfcbee20c which prevents a crash when
the first entry in the stack happens to be a BalanceMarker (and thus
doesn't have a `$localName` property). It also fixes an unrelated
issue where unpaired close-heading tags (like ``) get entity-escaped
instead of ignored.
Test cases exposing these bugs are added in
Ie854cf99f7e72bcca1bb8565ace558a43dcb6379.
Change-Id: Ia9a1d435be1be10512071f5ff626b68742863483
---
M includes/tidy/Balancer.php
1 file changed, 4 insertions(+), 2 deletions(-)
git pull ssh://gerrit.wikimedia.org:29418/mediawiki/core
refs/changes/67/299667/1
diff --git a/includes/tidy/Balancer.php b/includes/tidy/Balancer.php
index 9e96b14..4bcaf1a 100644
--- a/includes/tidy/Balancer.php
+++ b/includes/tidy/Balancer.php
@@ -1613,9 +1613,11 @@
// Loop backward through the list until we find a marker or an
// open element
+ $foundit = false;
while ( $entry->prevAFE ) {
$entry = $entry->prevAFE;
if ( $entry instanceof BalanceMarker ||
$stack->indexOf( $entry ) >= 0 ) {
+ $foundit = true;
break;
}
}
@@ -1624,7 +1626,7 @@
// the first element if we didn't find a marker or open
element),
// recreating formatting elements and pushing them back onto
the list
// of open elements.
- if ( $entry->prevAFE ) {
+ if ( $foundit ) {
$entry = $entry->nextAFE;
}
do {
@@ -2656,7 +2658,7 @@
case 'h5':
case 'h6':
if ( !$this->stack->inScope(
BalanceSets::$headingSet ) ) {
- return;
+ return true; # ignore
}
$this->stack->generateImpliedEndTags();
$this->stack->popTag( BalanceSets::$headingSet
);
--
To view, visit https://gerrit.wikimedia.org/r/299667
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: Ia9a1d435be1be10512071f5ff626b68742863483
Gerrit-PatchSet: 1
Gerrit-Project: mediawiki/core
Gerrit-Branch: master
Gerrit-Owner: Cscott
___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] Support tokenizing simple HTML comments in the Balancer. - change (mediawiki/core)
Cscott has uploaded a new change for review.
https://gerrit.wikimedia.org/r/299178
Change subject: Support tokenizing simple HTML comments in the Balancer.
..
Support tokenizing simple HTML comments in the Balancer.
Change-Id: Ib780595b13b7145e99867d16e3c225e6b2b91884
---
M includes/tidy/Balancer.php
M tests/phpunit/includes/tidy/BalancerTest.php
2 files changed, 96 insertions(+), 8 deletions(-)
git pull ssh://gerrit.wikimedia.org:29418/mediawiki/core
refs/changes/78/299178/1
diff --git a/includes/tidy/Balancer.php b/includes/tidy/Balancer.php
index 47d1ae8..98299d1 100644
--- a/includes/tidy/Balancer.php
+++ b/includes/tidy/Balancer.php
@@ -43,6 +43,11 @@
# as soon as possible (usually as soon as the tag is closed) to reduce
# its memory footprint.
+# We've been gradually lifting some of these restrictions to handle
+# non-sanitized output generated by extensions, but we shortcut the tokenizer
+# for speed (primarily by splitting on `<`) and so rely on syntactic
+# well-formedness.
+
# On the other hand, I've been pretty careful to note with comments in the
# code the places where this implementation omits features of the spec or
# depends on the MediaWiki Sanitizer. Perhaps in the future we'll want to
@@ -677,18 +682,28 @@
}
/**
+* Insert a comment at the appropriate place for inserting a node.
+* @param string $value Content of the comment.
+* @see
https://html.spec.whatwg.org/multipage/syntax.html#insert-a-comment
+*/
+ public function insertComment( $value ) {
+ // Just another type of text node, except for tidy p-wrapping.
+ return $this->insertText( '', true );
+ }
+
+ /**
* Insert text at the appropriate place for inserting a node.
* @param string $value
* @see
https://html.spec.whatwg.org/multipage/syntax.html#appropriate-place-for-inserting-a-node
*/
- public function insertText( $value ) {
+ public function insertText( $value, $isComment = false ) {
if (
$this->fosterParentMode &&
$this->currentNode->isA(
BalanceSets::$tableSectionRowSet )
) {
$this->fosterParent( $value );
} elseif (
- $this->tidyCompat &&
+ $this->tidyCompat && !$isComment &&
$this->currentNode->isA( BalanceSets::$tidyPWrapSet )
) {
$this->insertHTMLELement( 'mw:p-wrap', [] );
@@ -1723,9 +1738,10 @@
* - The document is never in "quirks mode".
* - All occurrences of < and > have been entity escaped, so we
* can parse tags by simply splitting on those two characters.
+ * The character < must not appear inside comments.
* Similarly, all attributes have been "cleaned" and are double-quoted
* and escaped.
- * - All comments and null characters are assumed to have been removed.
+ * - All null characters are assumed to have been removed.
* - We don't alter linefeeds after /.
* - The following elements are disallowed: , , , ,
* , , , , , ,
@@ -1755,12 +1771,44 @@
private $stack;
private $strict;
private $tidyCompat;
+ private $allowComments;
private $textIntegrationMode = false;
private $pendingTableText;
private $originalInsertionMode;
private $fragmentContext;
private $formElementPointer;
+
+ /**
+* Valid HTML5 comments.
+* Regex borrowed from Tim Starling's "remex-html" project.
+*/
+ const VALID_COMMENT_REGEX = "~ !--
+ ( # 1. Comment match detector
+ > | -> | # Invalid short close
+ ( # 2. Comment contents
+ (?:
+ (?! --> )
+ (?! --!> )
+ (?! --! \z )
+ (?! -- \z )
+ (?! - \z )
+ .
+ )*+
+ )
+ ( # 3. Comment close
+ --> | # Normal close
+ --!> | # Comment end bang
+ ( # 4. Indicate matches
requiring EOF
+ --! | # EOF in comment end bang state
+ -- |# EOF in comment end state
+ - |# EOF in comment end dash state
+ # EOF in comment state
+ )
+ )
+
[MediaWiki-commits] [Gerrit] Support tags in Balancer. - change (mediawiki/core)
Cscott has uploaded a new change for review. https://gerrit.wikimedia.org/r/299082 Change subject: Support tags in Balancer. .. Support tags in Balancer. Change-Id: I893fc231fea71f58449ed426d64ac99fdcb31d9e --- M includes/tidy/Balancer.php M tests/phpunit/includes/tidy/BalancerTest.php 2 files changed, 63 insertions(+), 9 deletions(-) git pull ssh://gerrit.wikimedia.org:29418/mediawiki/core refs/changes/82/299082/1 diff --git a/includes/tidy/Balancer.php b/includes/tidy/Balancer.php index ce4b0ca..47d1ae8 100644 --- a/includes/tidy/Balancer.php +++ b/includes/tidy/Balancer.php @@ -69,7 +69,7 @@ public static $unsupportedSet = [ self::HTML_NAMESPACE => [ 'html' => true, 'head' => true, 'body' => true, 'frameset' => true, - 'form' => true, 'frame' => true, + 'frame' => true, 'plaintext' => true, 'isindex' => true, 'textarea' => true, 'xmp' => true, 'iframe' => true, 'noembed' => true, 'noscript' => true, 'script' => true, @@ -185,7 +185,14 @@ ] ]; - # OMITTED: formAssociatedSet, since we don't allow + // See https://html.spec.whatwg.org/multipage/forms.html#form-associated-element + public static $formAssociatedSet = [ + self::HTML_NAMESPACE => [ + 'button' => true, 'fieldset' => true, 'input' => true, + 'keygen' => true, 'object' => true, 'output' => true, + 'select' => true, 'textarea' => true, 'img' => true + ] + ]; public static $inScopeSet = [ self::HTML_NAMESPACE => [ @@ -1721,11 +1728,10 @@ * - All comments and null characters are assumed to have been removed. * - We don't alter linefeeds after /. * - The following elements are disallowed: , , , , - * , , , , , , , + * , , , , , , * , ,
[MediaWiki-commits] [Gerrit] Support tags in Balancer. - change (mediawiki/core)
Cscott has uploaded a new change for review.
https://gerrit.wikimedia.org/r/299034
Change subject: Support tags in Balancer.
..
Support tags in Balancer.
Change-Id: Ibc346624a9d035c98a29132a541e7ed6d82b364e
---
M includes/tidy/Balancer.php
M tests/phpunit/includes/tidy/BalancerTest.php
2 files changed, 119 insertions(+), 13 deletions(-)
git pull ssh://gerrit.wikimedia.org:29418/mediawiki/core
refs/changes/34/299034/1
diff --git a/includes/tidy/Balancer.php b/includes/tidy/Balancer.php
index 9e96b14..6307424 100644
--- a/includes/tidy/Balancer.php
+++ b/includes/tidy/Balancer.php
@@ -72,7 +72,7 @@
'form' => true, 'frame' => true,
'plaintext' => true, 'isindex' => true, 'textarea' =>
true,
'xmp' => true, 'iframe' => true, 'noembed' => true,
- 'noscript' => true, 'select' => true, 'script' => true,
+ 'noscript' => true, 'script' => true,
'title' => true
]
];
@@ -225,6 +225,12 @@
public static $inTableScopeSet = [
self::HTML_NAMESPACE => [
'html' => true, 'table' => true, 'template' => true
+ ]
+ ];
+
+ public static $inInvertedSelectScopeSet = [
+ self::HTML_NAMESPACE => [
+ 'option' => true, 'optgroup' => true
]
];
@@ -782,6 +788,26 @@
*/
public function inTableScope( $tag ) {
return $this->inSpecificScope( $tag,
BalanceSets::$inTableScopeSet );
+ }
+
+ /**
+* Determine if the stack has $tag in select scope.
+* @param BalanceElement|array|string $tag
+* @return bool
+* @see
https://html.spec.whatwg.org/multipage/syntax.html#has-an-element-in-select-scope
+*/
+ public function inSelectScope( $tag ) {
+ // Can't use inSpecificScope to implement this, since it
involves
+ // *inverting* a set of tags. Implement manually.
+ foreach ( $this as $elt ) {
+ if ( $elt->isA( $tag ) ) {
+ return true;
+ }
+ if ( !$elt->isA( BalanceSets::$inInvertedSelectScopeSet
) ) {
+ return false;
+ }
+ }
+ return false;
}
/**
@@ -2074,8 +2100,6 @@
}
if ( $node->isHtml() ) {
switch ( $node->localName ) {
- # OMITTED:
- /*
case 'select':
$stacklen = $this->stack->length();
for ( $j = $i + 1; $j < $stacklen-1;
$j++ ) {
@@ -2090,7 +2114,6 @@
}
$this->switchMode( 'inSelectMode' );
return;
- */
case 'tr':
$this->switchMode( 'inRowMode' );
return;
@@ -2487,8 +2510,6 @@
# OMITTED:
# OMITTED:
- # OMITTED:
- /*
case 'select':
$this->afe->reconstruct( $this->stack );
$this->stack->insertHTMLElement( $value,
$attribs );
@@ -2504,7 +2525,6 @@
$this->switchMode( 'inSelectMode' );
return true;
}
- */
case 'optgroup':
case 'option':
@@ -3141,16 +3161,103 @@
return $this->inBodyMode( $token, $value, $attribs, $selfclose
);
}
- # OMITTED:
- /*
private function inSelectMode( $token, $value, $attribs = null,
$selfclose = false ) {
- Assert::invariant( false, 'Unimplemented' );
+ if ( $token === 'text' ) {
+ $this->stack->insertText( $value );
+ return true;
+ } elseif ( $token === 'eof' ) {
+ return $this->inBodyMode( $token, $value, $attribs,
$selfclose );
+ } elseif ( $token === 'tag' ) {
+ switch ( $value ) {
+ # OMITTED:
+ case 'option':
+ if ( $this->stack->currentNode->isHtmlNamed(
'option' ) ) {
+ $this->stack->pop();
+ }
+
[MediaWiki-commits] [Gerrit] Sync parserTests with core. - change (mediawiki...parsoid)
Cscott has uploaded a new change for review.
https://gerrit.wikimedia.org/r/296924
Change subject: Sync parserTests with core.
..
Sync parserTests with core.
Change-Id: I49f79b77594c1b4e69cbe7afec1f91e8ff985120
---
M tests/parserTests-blacklist.js
M tests/parserTests.txt
M tools/fetch-parserTests.txt.js
3 files changed, 4 insertions(+), 4 deletions(-)
git pull ssh://gerrit.wikimedia.org:29418/mediawiki/services/parsoid
refs/changes/24/296924/1
diff --git a/tests/parserTests-blacklist.js b/tests/parserTests-blacklist.js
index 19c9f9a..81ae363 100644
--- a/tests/parserTests-blacklist.js
+++ b/tests/parserTests-blacklist.js
@@ -934,7 +934,7 @@
add("html2wt", "BUG 41: Template parameters shown as broken links",
"{{{parameter}}}\n");
add("html2wt", "Template with targets containing wikilinks",
"{{[[wiki/Foo|foo]]}}\n\n{{[[wiki/Foo|foo]]}}\n\n{{[[foo}}]]\n");
add("html2wt", "int keyword", "You have lots of money (not!).\n");
-add("html2wt", "int keyword - non-existing message",
"\n");
+add("html2wt", "int keyword - non-existing message", "⧼var⧽\n");
add("html2wt", " and being included", "Foobar\n");
add("html2wt", " being included", "Foo\n");
add("html2wt", " and being included", "Foo\n");
diff --git a/tests/parserTests.txt b/tests/parserTests.txt
index fb27146..f45c666 100644
--- a/tests/parserTests.txt
+++ b/tests/parserTests.txt
@@ -11024,7 +11024,7 @@
!! wikitext
{{int:var}}
!! html
-
+⧼var⧽
!! end
diff --git a/tools/fetch-parserTests.txt.js b/tools/fetch-parserTests.txt.js
index c763927..68000ab 100755
--- a/tools/fetch-parserTests.txt.js
+++ b/tools/fetch-parserTests.txt.js
@@ -12,9 +12,9 @@
// and update these hashes automatically.
//
// You can use 'sha1sum -b tests/parser/parserTests.txt' to compute this value:
-var expectedSHA1 = "4bdc214a83d57881c30c1a18f47fc85957f0059a";
+var expectedSHA1 = "82b1c56d0a15db33b13e3bc1a1f7f9c817e0863c";
// git log --pretty=oneline -1 tests/parser/parserTests.txt
-var latestCommit = "32a2661a56db1be717ce431c67260bbea771558f";
+var latestCommit = "6cff81981569412ec779495370dfe9566cc175c9";
var fs = require('fs');
var path = require('path');
--
To view, visit https://gerrit.wikimedia.org/r/296924
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: I49f79b77594c1b4e69cbe7afec1f91e8ff985120
Gerrit-PatchSet: 1
Gerrit-Project: mediawiki/services/parsoid
Gerrit-Branch: master
Gerrit-Owner: Cscott
___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] Revert "Use explicit LZMA_STREAM_INIT initializer, instead o... - change (openzim)
Cscott has uploaded a new change for review.
https://gerrit.wikimedia.org/r/296752
Change subject: Revert "Use explicit LZMA_STREAM_INIT initializer, instead of
memset."
..
Revert "Use explicit LZMA_STREAM_INIT initializer, instead of memset."
This reverts commit 498539d869bbb9add9e795432520f305523e09bf.
Turns out that the clang compiler doesn't like this form of initializer:
it must be a GCC extension of some kind. There's nothing technically
wrong with the previous `memset` way of initializing the variable,
it just looks kind of gross. But functionality trumps aesthetics.
Change-Id: I1d5a12c6eb40706023f81323c04a356da3356f42
---
M zimlib/src/lzmastream.cpp
1 file changed, 3 insertions(+), 2 deletions(-)
git pull ssh://gerrit.wikimedia.org:29418/openzim refs/changes/52/296752/1
diff --git a/zimlib/src/lzmastream.cpp b/zimlib/src/lzmastream.cpp
index bd02bd8..d880933 100644
--- a/zimlib/src/lzmastream.cpp
+++ b/zimlib/src/lzmastream.cpp
@@ -58,10 +58,11 @@
}
LzmaStreamBuf::LzmaStreamBuf(std::streambuf* sink_, uint32_t preset,
lzma_check check, unsigned bufsize_)
-: stream(LZMA_STREAM_INIT),
- obuffer(bufsize_),
+: obuffer(bufsize_),
sink(sink_)
{
+std::memset(reinterpret_cast(&stream), 0, sizeof(stream));
+
checkError(
::lzma_easy_encoder(&stream, preset, check));
--
To view, visit https://gerrit.wikimedia.org/r/296752
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: I1d5a12c6eb40706023f81323c04a356da3356f42
Gerrit-PatchSet: 1
Gerrit-Project: openzim
Gerrit-Branch: master
Gerrit-Owner: Cscott
___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] Explain that ZIntStream only represents uint32_t values righ... - change (openzim)
Cscott has uploaded a new change for review. https://gerrit.wikimedia.org/r/296657 Change subject: Explain that ZIntStream only represents uint32_t values right now. .. Explain that ZIntStream only represents uint32_t values right now. Change-Id: Id712847949af1ce32b2e3118fc241be171124186 --- M zimlib/include/zim/zintstream.h 1 file changed, 4 insertions(+), 1 deletion(-) git pull ssh://gerrit.wikimedia.org:29418/openzim refs/changes/57/296657/1 diff --git a/zimlib/include/zim/zintstream.h b/zimlib/include/zim/zintstream.h index 1c78a1f..66fdfef 100644 --- a/zimlib/include/zim/zintstream.h +++ b/zimlib/include/zim/zintstream.h @@ -41,7 +41,10 @@ substracted from the actual number, so a 2 byte zero is actually a 128. The same logic continues on the 3rd, 4th, ... byte. Up to 7 additional bytes - are used, so the first byte must contain at least one 0. + could used, since the first byte must contain at least one 0. + + This particular implementation only represents uint32_t values (numbers up + to 2^32-1), so it will only ever emit 5 bytes per input value. binary range --- -- -- To view, visit https://gerrit.wikimedia.org/r/296657 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: newchange Gerrit-Change-Id: Id712847949af1ce32b2e3118fc241be171124186 Gerrit-PatchSet: 1 Gerrit-Project: openzim Gerrit-Branch: master Gerrit-Owner: Cscott ___ MediaWiki-commits mailing list MediaWiki-commits@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] Actually call the (previously-unused) ArticleSource#setFilen... - change (openzim)
Cscott has uploaded a new change for review.
https://gerrit.wikimedia.org/r/296632
Change subject: Actually call the (previously-unused) ArticleSource#setFilename
method.
..
Actually call the (previously-unused) ArticleSource#setFilename method.
Change-Id: I00d340f86c91419f1237976f6eb636ea8c32a743
---
M zimlib/src/zimcreator.cpp
1 file changed, 1 insertion(+), 0 deletions(-)
git pull ssh://gerrit.wikimedia.org:29418/openzim refs/changes/32/296632/1
diff --git a/zimlib/src/zimcreator.cpp b/zimlib/src/zimcreator.cpp
index 46c550f..ba47d24 100644
--- a/zimlib/src/zimcreator.cpp
+++ b/zimlib/src/zimcreator.cpp
@@ -110,6 +110,7 @@
? fname.substr(0, fname.size() - 4)
: fname;
log_debug("basename " << basename);
+ src.setFilename(fname);
INFO("create directory entries");
createDirentsAndClusters(src, basename + ".tmp");
--
To view, visit https://gerrit.wikimedia.org/r/296632
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: I00d340f86c91419f1237976f6eb636ea8c32a743
Gerrit-PatchSet: 1
Gerrit-Project: openzim
Gerrit-Branch: master
Gerrit-Owner: Cscott
___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] Bug fix: preserve cluster compression type after clear(). - change (openzim)
Cscott has uploaded a new change for review.
https://gerrit.wikimedia.org/r/296547
Change subject: Bug fix: preserve cluster compression type after clear().
..
Bug fix: preserve cluster compression type after clear().
By nulling out the cluster implementation on clear, we were also
resetting the cluster compression type. This caused the ZimCreator
API to not compress compressible clusters after the first one.
This is a follow-up to f5de40f94b30795f42bb9388cbb46df9cd605167,
which reused cluster objects instead of recreating them from scratch
each time.
Related issue: the `size()` of an empty cluster is actually 4, not 0,
since even an empty cluster contains an field which counts the number
of offsets in the cluster.
Change-Id: I468a1719a33c450db9a28d9704b539bdb97cd7fc
---
M zimlib/include/zim/cluster.h
1 file changed, 2 insertions(+), 2 deletions(-)
git pull ssh://gerrit.wikimedia.org:29418/openzim refs/changes/47/296547/1
diff --git a/zimlib/include/zim/cluster.h b/zimlib/include/zim/cluster.h
index c26c24c..bd55cb5 100644
--- a/zimlib/include/zim/cluster.h
+++ b/zimlib/include/zim/cluster.h
@@ -88,8 +88,8 @@
Blob getBlob(size_type n) const;
size_type count() const { return impl ? impl->getCount() : 0; }
- size_type size() const{ return impl ? impl->getSize() : 0; }
- void clear() { impl = 0; }
+ size_type size() const{ return impl ? impl->getSize():
sizeof(size_type); }
+ void clear() { if (impl) impl->clear(); }
void addBlob(const char* data, unsigned size) { getImpl()->addBlob(data,
size); }
void addBlob(const Blob& blob){
getImpl()->addBlob(blob); }
--
To view, visit https://gerrit.wikimedia.org/r/296547
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: I468a1719a33c450db9a28d9704b539bdb97cd7fc
Gerrit-PatchSet: 1
Gerrit-Project: openzim
Gerrit-Branch: master
Gerrit-Owner: Cscott
___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] Use explicit LZMA_STREAM_INIT initializer, instead of memset. - change (openzim)
Cscott has uploaded a new change for review.
https://gerrit.wikimedia.org/r/296548
Change subject: Use explicit LZMA_STREAM_INIT initializer, instead of memset.
..
Use explicit LZMA_STREAM_INIT initializer, instead of memset.
This follows the recommendations in lzma.h.
Change-Id: Ib6b392ba3c6a249fa55b8cf8c04cdae1ae407925
---
M zimlib/src/lzmastream.cpp
1 file changed, 2 insertions(+), 3 deletions(-)
git pull ssh://gerrit.wikimedia.org:29418/openzim refs/changes/48/296548/1
diff --git a/zimlib/src/lzmastream.cpp b/zimlib/src/lzmastream.cpp
index d880933..bd02bd8 100644
--- a/zimlib/src/lzmastream.cpp
+++ b/zimlib/src/lzmastream.cpp
@@ -58,11 +58,10 @@
}
LzmaStreamBuf::LzmaStreamBuf(std::streambuf* sink_, uint32_t preset,
lzma_check check, unsigned bufsize_)
-: obuffer(bufsize_),
+: stream(LZMA_STREAM_INIT),
+ obuffer(bufsize_),
sink(sink_)
{
-std::memset(reinterpret_cast(&stream), 0, sizeof(stream));
-
checkError(
::lzma_easy_encoder(&stream, preset, check));
--
To view, visit https://gerrit.wikimedia.org/r/296548
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: Ib6b392ba3c6a249fa55b8cf8c04cdae1ae407925
Gerrit-PatchSet: 1
Gerrit-Project: openzim
Gerrit-Branch: master
Gerrit-Owner: Cscott
___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] Bug fix: correctly update cluster offsets in directory entries. - change (openzim)
Cscott has uploaded a new change for review.
https://gerrit.wikimedia.org/r/296158
Change subject: Bug fix: correctly update cluster offsets in directory entries.
..
Bug fix: correctly update cluster offsets in directory entries.
This is a follow-up to f5de40f94b30795f42bb9388cbb46df9cd605167.
When we moved the blob writing to the main dirent-creation loop,
we ended up making separate *copies* of the dirents and updating
blob/cluster information in these, instead of the dirents in the
main list which will eventually be written. Make the auxilliary
lists contain dirent *pointers* to avoid this problem.
Change-Id: I008fa700acd90c3c51614bde65d61ffbc6061872
---
M zimlib/include/zim/writer/zimcreator.h
M zimlib/src/zimcreator.cpp
2 files changed, 10 insertions(+), 9 deletions(-)
git pull ssh://gerrit.wikimedia.org:29418/openzim refs/changes/58/296158/1
diff --git a/zimlib/include/zim/writer/zimcreator.h
b/zimlib/include/zim/writer/zimcreator.h
index 6f47402..2e52d7e 100644
--- a/zimlib/include/zim/writer/zimcreator.h
+++ b/zimlib/include/zim/writer/zimcreator.h
@@ -33,6 +33,7 @@
{
public:
typedef std::vector DirentsType;
+typedef std::vector DirentPtrsType;
typedef std::vector SizeVectorType;
typedef std::vector OffsetsType;
typedef std::map MimeTypes;
diff --git a/zimlib/src/zimcreator.cpp b/zimlib/src/zimcreator.cpp
index bc977ea..46c550f 100644
--- a/zimlib/src/zimcreator.cpp
+++ b/zimlib/src/zimcreator.cpp
@@ -144,7 +144,7 @@
// because we don't know which one will fill up first. We also need
// to track the dirents currently in each, so we can fix up the
// cluster index if the other one ends up written first.
- DirentsType compDirents, uncompDirents;
+ DirentPtrsType compDirents, uncompDirents;
Cluster compCluster, uncompCluster;
compCluster.setCompression(compression);
uncompCluster.setCompression(zimcompNone);
@@ -188,11 +188,11 @@
}
}
-dirents.push_back(dirent);
currentSize +=
dirent.getDirentSize() /* for directory entry */ +
sizeof(offset_type) /* for url pointer list */ +
sizeof(size_type) /* for title pointer list */;
+dirents.push_back(dirent);
// If this is a redirect, we're done: there's no blob to add.
if (dirent.isRedirect())
@@ -217,7 +217,7 @@
}
Cluster *cluster;
-DirentsType *myDirents, *otherDirents;
+DirentPtrsType *myDirents, *otherDirents;
if (dirent.isCompress())
{
cluster = &compCluster;
@@ -230,9 +230,9 @@
myDirents = &uncompDirents;
otherDirents = &compDirents;
}
-myDirents->push_back(dirent);
-dirent.setCluster(clusterOffsets.size(), cluster->count());
+dirents.back().setCluster(clusterOffsets.size(), cluster->count());
cluster->addBlob(blob);
+myDirents->push_back(&(dirents.back()));
// If cluster is now large enough, write it to disk.
if (cluster->size() >= minChunkSize * 1024)
@@ -247,10 +247,10 @@
cluster->clear();
myDirents->clear();
// Update the cluster number of the dirents *not* written to disk.
- for (DirentsType::iterator di = otherDirents->begin();
+ for (DirentPtrsType::iterator di = otherDirents->begin();
di != otherDirents->end(); ++di)
{
-di->setCluster(clusterOffsets.size(), di->getBlobNumber());
+(*di)->setCluster(clusterOffsets.size(), (*di)->getBlobNumber());
}
offset_type end = out.tellp();
currentSize += (end - start) +
@@ -263,10 +263,10 @@
{
clusterOffsets.push_back(out.tellp());
out << compCluster;
-for (DirentsType::iterator di = uncompDirents.begin();
+for (DirentPtrsType::iterator di = uncompDirents.begin();
di != uncompDirents.end(); ++di)
{
- di->setCluster(clusterOffsets.size(), di->getBlobNumber());
+ (*di)->setCluster(clusterOffsets.size(), (*di)->getBlobNumber());
}
}
compCluster.clear();
--
To view, visit https://gerrit.wikimedia.org/r/296158
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: I008fa700acd90c3c51614bde65d61ffbc6061872
Gerrit-PatchSet: 1
Gerrit-Project: openzim
Gerrit-Branch: master
Gerrit-Owner: Cscott
___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] Don't always update tests. - change (mediawiki...parsoid)
Cscott has uploaded a new change for review.
https://gerrit.wikimedia.org/r/295772
Change subject: Don't always update tests.
..
Don't always update tests.
Change-Id: I5db3aab9240b20797f6cd696f3440bc3b598515f
---
M bin/parserTests.js
1 file changed, 2 insertions(+), 2 deletions(-)
git pull ssh://gerrit.wikimedia.org:29418/mediawiki/services/parsoid
refs/changes/72/295772/1
diff --git a/bin/parserTests.js b/bin/parserTests.js
index 970d074..b0d6624 100755
--- a/bin/parserTests.js
+++ b/bin/parserTests.js
@@ -2152,14 +2152,14 @@
}
// Write updated tests from failed ones
- if (options.hasOwnProperty('update-tests') ||
+ if (options['update-tests'] ||
booleanOption(options['update-unexpected'])) {
var updateFormat = (options['update-tests'] === 'raw') ?
'raw' : 'actualNormalized';
var parserTestsFilename = __dirname +
'/../tests/parserTests.txt';
var parserTests = fs.readFileSync(parserTestsFilename,
'utf8');
this.stats.modes.wt2html.failList.forEach(function(fail) {
- if (booleanOption(options['update-tests'] ||
fail.unexpected)) {
+ if (options['update-tests'] || fail.unexpected)
{
var exp = new RegExp("(" +
/!!\s*test\s*/.source +
Util.escapeRegExp(fail.title) +
/(?:(?!!!\s*end)[\s\S])*/.source +
")(" +
Util.escapeRegExp(fail.expected) + ")", "m");
--
To view, visit https://gerrit.wikimedia.org/r/295772
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: I5db3aab9240b20797f6cd696f3440bc3b598515f
Gerrit-PatchSet: 1
Gerrit-Project: mediawiki/services/parsoid
Gerrit-Branch: master
Gerrit-Owner: Cscott
___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] Use improved format specifier for TemplateData. - change (mediawiki...parsoid)
Cscott has uploaded a new change for review.
https://gerrit.wikimedia.org/r/295741
Change subject: Use improved format specifier for TemplateData.
..
Use improved format specifier for TemplateData.
Thanks to Thiemo Mättig for the suggestion and specification at
Wikimania 2016 in Esino Lario.
Bug: T138492
Change-Id: I3f32aa308259c935d01e277b9ff4886d1f1c97ae
---
M lib/html2wt/WikitextSerializer.js
M tests/parserTests-blacklist.js
2 files changed, 53 insertions(+), 38 deletions(-)
git pull ssh://gerrit.wikimedia.org:29418/mediawiki/services/parsoid
refs/changes/41/295741/1
diff --git a/lib/html2wt/WikitextSerializer.js
b/lib/html2wt/WikitextSerializer.js
index ea47a61..3c8b1b8 100644
--- a/lib/html2wt/WikitextSerializer.js
+++ b/lib/html2wt/WikitextSerializer.js
@@ -373,6 +373,7 @@
// See
https://github.com/wikimedia/mediawiki-extensions-TemplateData/blob/master/Specification.md
// for the templatedata specification.
+// Formatting spec is at T138492.
WSP._serializeTemplate = function(node, tpl, isTpl, tplData) {
// open the transclusion
var buf = '{{' + tpl.target.wt;
@@ -523,29 +524,37 @@
// this strategy prevents unnecessary normalization
// of edited transclusions which don't have valid
// templatedata formatting information.
+ // See https://phabricator.wikimedia.org/T138492 for details on
format specification string.
- var defaultBlockSpc = [' ', ' ', ' ', '\n']; // "| foo = bar\n"
- var defaultInlineSpc = ['', '', '', '']; // "|foo=bar"
+ var defaultBlockSpc = '| _ = _\n'; // "block"
+ var defaultInlineSpc = '|_=_'; // "inline"
// FIXME: Do a full regexp test maybe?
if (/.*data-parsoid\/0.0.1"$/.test(env.page.dpContentType)) {
// For previous versions of data-parsoid,
// wt2html pipeline used "|foo = bar" style args
// as the default.
- defaultInlineSpc = ['', ' ', ' ', ''];
+ defaultInlineSpc = "|_ = _";
}
var format = tplData && tplData.format ?
tplData.format.toLowerCase() : null;
- var useBlockFormat = format === 'block';
- var useInlineFormat = format === 'inline' || (!useBlockFormat
&& DU.isNewElt(node));
- if (useBlockFormat &&
!TRAILING_COMMENT_OR_WS_AFTER_NL_REGEXP.test(buf)) {
- buf += '\n';
- } else if (useInlineFormat) {
- buf = buf.replace(/\n/g, ' ').trim();
+ if (format === 'block') { format = defaultBlockSpc; }
+ if (format === 'inline') { format = defaultInlineSpc; }
+ // Check format string for validity.
+ if (format && !/^ *[|] *_+ *= *_+ *\n?$/.test(format)) { format
= null; }
+ var isBlockFormat = format && /\n/.test(format);
+ if (format) {
+ // "magic case": If the format string ends with a
newline, an extra newline is added
+ // between the template name and the first parameter.
+ if (isBlockFormat) {
+ if
(!TRAILING_COMMENT_OR_WS_AFTER_NL_REGEXP.test(buf)) {
+ buf += '\n';
+ }
+ } else {
+ buf = buf.replace(/\n/g, ' ').trim();
+ }
}
for (var i = 0; i < argBuf.length; i++) {
- buf += '|';
-
var arg = argBuf[i];
var name = arg.name;
var val = arg.value;
@@ -553,45 +562,53 @@
// We are serializing a positional parameter.
// Whitespace is significant for these and
// formatting would change semantics.
- buf += val;
+ buf += '|' + val;
} else {
// We are serializing a named parameter.
// arg.value has had its whitespace trimmed
already.
- var spc;
+ var sfmt = format;
+ var forceInline = format && !isBlockFormat;
if (name === '') {
// No spacing for blank parameters
({{foo|=bar}})
// This should be an edge case and
probably only for
// inline-formatted templates, but we
are consciously
// forcing this default he
[MediaWiki-commits] [Gerrit] Move `throw` statement out of header file. - change (openzim)
Cscott has uploaded a new change for review.
https://gerrit.wikimedia.org/r/295720
Change subject: Move `throw` statement out of header file.
..
Move `throw` statement out of header file.
This ensures that we don't fail compilation when including the header
file in an environment where exception handling is disabled (like
when binding to node.js).
Change-Id: Ib49060c7fe479054e58c20249dd9b3236ea7eb03
---
M zimlib/include/zim/writer/articlesource.h
M zimlib/src/articlesource.cpp
2 files changed, 5 insertions(+), 1 deletion(-)
git pull ssh://gerrit.wikimedia.org:29418/openzim refs/changes/20/295720/1
diff --git a/zimlib/include/zim/writer/articlesource.h
b/zimlib/include/zim/writer/articlesource.h
index 94ee91b..54653ef 100644
--- a/zimlib/include/zim/writer/articlesource.h
+++ b/zimlib/include/zim/writer/articlesource.h
@@ -97,7 +97,8 @@
* code to not use it.
* This should be removed once every users switch to new API.
*/
-virtual Blob getData(const std::string& aid) { throw "This should not
be called"; };
+virtual Blob getData(const std::string& aid);
+
/**/
};
diff --git a/zimlib/src/articlesource.cpp b/zimlib/src/articlesource.cpp
index cc72ae2..a2087a7 100644
--- a/zimlib/src/articlesource.cpp
+++ b/zimlib/src/articlesource.cpp
@@ -80,6 +80,9 @@
std::cerr << " You should override Article::getData
directly." << std::endl;
return __source->getData(getAid());
}
+Blob ArticleSource::getData(const std::string& aid) {
+throw std::runtime_error("This should not be called");
+}
/**/
Uuid ArticleSource::getUuid()
--
To view, visit https://gerrit.wikimedia.org/r/295720
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: Ib49060c7fe479054e58c20249dd9b3236ea7eb03
Gerrit-PatchSet: 1
Gerrit-Project: openzim
Gerrit-Branch: master
Gerrit-Owner: Cscott
___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] Allow extensions to handle specific contentmodels. - change (mediawiki...parsoid)
Cscott has uploaded a new change for review.
https://gerrit.wikimedia.org/r/295707
Change subject: Allow extensions to handle specific contentmodels.
..
Allow extensions to handle specific contentmodels.
Some extensions (for example, Extension:ProofreadPage) do more than
register specific extension tags: they also hook the parser to declare
responsibility for a specific contentmodel (like "proofread-page" or "json").
These are
https://www.mediawiki.org/wiki/Category:ContentHandler_extensions
(as opposed to https://www.mediawiki.org/wiki/Category:Tag_extensions).
See https://www.mediawiki.org/wiki/Manual:ContentHandler for more
details.
We abstract the top-level parser entry points to allow dispatching
to extensions to parse alternative content models and add a
core extension as a demonstration that handles the "json"
content model, rendering it in DOM as an HTML table (as
the json content model in mediawiki core does).
Change-Id: I7ca31c99de8e04b1359bc521df121db0eb69e384
---
M bin/parse.js
M bin/roundtrip-test.js
M lib/api/routes.js
M lib/config/ParsoidConfig.js
M lib/config/WikiConfig.js
M lib/config/extapi.js
A lib/ext/JSON/index.js
M lib/utils/DOMUtils.js
M lib/wt2html/DOMPostProcessor.js
M package.json
M tests/mocha/api.js
M tests/mocha/parse.js
M tests/mocha/test.helpers.js
M tests/mockAPI.js
14 files changed, 550 insertions(+), 53 deletions(-)
git pull ssh://gerrit.wikimedia.org:29418/mediawiki/services/parsoid
refs/changes/07/295707/1
diff --git a/bin/parse.js b/bin/parse.js
index 852cecd..4a53b24 100755
--- a/bin/parse.js
+++ b/bin/parse.js
@@ -72,6 +72,11 @@
'boolean': false,
'default': ParserEnv.prototype.defaultPageName,
},
+ 'contentmodel': {
+ description: 'The content model of the input. Defaults to
"wikitext" but extensions may support others (for example, "json").',
+ 'boolean': false,
+ 'default': null,
+ },
'oldid': {
description: 'Oldid of the given page.',
'boolean': false,
@@ -165,19 +170,26 @@
if (pb) {
DU.applyPageBundle(doc, pb);
}
+ if (argv.contentmodel) {
+ env.page.meta.revision.contentmodel = argv.contentmodel;
+ }
return DU.serializeDOM(env, doc.body, argv.selser).then(function(out) {
if (argv.html2wt || argv.wt2wt) {
return { trailingNL: true, out: out, env: env };
} else {
- return startsAtWikitext(argv, env, out);
+ env.setPageSrcInfo(out);
+ return startsAtWikitext(argv, env);
}
});
};
-startsAtWikitext = function(argv, env, input) {
- env.setPageSrcInfo(input);
+startsAtWikitext = function(argv, env) {
+ // input string is in env.page.src.
+ console.assert(env.page.meta);
+
+ var handler = env.conf.wiki.getContentHandler(env, argv.contentmodel);
// Kick off the pipeline by feeding the input into the parser pipeline
- return env.pipelineFactory.parse(env, env.page.src).then(function(doc) {
+ return handler.toHTML(env).then(function(doc) {
if (argv.lint) {
env.log("end/parse");
}
@@ -249,13 +261,15 @@
}
if (typeof input === 'string') {
- return input;
+ env.setPageSrcInfo(input.replace(/\r/g, ''));
+ return;
}
if (argv.inputfile) {
// read input from the file, then process
var fileContents = fs.readFileSync(argv.inputfile,
'utf8');
- return fileContents;
+ env.setPageSrcInfo(fileContents.replace(/\r/g, ''));
+ return;
}
// Send a message to stderr if there is no input for a while,
since the
@@ -280,17 +294,17 @@
clearTimeout(stdinTimer);
// parse page if no input
if (inputChunks.length > 0) {
- return inputChunks.join('');
+
env.setPageSrcInfo(inputChunks.join('').replace(/\r/g, ''));
+ return;
} else if (argv.html2wt || argv.html2html) {
env.log("fatal", "Pages start at wikitext.");
}
var target = env.normalizeAndResolvePageTitle();
return TemplateRequest
- .setPageSrcInfo(env, target, argv.oldid)
- .then(function() { return env.page.src; });
+ .setPageSrcInfo(env, target, argv.oldid);
[MediaWiki-commits] [Gerrit] Move zim::writer::ArticleSource::getData() to zim::writer::A... - change (openzim)
Cscott has uploaded a new change for review.
https://gerrit.wikimedia.org/r/295460
Change subject: Move zim::writer::ArticleSource::getData() to
zim::writer::Article::getData().
..
Move zim::writer::ArticleSource::getData() to zim::writer::Article::getData().
When building a ZIM file incrementally, it makes more sense to provide
the data upfront as soon as you return the Article object from
ArticleSource::getNextArticle. This is the way that the Category
object already works.
Change-Id: I78f2a69ae3931cc43a51cdab360468e13fcc54cb
---
M zimlib/examples/createZimExample.cpp
M zimlib/include/zim/writer/articlesource.h
M zimlib/src/zimcreator.cpp
M zimwriterfs/zimwriterfs.cpp
4 files changed, 23 insertions(+), 23 deletions(-)
git pull ssh://gerrit.wikimedia.org:29418/openzim refs/changes/60/295460/1
diff --git a/zimlib/examples/createZimExample.cpp
b/zimlib/examples/createZimExample.cpp
index 914000c..95b8d09 100644
--- a/zimlib/examples/createZimExample.cpp
+++ b/zimlib/examples/createZimExample.cpp
@@ -40,7 +40,7 @@
virtual std::string getMimeType() const;
virtual std::string getRedirectAid() const;
-zim::Blob data()
+virtual zim::Blob getData() const
{ return zim::Blob(&_data[0], _data.size()); }
};
@@ -96,7 +96,6 @@
explicit TestArticleSource(unsigned max = 16);
virtual const zim::writer::Article* getNextArticle();
-virtual zim::Blob getData(const std::string& aid);
};
TestArticleSource::TestArticleSource(unsigned max)
@@ -119,14 +118,6 @@
unsigned n = _next++;
return &_articles[n];
-}
-
-zim::Blob TestArticleSource::getData(const std::string& aid)
-{
- unsigned n;
- std::istringstream s(aid);
- s >> n;
- return _articles[n-1].data();
}
int main(int argc, char* argv[])
diff --git a/zimlib/include/zim/writer/articlesource.h
b/zimlib/include/zim/writer/articlesource.h
index 1fda337..c0ceefc 100644
--- a/zimlib/include/zim/writer/articlesource.h
+++ b/zimlib/include/zim/writer/articlesource.h
@@ -45,6 +45,7 @@
virtual bool shouldCompress() const;
virtual std::string getRedirectAid() const;
virtual std::string getParameter() const;
+virtual Blob getData() const = 0;
// returns the next category id, to which the article is assigned to
virtual std::string getNextCategory();
@@ -63,7 +64,6 @@
public:
virtual void setFilename(const std::string& fname) { }
virtual const Article* getNextArticle() = 0;
-virtual Blob getData(const std::string& aid) = 0;
virtual Uuid getUuid();
virtual std::string getMainPage();
virtual std::string getLayoutPage();
diff --git a/zimlib/src/zimcreator.cpp b/zimlib/src/zimcreator.cpp
index ac2720b..7cf21c6 100644
--- a/zimlib/src/zimcreator.cpp
+++ b/zimlib/src/zimcreator.cpp
@@ -201,7 +201,7 @@
}
// Add blob data to compressed or uncompressed cluster.
-Blob blob = src.getData(dirent.getAid());
+Blob blob = article->getData();
if (blob.size() > 0)
{
isEmpty = false;
diff --git a/zimwriterfs/zimwriterfs.cpp b/zimwriterfs/zimwriterfs.cpp
index f110e29..bd77862 100644
--- a/zimwriterfs/zimwriterfs.cpp
+++ b/zimwriterfs/zimwriterfs.cpp
@@ -401,9 +401,12 @@
return retVal;
}
+class ArticleSource;
+
/* Article class */
class Article : public zim::writer::Article {
protected:
+ArticleSource *source;
char ns;
bool invalid;
std::string aid;
@@ -414,10 +417,8 @@
std::string data;
public:
-Article() {
- invalid = false;
-}
-explicit Article(const std::string& id, const bool detectRedirects);
+Article(ArticleSource *source_) : source(source_), invalid(false) { }
+explicit Article(ArticleSource *source, const std::string& id, const bool
detectRedirects);
virtual std::string getAid() const;
virtual char getNamespace() const;
virtual std::string getUrl() const;
@@ -427,11 +428,13 @@
virtual std::string getMimeType() const;
virtual std::string getRedirectAid() const;
virtual bool shouldCompress() const;
+virtual zim::Blob getData() const;
};
class MetadataArticle : public Article {
public:
- MetadataArticle(std::string &id) {
+ MetadataArticle(ArticleSource *source, std::string &id) :
+ Article(source) {
if (id == "Favicon") {
aid = "/-/" + id;
mimeType="image/png";
@@ -449,7 +452,8 @@
class RedirectArticle : public Article {
public:
- RedirectArticle(const std::string &line) {
+ RedirectArticle(ArticleSource *source, const std::string &line) :
+ Article(source) {
size_t start;
size_t end;
ns = line[0];
@@ -633,8 +637,7 @@
return computeRelativePath(baseUrl, newUrl);
}
-Article::Article(const std::string& path, const bool detectRedirects = true) {
- invalid = false;
+Article::Article(ArticleSource *source_, const std::string& pa
[MediaWiki-commits] [Gerrit] Update build instructions for MacOS X. - change (openzim)
Cscott has uploaded a new change for review. https://gerrit.wikimedia.org/r/295382 Change subject: Update build instructions for MacOS X. .. Update build instructions for MacOS X. Change-Id: I0190ada3bf02c42ed34cb6d253516a9c662357e1 --- M zimlib/README.md M zimwriterfs/README.md 2 files changed, 31 insertions(+), 2 deletions(-) git pull ssh://gerrit.wikimedia.org:29418/openzim refs/changes/82/295382/1 diff --git a/zimlib/README.md b/zimlib/README.md index ce35ed9..9aef82c 100644 --- a/zimlib/README.md +++ b/zimlib/README.md @@ -16,5 +16,24 @@ make ``` +OSX compilation +--- +On MacOSX, you'll need to install some packages from +[homebrew](http://brew.sh/): +``` +brew update +brew install xz libmagic +``` +You'll also want to add `/usr/local/include` to your search path, +for example: +``` +./autogen.sh +./configure CFLAGS=-I/usr/local/include CXXFLAGS=-I/usr/local/include +make +``` + +License +--- + The `zimlib` library is released under the GPLv2 license terms. diff --git a/zimwriterfs/README.md b/zimwriterfs/README.md index 8695643..1cbb7dc 100644 --- a/zimwriterfs/README.md +++ b/zimwriterfs/README.md @@ -44,9 +44,19 @@ ``` OSX compilation - +--- +OSX builds are similar to Linux, except we use homebrew. Change to +`../zimlib` and build zimlib as instructed in the README there. Then +return here and: +``` +brew install gumbo-parser +./autogen.sh +./configure CXXFLAGS="-I../zimlib/include -I/usr/local/include" LDFLAGS=-L../zimlib/src/.libs +make +``` -On MaxOSX, a script helps you build zimwriterfs both statically and dynamically. +Alternatively, there is a script included here to help you build both +static and dynamic binaries for `zimwriterfs`. You must have a working and set up Kiwix repository (with dependencies ready). 1. Install libmagic with brew (it's important) -- To view, visit https://gerrit.wikimedia.org/r/295382 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I0190ada3bf02c42ed34cb6d253516a9c662357e1 Gerrit-PatchSet: 1 Gerrit-Project: openzim Gerrit-Branch: master Gerrit-Owner: Cscott ___ MediaWiki-commits mailing list MediaWiki-commits@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] Compress blobs and track file size immediately after adding ... - change (openzim)
Cscott has uploaded a new change for review.
https://gerrit.wikimedia.org/r/295383
Change subject: Compress blobs and track file size immediately after adding
each article.
..
Compress blobs and track file size immediately after adding each article.
Rather than first collecting all the directory entries and only afterwards
writing the blobs, write each cluster on the fly as we see each article.
Keep track of both compressed and uncompressed clusters so that we don't
needlessly terminate compressed clusters just because we happen to have
encountered an uncompressible file. Account for additions to each of
the various indices as we go so that we maintain a fairly accurate
size for the file at every point, which will allow us to stop adding
articles once the ZIM file gets to a certain size.
Change-Id: Ib644fff4cb804320a07aadbea499c8416df66adc
---
M zimlib/include/zim/writer/zimcreator.h
M zimlib/src/zimcreator.cpp
2 files changed, 124 insertions(+), 86 deletions(-)
git pull ssh://gerrit.wikimedia.org:29418/openzim refs/changes/83/295383/1
diff --git a/zimlib/include/zim/writer/zimcreator.h
b/zimlib/include/zim/writer/zimcreator.h
index 5134d98..4296371 100644
--- a/zimlib/include/zim/writer/zimcreator.h
+++ b/zimlib/include/zim/writer/zimcreator.h
@@ -52,10 +52,10 @@
CompressionType compression;
bool isEmpty;
offset_type clustersSize;
+offset_type currentSize;
-void createDirents(ArticleSource& src);
+void createDirents(ArticleSource& src, const std::string& tmpfname);
void createTitleIndex(ArticleSource& src);
-void createClusters(ArticleSource& src, const std::string& tmpfname);
void fillHeader(ArticleSource& src);
void write(const std::string& fname, const std::string& tmpfname);
@@ -84,6 +84,10 @@
void setMinChunkSize(int s) { minChunkSize = s; }
void create(const std::string& fname, ArticleSource& src);
+
+/* The user can query `currentSize` after each article has been
+ * added to the ZIM file. */
+offset_type getCurrentSize() { return currentSize; }
};
}
diff --git a/zimlib/src/zimcreator.cpp b/zimlib/src/zimcreator.cpp
index 66ce902..208c050 100644
--- a/zimlib/src/zimcreator.cpp
+++ b/zimlib/src/zimcreator.cpp
@@ -56,28 +56,30 @@
: minChunkSize(1024-64),
nextMimeIdx(0),
#ifdef ENABLE_LZMA
-compression(zimcompLzma)
+compression(zimcompLzma),
#elif ENABLE_BZIP2
-compression(zimcompBzip2)
+compression(zimcompBzip2),
#elif ENABLE_ZLIB
-compression(zimcompZip)
+compression(zimcompZip),
#else
-compression(zimcompNone)
+compression(zimcompNone),
#endif
+currentSize(0)
{
}
ZimCreator::ZimCreator(int& argc, char* argv[])
: nextMimeIdx(0),
#ifdef ENABLE_LZMA
-compression(zimcompLzma)
+compression(zimcompLzma),
#elif ENABLE_BZIP2
-compression(zimcompBzip2)
+compression(zimcompBzip2),
#elif ENABLE_ZLIB
-compression(zimcompZip)
+compression(zimcompZip),
#else
-compression(zimcompNone)
+compression(zimcompNone),
#endif
+currentSize(0)
{
Arg minChunkSizeArg(argc, argv, "--min-chunk-size");
if (minChunkSizeArg.isSet())
@@ -110,15 +112,12 @@
log_debug("basename " << basename);
INFO("create directory entries");
- createDirents(src);
+ createDirents(src, basename + ".tmp");
INFO(dirents.size() << " directory entries created");
INFO("create title index");
createTitleIndex(src);
INFO(dirents.size() << " title index created");
-
- INFO("create clusters");
- createClusters(src, basename + ".tmp");
INFO(clusterOffsets.size() << " clusters created");
INFO("fill header");
@@ -132,9 +131,22 @@
INFO("ready");
}
-void ZimCreator::createDirents(ArticleSource& src)
+void ZimCreator::createDirents(ArticleSource& src, const std::string&
tmpfname)
{
INFO("collect articles");
+ std::ofstream out(tmpfname.c_str());
+ currentSize =
+80 /* for header */ +
+1 /* for mime type table termination */;
+
+ // We keep both a "compressed cluster" and an "uncompressed cluster"
+ // because we don't know which one will fill up first. We also need
+ // to track the dirents currently in each, so we can fix up the
+ // cluster index if the other one ends up written first.
+ DirentsType compDirents, uncompDirents;
+ Cluster compCluster, uncompCluster;
+ compCluster.setCompression(compression);
+ uncompCluster.setCompression(zimcompNone);
const Article* article;
while ((article = src.getNextArticle()) != 0)
@@ -163,13 +175,106 @@
}
else
{
+ uint16_t oldMimeIdx = nextMimeIdx;
d
[MediaWiki-commits] [Gerrit] Include files from srcdir; built files in builddir. - change (openzim)
Cscott has uploaded a new change for review. https://gerrit.wikimedia.org/r/295111 Change subject: Include files from srcdir; built files in builddir. .. Include files from srcdir; built files in builddir. Change-Id: I84f097567b944601534cdbc3313216c0364211bd --- M zimlib/examples/Makefile.am M zimlib/src/Makefile.am M zimlib/src/tools/Makefile.am M zimlib/test/Makefile.am M zimreader/src/Makefile.am M zimwriterdb/src/Makefile.am M zimwriterdb/test/Makefile.am 7 files changed, 7 insertions(+), 7 deletions(-) git pull ssh://gerrit.wikimedia.org:29418/openzim refs/changes/11/295111/1 diff --git a/zimlib/examples/Makefile.am b/zimlib/examples/Makefile.am index 13ac63f..3157cf0 100644 --- a/zimlib/examples/Makefile.am +++ b/zimlib/examples/Makefile.am @@ -1,4 +1,4 @@ -AM_CPPFLAGS=-I$(top_builddir)/include +AM_CPPFLAGS=-I$(top_srcdir)/include noinst_PROGRAMS = createZimExample createZimExample_SOURCES = createZimExample.cpp LDADD = $(top_builddir)/src/libzim.la diff --git a/zimlib/src/Makefile.am b/zimlib/src/Makefile.am index 823a3b6..c0bd632 100644 --- a/zimlib/src/Makefile.am +++ b/zimlib/src/Makefile.am @@ -1,4 +1,4 @@ -AM_CPPFLAGS=-I$(top_builddir)/include +AM_CPPFLAGS=-I$(top_srcdir)/include lib_LTLIBRARIES = libzim.la diff --git a/zimlib/src/tools/Makefile.am b/zimlib/src/tools/Makefile.am index a50b7c8..c88ef42 100644 --- a/zimlib/src/tools/Makefile.am +++ b/zimlib/src/tools/Makefile.am @@ -1,4 +1,4 @@ -AM_CPPFLAGS=-I$(top_builddir)/include +AM_CPPFLAGS=-I$(top_srcdir)/include if MAKE_BENCHMARK ZIMBENCH = zimbench endif diff --git a/zimlib/test/Makefile.am b/zimlib/test/Makefile.am index 34dad12..29dd7bd 100644 --- a/zimlib/test/Makefile.am +++ b/zimlib/test/Makefile.am @@ -1,4 +1,4 @@ -AM_CPPFLAGS=-I$(top_builddir)/include +AM_CPPFLAGS=-I$(top_srcdir)/include noinst_PROGRAMS = zimlib-test diff --git a/zimreader/src/Makefile.am b/zimreader/src/Makefile.am index ff40f97..b7fc747 100644 --- a/zimreader/src/Makefile.am +++ b/zimreader/src/Makefile.am @@ -10,7 +10,7 @@ .css.cpp: ecppc $(ECPPFLAGS) -b $(ECPPFLAGS_CSS) $< -AM_CPPFLAGS=-I$(top_builddir)/include +AM_CPPFLAGS=-I$(top_srcdir)/include bin_PROGRAMS = zimreader diff --git a/zimwriterdb/src/Makefile.am b/zimwriterdb/src/Makefile.am index 46b0fac..854f2cb 100644 --- a/zimwriterdb/src/Makefile.am +++ b/zimwriterdb/src/Makefile.am @@ -1,6 +1,6 @@ bin_PROGRAMS = zimwriterdb zimindexer zimcreatorsearch wikizim -AM_CPPFLAGS=-I$(top_builddir)/include +AM_CPPFLAGS=-I$(top_srcdir)/include # wikizim # diff --git a/zimwriterdb/test/Makefile.am b/zimwriterdb/test/Makefile.am index fffe02a..52262b2 100644 --- a/zimwriterdb/test/Makefile.am +++ b/zimwriterdb/test/Makefile.am @@ -6,4 +6,4 @@ createzim_t_LDFLAGS = -lcxxtools -lzim createzim_t_SOURCES = createzim-t.cpp -AM_CPPFLAGS=-I$(top_builddir)/include +AM_CPPFLAGS=-I$(top_srcdir)/include -- To view, visit https://gerrit.wikimedia.org/r/295111 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I84f097567b944601534cdbc3313216c0364211bd Gerrit-PatchSet: 1 Gerrit-Project: openzim Gerrit-Branch: master Gerrit-Owner: Cscott ___ MediaWiki-commits mailing list MediaWiki-commits@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] Use the system's libgumbo. - change (openzim)
Cscott has uploaded a new change for review. https://gerrit.wikimedia.org/r/295039 Change subject: Use the system's libgumbo. .. Use the system's libgumbo. Change-Id: Id6b06033bba6c1d47d305b50259e7eaf41c073e2 --- M zimwriterfs/Makefile.am M zimwriterfs/README.md M zimwriterfs/configure.ac D zimwriterfs/gumbo/attribute.c D zimwriterfs/gumbo/attribute.h D zimwriterfs/gumbo/char_ref.c D zimwriterfs/gumbo/char_ref.h D zimwriterfs/gumbo/error.c D zimwriterfs/gumbo/error.h D zimwriterfs/gumbo/gumbo.h D zimwriterfs/gumbo/insertion_mode.h D zimwriterfs/gumbo/parser.c D zimwriterfs/gumbo/parser.h D zimwriterfs/gumbo/string_buffer.c D zimwriterfs/gumbo/string_buffer.h D zimwriterfs/gumbo/string_piece.c D zimwriterfs/gumbo/string_piece.h D zimwriterfs/gumbo/tag.c D zimwriterfs/gumbo/token_type.h D zimwriterfs/gumbo/tokenizer.c D zimwriterfs/gumbo/tokenizer.h D zimwriterfs/gumbo/tokenizer_states.h D zimwriterfs/gumbo/utf8.c D zimwriterfs/gumbo/utf8.h D zimwriterfs/gumbo/util.c D zimwriterfs/gumbo/util.h D zimwriterfs/gumbo/vector.c D zimwriterfs/gumbo/vector.h 28 files changed, 9 insertions(+), 33,052 deletions(-) git pull ssh://gerrit.wikimedia.org:29418/openzim refs/changes/39/295039/1 -- To view, visit https://gerrit.wikimedia.org/r/295039 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: newchange Gerrit-Change-Id: Id6b06033bba6c1d47d305b50259e7eaf41c073e2 Gerrit-PatchSet: 1 Gerrit-Project: openzim Gerrit-Branch: master Gerrit-Owner: Cscott ___ MediaWiki-commits mailing list MediaWiki-commits@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] Update .gitignore files. - change (openzim)
Cscott has uploaded a new change for review. https://gerrit.wikimedia.org/r/295037 Change subject: Update .gitignore files. .. Update .gitignore files. Change-Id: Iad8d95fa2deab178396e8804b6e195a5b1520a4a --- C zimlib/.gitignore R zimwriterfs/.gitignore 2 files changed, 3 insertions(+), 11 deletions(-) git pull ssh://gerrit.wikimedia.org:29418/openzim refs/changes/37/295037/1 diff --git a/.gitignore b/zimlib/.gitignore similarity index 75% copy from .gitignore copy to zimlib/.gitignore index c50679e..0a4a991 100644 --- a/.gitignore +++ b/zimlib/.gitignore @@ -4,7 +4,6 @@ compile config.* configure -createZimExample depcomp .deps .dirstamp @@ -25,8 +24,6 @@ .svn .*.swp *.zim -zimdiff -zimdump -zimpatch -zimsearch -zimwriterfs +examples/createZimExample +src/tools/zimdump +src/tools/zimsearch diff --git a/.gitignore b/zimwriterfs/.gitignore similarity index 80% rename from .gitignore rename to zimwriterfs/.gitignore index c50679e..1aaab40 100644 --- a/.gitignore +++ b/zimwriterfs/.gitignore @@ -4,7 +4,6 @@ compile config.* configure -createZimExample depcomp .deps .dirstamp @@ -25,8 +24,4 @@ .svn .*.swp *.zim -zimdiff -zimdump -zimpatch -zimsearch zimwriterfs -- To view, visit https://gerrit.wikimedia.org/r/295037 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: newchange Gerrit-Change-Id: Iad8d95fa2deab178396e8804b6e195a5b1520a4a Gerrit-PatchSet: 1 Gerrit-Project: openzim Gerrit-Branch: master Gerrit-Owner: Cscott ___ MediaWiki-commits mailing list MediaWiki-commits@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] Update zimwriterfs makefiles. - change (openzim)
Cscott has uploaded a new change for review. https://gerrit.wikimedia.org/r/295038 Change subject: Update zimwriterfs makefiles. .. Update zimwriterfs makefiles. Change-Id: Ic0f93e19eb3e07d86c115bd15f47ef5fbc74f954 --- M zimwriterfs/Makefile.am M zimwriterfs/README.md M zimwriterfs/configure.ac 3 files changed, 21 insertions(+), 49 deletions(-) git pull ssh://gerrit.wikimedia.org:29418/openzim refs/changes/38/295038/1 diff --git a/zimwriterfs/Makefile.am b/zimwriterfs/Makefile.am index 308149d..e52da96 100644 --- a/zimwriterfs/Makefile.am +++ b/zimwriterfs/Makefile.am @@ -1,5 +1,3 @@ AUTOMAKE_OPTIONS=subdir-objects bin_PROGRAMS=zimwriterfs zimwriterfs_SOURCES= zimwriterfs.cpp gumbo/utf8.c gumbo/string_buffer.c gumbo/parser.c gumbo/error.c gumbo/string_piece.c gumbo/tag.c gumbo/vector.c gumbo/tokenizer.c gumbo/util.c gumbo/char_ref.c gumbo/attribute.c -zimwriterfs_CXXFLAGS=$(LIBZIM_CFLAGS) $(LIBLZMA_CFLAGS) $(LIBZ_CFLAGS) $(LIBMAGIC_CFLAGS) $(LIBPTHREAD_CFLAGS) $(CFLAGS) $(CXXFLAGS) -zimwriterfs_LDFLAGS=$(LIBZIM_LDFLAGS) $(LIBLZMA_LDFLAGS) $(LIBZ_LDFLAGS) $(LIBMAGIC_LDFLAGS) $(LIBPTHREAD_LDFLAGS) \ No newline at end of file diff --git a/zimwriterfs/README.md b/zimwriterfs/README.md index 797f6fb..db8a01d 100644 --- a/zimwriterfs/README.md +++ b/zimwriterfs/README.md @@ -29,10 +29,16 @@ packaged), resp. for the mimeType detection * libz (http://www.zlib.net/), resp. for unpack compressed HTML files +On Debian, you can ensure these are installed with: +``` +sudo apt-get install liblzma-dev libmagic-dev zlib1g-dev +cd ../zimlib && ./autogen.sh && ./configure && make && cd ../zimwriterfs +``` + Once the dependencies are in place, to build: ``` ./autogen.sh -./configure +./configure CXXFLAGS=-I../zimlib/include LDFLAGS=-L../zimlib/src/.libs make ``` diff --git a/zimwriterfs/configure.ac b/zimwriterfs/configure.ac index 5a01142..89294fa 100644 --- a/zimwriterfs/configure.ac +++ b/zimwriterfs/configure.ac @@ -33,70 +33,38 @@ AC_MSG_ERROR([[cannot find pkg-config]]) fi -# Check if the liblzma is available +# Set up CXXFLAGS/LDFLAGS and ensure they are substituted +AC_ARG_VAR(CXXFLAGS, [C++ compiler flags]) +AC_ARG_VAR(LDFLAGS, linker flags) +CXXFLAGS="-O3 -Igumbo $CXXFLAGS" + +# Check if the liblzma library is available AC_CHECK_HEADER([lzma.h],, [AC_MSG_ERROR([[cannot find lzma header]])]) AC_CHECK_LIB([lzma], [lzma_version_string],, [AC_MSG_ERROR([[cannot find lzma]])]) -# Check if the libzim is available +# Check if the libzim library is available AC_CHECK_HEADER([zim/zim.h],, [AC_MSG_ERROR([[cannot find libzim header]])]) AC_CHECK_LIB([zim], [zim_MD5Init],, [AC_MSG_ERROR([[cannot find libzim]])]) -# Check if the libmagic is available +# Check if the libz library is available +AC_CHECK_HEADER([zlib.h],, [AC_MSG_ERROR([[cannot find libz header]])]) +AC_CHECK_LIB([z], [deflate],, [AC_MSG_ERROR([[cannot find libz]])]) + +# Check if the libmagic library is available AC_CHECK_HEADER([magic.h],, [AC_MSG_ERROR([[cannot find libmagic header]])]) AC_CHECK_LIB([magic], [magic_file],, [AC_MSG_ERROR([[cannot find libmagic]])]) -# Check if the libpthread is available +# Check if the libpthread library is available AC_CHECK_HEADER([pthread.h],, [AC_MSG_ERROR([[cannot find libpthread header]])]) AC_CHECK_LIB([pthread], [pthread_exit],, [AC_MSG_ERROR([[cannot find libpthread]])]) -# Set current language to C++ -AC_LANG(C++) - # Check the existence of stat64 (to handle file >2GB) in the libc AC_CHECK_FUNCS([stat64]) - -# cxxflags -CXXFLAGS="-O3 -Igumbo $CXXFLAGS" -CFLAGS="-O3 -std=gnu99 -std=c99" - -# liblzma -LIBLZMA_CFLAGS="" -LIBLZMA_LDFLAGS=" -llzma" - -# libzim -LIBZIM_CFLAGS="" -LIBZIM_LDFLAGS=" -lzim" - -# libz -LIBZ_CFLAGS="" -LIBZ_LDFLAGS=" -lz" - -# libmagic -LIBMAGIC_CFLAGS="" -LIBMAGIC_LDFLAGS=" -lmagic" - -# libpthread -LIBPTHREAD_CFLAGS="" -LIBPTHREAD_LDFLAGS=" -lpthread" AC_DEFINE_UNQUOTED(CLUSTER_CACHE_SIZE, 16, [set zim cluster cache size to number of cached chunks]) AC_DEFINE_UNQUOTED(DIRENT_CACHE_SIZE, 512, [set zim dirent cache size to number of cached chunks]) AC_DEFINE_UNQUOTED(LZMA_MEMORY_SIZE, 128, [set lzma uncompress memory size to number of MB]) AC_DEFINE(ENABLE_LZMA, [1], [defined if lzma compression is enabled]) - -# export variables -AC_SUBST(CXXFLAGS) -AC_SUBST(CFLAGS) -AC_SUBST(LIBLZMA_CFLAGS) -AC_SUBST(LIBLZMA_LDFLAGS) -AC_SUBST(LIBZIM_CFLAGS) -AC_SUBST(LIBZIM_LDFLAGS) -AC_SUBST(LIBZ_CFLAGS) -AC_SUBST(LIBZ_LDFLAGS) -AC_SUBST(LIBMAGIC_CFLAGS) -AC_SUBST(LIBMAGIC_LDFLAGS) -AC_SUBST(LIBPTHREAD_CFLAGS) -AC_SUBST(LIBPTHREAD_LDFLAGS) # Configure the output files AC_CONFIG_FILES([ @@ -104,4 +72,4 @@ ]) AC_PROG_INSTALL -AC_OUTPUT \ No newline at end of file +AC_OUTPUT -- To view, visit https://gerrit.wikimedia.org/r/295038 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: newchange Gerrit-Change-Id: Ic0f93e19eb3e07d86c115bd15f47ef
[MediaWiki-commits] [Gerrit] Convert READMEs to markdown. - change (openzim)
Cscott has uploaded a new change for review. https://gerrit.wikimedia.org/r/295035 Change subject: Convert READMEs to markdown. .. Convert READMEs to markdown. Change-Id: I596bec7ef70bd0a7d45ba67d4fa313d93c861890 --- A README.md D zimlib/README A zimlib/README.md D zimreader-java/README A zimreader-java/README.md D zimreader/README A zimreader/README.md A zimwriterdb/README.md R zimwriterfs/README.md 9 files changed, 92 insertions(+), 61 deletions(-) git pull ssh://gerrit.wikimedia.org:29418/openzim refs/changes/35/295035/1 diff --git a/README.md b/README.md new file mode 100644 index 000..c5fee8e --- /dev/null +++ b/README.md @@ -0,0 +1,11 @@ +This OpenZIM project repository contains the sources for: +* [zimlib](./zimlib#readme): A library for reading and writing ZIM files. +* [zimwriterfs](./zimwriterfs#readme): A tool for creating ZIM files based on + contents on a local filesystem. +* [zimreader-java](./zimreader-java#readme): A ZIM reader written in Java. + +Old code, provided for archive purposes: +* [zimreader](./zimreader#readme): An old reader for the Zeno file format, the + predecessor to the ZIM file format. +* [zimwriterdb](./zimwriterdb#readme): Demonstrates how to write a ZIM file + based on data in a database. diff --git a/zimlib/README b/zimlib/README deleted file mode 100644 index e12d9c0..000 --- a/zimlib/README +++ /dev/null @@ -1,6 +0,0 @@ -The zimlib is the standard implementation of the ZIM specification. It -is a library which implements the read and write method for ZIM -files. The zimlib is released under the GPLv2 license terms. Use the -zimlib in your own software - like reader applications - to make them -ZIM-capable without the need having to dig too much into the ZIM file -format. diff --git a/zimlib/README.md b/zimlib/README.md new file mode 100644 index 000..590041d --- /dev/null +++ b/zimlib/README.md @@ -0,0 +1,13 @@ +zimlib +-- + +The `zimlib` library is the standard implementation of the ZIM +specification. It is a library which implements read and write +methods for ZIM files. + +Use the zimlib in your own software --- for example, reader +applications --- to make them ZIM-capable without the need having to +dig too much into the ZIM file format. + +The `zimlib` library is released under the GPLv2 license +terms. diff --git a/zimreader-java/README b/zimreader-java/README deleted file mode 100644 index 973e7b2..000 --- a/zimreader-java/README +++ /dev/null @@ -1,46 +0,0 @@ - -ZIMReader in Java -= - -This is a port of the ZIMReader in Java. One -of the aims of this project is to enable mobile -users developing on Android, J2ME and other -platforms to use ZIM files and build offline -Wikipedia readers. - -I'll soon add a javadoc, in the mean time -you can go through the comments that I have -provided in the source code. Also, try running -the example ZIMTest.java. - -This code was built on Java 1.6 and has not -been tested on previous versions. However, I'll -do that soon on previous ones as well. In the -next release, I intend to provide an Ant file. - -If you find any bugs, please report them to - or visit the -IRC channel #openzim on Freenode and ping -'gremmachook'. - -This library is licensed under the LGPL v3.0 -license. However, I understand that sometimes -licensing can be a problem for you. I would be -happy to provide a alternate lesser permissive -license if the need be. - -Found this library useful? Drop in a mail, I -love to hear feedback. - -Before this ends, I'd like to thank Lasse Collin -, who maintains the -Tukaani project, for his port of XZ in Java, -without which it wouldn't have been possible for -me to write this library. - - --- Arunesh Mathur - - - - diff --git a/zimreader-java/README.md b/zimreader-java/README.md new file mode 100644 index 000..abce858 --- /dev/null +++ b/zimreader-java/README.md @@ -0,0 +1,42 @@ +ZIMReader in Java += + +This is a port of the ZIMReader in Java. One +of the aims of this project is to enable mobile +users developing on Android, J2ME and other +platforms to use ZIM files and build offline +Wikipedia readers. + +I'll soon add a javadoc, in the mean time +you can go through the comments that I have +provided in the source code. Also, try running +the example ZIMTest.java. + +This code was built on Java 1.6 and has not +been tested on previous versions. However, I'll +do that soon on previous ones as well. In the +next release, I intend to provide an Ant file. + +If you find any bugs, please report them to + or visit the +IRC channel #openzim on Freenode and ping +'gremmachook'. + +This library is licensed under the LGPL v3.0 +license. However, I understand that sometimes +licensing can be a pr
[MediaWiki-commits] [Gerrit] Minor wording and typo fixes to the zimwriterfs README. - change (openzim)
Cscott has uploaded a new change for review. https://gerrit.wikimedia.org/r/295032 Change subject: Minor wording and typo fixes to the zimwriterfs README. .. Minor wording and typo fixes to the zimwriterfs README. Change-Id: Ia408f755b38f079f3063a6f0bc02ae95519f5e1c --- M zimwriterfs/README 1 file changed, 7 insertions(+), 7 deletions(-) git pull ssh://gerrit.wikimedia.org:29418/openzim refs/changes/32/295032/1 diff --git a/zimwriterfs/README b/zimwriterfs/README index 91ea637..811d4a8 100644 --- a/zimwriterfs/README +++ b/zimwriterfs/README @@ -1,15 +1,15 @@ -zimwriterfs is a console tool to create ZIM (http://www.openzim.org) -files from a localy stored directory containing a "self-sufficient" -HTML content (with pictures, javascript, stylesheets). The result will +`zimwriterfs` is a console tool to create [ZIM](http://www.openzim.org) +files from a locally-stored directory containing "self-sufficient" +HTML content (with pictures, javascript, and stylesheets). The result will contain all the files of the local directory compressed and merged in the ZIM file. Nothing more, nothing less. The generated file can be -open with a ZIM reader, Kiwix (http://www.kiwix.org) for example, but -you have other one (http://openzim.org/wiki/ZIM_Readers). +opened with a ZIM reader; [Kiwix](http://www.kiwix.org) is one example, but +there are [others](http://openzim.org/wiki/ZIM_Readers). -zimwriterfs works for now only on POSIX compatible systems, you simply +`zimwriterfs` works for now only on POSIX-compatible systems, you simply need to compile it and run it. The software does not need a lot of resources, but if you create a pretty big ZIM files, then it could -take a wile to complete. +take a while to complete. GNU/Linux compilation - -- To view, visit https://gerrit.wikimedia.org/r/295032 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: newchange Gerrit-Change-Id: Ia408f755b38f079f3063a6f0bc02ae95519f5e1c Gerrit-PatchSet: 1 Gerrit-Project: openzim Gerrit-Branch: master Gerrit-Owner: Cscott ___ MediaWiki-commits mailing list MediaWiki-commits@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] Bump src to 67816adf for deploy. - change (mediawiki...deploy)
Cscott has uploaded a new change for review. https://gerrit.wikimedia.org/r/289550 Change subject: Bump src to 67816adf for deploy. .. Bump src to 67816adf for deploy. Change-Id: If030b47cc2ca7268c18e06cc7ad31ff95e348e03 --- M src 1 file changed, 0 insertions(+), 0 deletions(-) git pull ssh://gerrit.wikimedia.org:29418/mediawiki/services/parsoid/deploy refs/changes/50/289550/1 diff --git a/src b/src index b692ba9..67816ad 16 --- a/src +++ b/src -Subproject commit b692ba97df11a20978c155e2316578aac2ac973d +Subproject commit 67816adfb2528e5ff45c7a27e9500e7bd7da0f96 -- To view, visit https://gerrit.wikimedia.org/r/289550 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: newchange Gerrit-Change-Id: If030b47cc2ca7268c18e06cc7ad31ff95e348e03 Gerrit-PatchSet: 1 Gerrit-Project: mediawiki/services/parsoid/deploy Gerrit-Branch: master Gerrit-Owner: Cscott ___ MediaWiki-commits mailing list MediaWiki-commits@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] Updating to latest masters - change (mediawiki...ocg-collection)
Cscott has uploaded a new change for review. https://gerrit.wikimedia.org/r/287749 Change subject: Updating to latest masters .. Updating to latest masters Deployed new service (ac2dff089f66fd5f6d50d7e7cd046cfd5558ac59): * Update npm dependencies; switch to `babybird` Promise implementation. Deployed new bundler (e33963029ff99f63a54fae78515b742375d88a3c): * Update npm dependencies; switch to `babybird` Promise implementation. Deployed new latexer (544a26ae19eb0e0c0ae9bea98e24630b8ca7982b): * Update npm dependencies; switch to `babybird` Promise implementation. Deployed new texter (df26683c689eb282dba55b0f287fd71307c4a2e2): * Update npm dependencies; switch to `babybird` Promise implementation. Change-Id: If609c5fbb4de77d338b2427e668eb8c71cd33044 --- M mw-ocg-bundler M mw-ocg-latexer M mw-ocg-service M mw-ocg-texter 4 files changed, 0 insertions(+), 0 deletions(-) git pull ssh://gerrit.wikimedia.org:29418/mediawiki/services/ocg-collection refs/changes/49/287749/1 diff --git a/mw-ocg-bundler b/mw-ocg-bundler index d1b9c6b..e339630 16 --- a/mw-ocg-bundler +++ b/mw-ocg-bundler -Subproject commit d1b9c6b1e73766d056a3a80773da1e2cd38bd851 +Subproject commit e33963029ff99f63a54fae78515b742375d88a3c diff --git a/mw-ocg-latexer b/mw-ocg-latexer index 77b0b9d..544a26a 16 --- a/mw-ocg-latexer +++ b/mw-ocg-latexer -Subproject commit 77b0b9d45bf15021a7b2e79a8bee3f528e2e5943 +Subproject commit 544a26ae19eb0e0c0ae9bea98e24630b8ca7982b diff --git a/mw-ocg-service b/mw-ocg-service index fc936c8..ac2dff0 16 --- a/mw-ocg-service +++ b/mw-ocg-service -Subproject commit fc936c8ad7893d606ff67e5fa2fe4f62e755e5fe +Subproject commit ac2dff089f66fd5f6d50d7e7cd046cfd5558ac59 diff --git a/mw-ocg-texter b/mw-ocg-texter index 6b6ef44..df26683 16 --- a/mw-ocg-texter +++ b/mw-ocg-texter -Subproject commit 6b6ef440cb03c16d79c76e123044ee8e48b9c6b8 +Subproject commit df26683c689eb282dba55b0f287fd71307c4a2e2 -- To view, visit https://gerrit.wikimedia.org/r/287749 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: newchange Gerrit-Change-Id: If609c5fbb4de77d338b2427e668eb8c71cd33044 Gerrit-PatchSet: 1 Gerrit-Project: mediawiki/services/ocg-collection Gerrit-Branch: master Gerrit-Owner: Cscott ___ MediaWiki-commits mailing list MediaWiki-commits@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] Merge branch 'master' into wmf-deploy - change (mediawiki...ocg-collection)
Cscott has submitted this change and it was merged. Change subject: Merge branch 'master' into wmf-deploy .. Merge branch 'master' into wmf-deploy Change-Id: Ifca7bde984e7ea1b21f9d3a1565675735cabea96 --- 0 files changed, 0 insertions(+), 0 deletions(-) Approvals: Cscott: Verified; Looks good to me, approved -- To view, visit https://gerrit.wikimedia.org/r/287751 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: merged Gerrit-Change-Id: Ifca7bde984e7ea1b21f9d3a1565675735cabea96 Gerrit-PatchSet: 1 Gerrit-Project: mediawiki/services/ocg-collection Gerrit-Branch: wmf-deploy Gerrit-Owner: Cscott Gerrit-Reviewer: Cscott ___ MediaWiki-commits mailing list MediaWiki-commits@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] Updating to latest masters - change (mediawiki...ocg-collection)
Cscott has submitted this change and it was merged. Change subject: Updating to latest masters .. Updating to latest masters Deployed new service (ac2dff089f66fd5f6d50d7e7cd046cfd5558ac59): * Update npm dependencies; switch to `babybird` Promise implementation. Deployed new bundler (e33963029ff99f63a54fae78515b742375d88a3c): * Update npm dependencies; switch to `babybird` Promise implementation. Deployed new latexer (544a26ae19eb0e0c0ae9bea98e24630b8ca7982b): * Update npm dependencies; switch to `babybird` Promise implementation. Deployed new texter (df26683c689eb282dba55b0f287fd71307c4a2e2): * Update npm dependencies; switch to `babybird` Promise implementation. Change-Id: If609c5fbb4de77d338b2427e668eb8c71cd33044 --- M mw-ocg-bundler M mw-ocg-latexer M mw-ocg-service M mw-ocg-texter 4 files changed, 0 insertions(+), 0 deletions(-) Approvals: Cscott: Verified; Looks good to me, approved diff --git a/mw-ocg-bundler b/mw-ocg-bundler index d1b9c6b..e339630 16 --- a/mw-ocg-bundler +++ b/mw-ocg-bundler -Subproject commit d1b9c6b1e73766d056a3a80773da1e2cd38bd851 +Subproject commit e33963029ff99f63a54fae78515b742375d88a3c diff --git a/mw-ocg-latexer b/mw-ocg-latexer index 77b0b9d..544a26a 16 --- a/mw-ocg-latexer +++ b/mw-ocg-latexer -Subproject commit 77b0b9d45bf15021a7b2e79a8bee3f528e2e5943 +Subproject commit 544a26ae19eb0e0c0ae9bea98e24630b8ca7982b diff --git a/mw-ocg-service b/mw-ocg-service index fc936c8..ac2dff0 16 --- a/mw-ocg-service +++ b/mw-ocg-service -Subproject commit fc936c8ad7893d606ff67e5fa2fe4f62e755e5fe +Subproject commit ac2dff089f66fd5f6d50d7e7cd046cfd5558ac59 diff --git a/mw-ocg-texter b/mw-ocg-texter index 6b6ef44..df26683 16 --- a/mw-ocg-texter +++ b/mw-ocg-texter -Subproject commit 6b6ef440cb03c16d79c76e123044ee8e48b9c6b8 +Subproject commit df26683c689eb282dba55b0f287fd71307c4a2e2 -- To view, visit https://gerrit.wikimedia.org/r/287749 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: merged Gerrit-Change-Id: If609c5fbb4de77d338b2427e668eb8c71cd33044 Gerrit-PatchSet: 1 Gerrit-Project: mediawiki/services/ocg-collection Gerrit-Branch: master Gerrit-Owner: Cscott Gerrit-Reviewer: Cscott ___ MediaWiki-commits mailing list MediaWiki-commits@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] Merge branch 'master' into wmf-deploy - change (mediawiki...ocg-collection)
Cscott has uploaded a new change for review. https://gerrit.wikimedia.org/r/287751 Change subject: Merge branch 'master' into wmf-deploy .. Merge branch 'master' into wmf-deploy Change-Id: Ifca7bde984e7ea1b21f9d3a1565675735cabea96 --- 0 files changed, 0 insertions(+), 0 deletions(-) git pull ssh://gerrit.wikimedia.org:29418/mediawiki/services/ocg-collection refs/changes/51/287751/1 -- To view, visit https://gerrit.wikimedia.org/r/287751 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: newchange Gerrit-Change-Id: Ifca7bde984e7ea1b21f9d3a1565675735cabea96 Gerrit-PatchSet: 1 Gerrit-Project: mediawiki/services/ocg-collection Gerrit-Branch: wmf-deploy Gerrit-Owner: Cscott ___ MediaWiki-commits mailing list MediaWiki-commits@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] Add option to emit `blessed`-style markup. - change (mediawiki...text_renderer)
Cscott has uploaded a new change for review.
https://gerrit.wikimedia.org/r/287261
Change subject: Add option to emit `blessed`-style markup.
..
Add option to emit `blessed`-style markup.
Change-Id: I75eb7fbc0cec5119855a996670792a56d710cee5
---
M bin/mw-ocg-texter
M lib/index.js
2 files changed, 57 insertions(+), 11 deletions(-)
git pull
ssh://gerrit.wikimedia.org:29418/mediawiki/extensions/Collection/OfflineContentGenerator/text_renderer
refs/changes/61/287261/1
diff --git a/bin/mw-ocg-texter b/bin/mw-ocg-texter
index 346005e..82c9b08 100755
--- a/bin/mw-ocg-texter
+++ b/bin/mw-ocg-texter
@@ -13,6 +13,8 @@
'Save text to the given ', null)
.option('-w, --no-wrap',
'Don\'t wrap; use newlines only to separate paragraphs')
+ .option('-b, --blessed',
+ 'Add `blessed` markup to output')
.option('-r, --no-refs',
'Suppress references (and other metadata sections)')
.option('-v, --verbose',
@@ -73,6 +75,7 @@
var options = {
bundle: bundlefile,
noWrap: !program.wrap,
+ blessed: program.blessed,
noRefs: !program.refs,
debug: !!program.debug,
output: program.output,
diff --git a/lib/index.js b/lib/index.js
index 630e096..db875c0 100644
--- a/lib/index.js
+++ b/lib/index.js
@@ -31,7 +31,7 @@
// Convert plain text (with HTML whitespace semantics) to an appropriately
// simplified string
-var textEscape = function(str) {
+var textEscape = function(str, options) {
// Compress multiple newlines (and use unix-style newlines exclusively).
str = str.replace(/\r\n?/g, '\n').replace(/\n\n+/g, '\n');
// Trim leading and trailing newlines for consistent output.
@@ -40,6 +40,11 @@
// XXX Only in en locales?
str = str.replace(/[\u201C\u201D]/g, '"');
str = str.replace(/[\u2018\u2019]/g, "'");
+ if (options.blessed) {
+ str = str.replace(/[{}]/g, function(c) {
+ return (c==='{') ? '{open}' : '{close}';
+ });
+ }
return str;
};
@@ -155,7 +160,13 @@
};
Formatter.prototype.writeHeading = function(level, heading) {
this.paragraphBreak();
+ if (this.options.blessed) {
+ this.write('{left}{bold}');
+ }
this.write(heading.trim());
+ if (this.options.blessed) {
+ this.write('{/bold}{/left}');
+ }
this.paragraphBreak();
};
Formatter.prototype.indent = function(tag) {
@@ -236,7 +247,10 @@
this.usedLanguages = new Set();
this.listInfo = { depth: 0 };
};
-
+// Helper function to pass along options.
+Visitor.prototype.textEscape = function(str) {
+ return textEscape(str, this.options);
+};
// Helper function -- collect all text from the children of `node` as
// HTML non-block/TeX non-paragraph content. Invoke `f` with the result,
// suitable for inclusion in a TeX non-paragraph context.
@@ -321,7 +335,7 @@
case node.TEXT_NODE:
case node.CDATA_SECTION_NODE:
var text = this.format.state.pre ? node.data :
- textEscape(node.data);
+ this.textEscape(node.data);
if (text) {
this.format.write(text);
}
@@ -357,8 +371,34 @@
};
Visitor.prototype.visitA = function(node) {
- // Ignore the href.
+ var wikiLink = node.getAttribute('rel') === 'mw:WikiLink';
+ if (this.options.blessed) {
+ this.format.write('{blue-fg}');
+ if (wikiLink) {
+ var href = node.getAttribute('href')
+ .replace(/^.+\//, '').replace('}', '%7D');
+ this.format.write('{link ' + href + '}');
+ }
+ }
this.visitChildren(node);
+ if (this.options.blessed) {
+ if (wikiLink) {
+ this.format.write('{/link}');
+ }
+ this.format.write('{/blue-fg}');
+ }
+};
+
+Visitor.prototype.visitB = function(node) {
+ if (this.options.blessed) { this.format.write('{bold}'); }
+ this.visitChildren(node);
+ if (this.options.blessed) { this.format.write('{/bold}'); }
+};
+
+Visitor.prototype.visitU = function(node) {
+ if (this.options.blessed) { this.format.write('{underline}'); }
+ this.visitChildren(node);
+ if (this.options.blessed) { this.format.write('{/underline}'); }
};
Visitor.prototype.visitP = function(node) {
@@ -438,7 +478,7 @@
}));
} else {
// Oh, well, just print it w/o subscripting.
- this.format.write(textEscape(contents));
+ this.format.write(this.textEscape(contents));
}
});
};
@@ -455,14 +495,1
[MediaWiki-commits] [Gerrit] Update npm dependencies; switch to `babybird` Promise implem... - change (mediawiki...zim_renderer)
Cscott has uploaded a new change for review.
https://gerrit.wikimedia.org/r/287262
Change subject: Update npm dependencies; switch to `babybird` Promise
implementation.
..
Update npm dependencies; switch to `babybird` Promise implementation.
Change-Id: I4b55923a08207a8acc7044724f8699b1a4da3c3a
---
M bin/mw-ocg-zimwriter
M lib/db.js
M lib/index.js
M lib/p.js
A lib/promise.js
M package.json
M test/samples.js
7 files changed, 16 insertions(+), 14 deletions(-)
git pull
ssh://gerrit.wikimedia.org:29418/mediawiki/extensions/Collection/OfflineContentGenerator/zim_renderer
refs/changes/62/287262/1
diff --git a/bin/mw-ocg-zimwriter b/bin/mw-ocg-zimwriter
index 3d1393c..f177abc 100755
--- a/bin/mw-ocg-zimwriter
+++ b/bin/mw-ocg-zimwriter
@@ -1,6 +1,6 @@
#!/usr/bin/env node
require('core-js/shim');
-var Promise = require('prfun');
+var Promise = require('../lib/promise.js');
var program = require('commander');
var util = require('util');
diff --git a/lib/db.js b/lib/db.js
index a2570c0..11f27de 100644
--- a/lib/db.js
+++ b/lib/db.js
@@ -1,7 +1,7 @@
// Helpers to create/read key/value mappings in sqlite db.
'use strict';
require('core-js/shim');
-var Promise = require('prfun');
+var Promise = require('./promise.js');
var sqlite3 = require('sqlite3');
diff --git a/lib/index.js b/lib/index.js
index 354e0ef..7ca6ce7 100644
--- a/lib/index.js
+++ b/lib/index.js
@@ -3,7 +3,7 @@
'use strict';
require('core-js/shim');
-var Promise = require('prfun');
+var Promise = require('./promise.js');
var json = require('../package.json');
diff --git a/lib/p.js b/lib/p.js
index 17f0841..ee4e703 100644
--- a/lib/p.js
+++ b/lib/p.js
@@ -1,7 +1,7 @@
// Helpers for promises.
'use strict';
require('core-js/shim');
-var Promise = require('prfun');
+var Promise = require('./promise.js');
var spawn = require('child_process').spawn;
diff --git a/lib/promise.js b/lib/promise.js
new file mode 100644
index 000..2733cf7
--- /dev/null
+++ b/lib/promise.js
@@ -0,0 +1 @@
+module.exports = require('prfun/wrap')(require('babybird'));
diff --git a/package.json b/package.json
index 0357715..a01fe34 100644
--- a/package.json
+++ b/package.json
@@ -27,19 +27,20 @@
"node": ">=0.8"
},
"dependencies": {
-"commander": "~2.8.1",
-"core-js": "~0.9.18",
-"domino": "~1.0.19",
-"prfun": "~2.1.1",
+"babybird": "~0.0.1",
+"commander": "~2.9.0",
+"core-js": "~2.3.0",
+"domino": "~1.0.24",
+"prfun": "~2.1.4",
"readable-stream": "~1.0.33",
"sqlite3": "~3.1.0",
-"tmp": "~0.0.27"
+"tmp": "~0.0.28"
},
"devDependencies": {
-"jscs": "~2.5.0",
-"jshint": "~2.8.0",
-"mocha": "~2.3.3",
-"npm-travis": "~1.0.0"
+"jscs": "~2.11.0",
+"jshint": "~2.9.2",
+"mocha": "~2.4.5",
+"npm-travis": "~1.0.2"
},
"bin": {
"mw-ocg-zimwriter": "./bin/mw-ocg-zimwriter"
diff --git a/test/samples.js b/test/samples.js
index 9042713..f2e5dd5 100644
--- a/test/samples.js
+++ b/test/samples.js
@@ -1,7 +1,7 @@
/* global describe, it */
'use strict';
require('core-js/shim');
-var Promise = require('prfun');
+var Promise = require('../lib/promise.js');
var assert = require('assert');
var fs = require('fs');
--
To view, visit https://gerrit.wikimedia.org/r/287262
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: I4b55923a08207a8acc7044724f8699b1a4da3c3a
Gerrit-PatchSet: 1
Gerrit-Project:
mediawiki/extensions/Collection/OfflineContentGenerator/zim_renderer
Gerrit-Branch: master
Gerrit-Owner: Cscott
___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] Update npm dependencies; switch to `babybird` Promise implem... - change (mediawiki...text_renderer)
Cscott has uploaded a new change for review.
https://gerrit.wikimedia.org/r/287260
Change subject: Update npm dependencies; switch to `babybird` Promise
implementation.
..
Update npm dependencies; switch to `babybird` Promise implementation.
Change-Id: If9d826632bd83df912e670ab2da975afed4bc3c3
---
M bin/mw-ocg-texter
M lib/db.js
M lib/index.js
M lib/p.js
A lib/promise.js
M lib/standalone.js
M package.json
M test/samples.js
8 files changed, 18 insertions(+), 16 deletions(-)
git pull
ssh://gerrit.wikimedia.org:29418/mediawiki/extensions/Collection/OfflineContentGenerator/text_renderer
refs/changes/60/287260/1
diff --git a/bin/mw-ocg-texter b/bin/mw-ocg-texter
index 72cc7f9..346005e 100755
--- a/bin/mw-ocg-texter
+++ b/bin/mw-ocg-texter
@@ -1,6 +1,6 @@
#!/usr/bin/env node
require('core-js/shim');
-var Promise = require('prfun');
+var Promise = require('../lib/promise.js');
var program = require('commander');
var texter = require('../');
diff --git a/lib/db.js b/lib/db.js
index a2570c0..11f27de 100644
--- a/lib/db.js
+++ b/lib/db.js
@@ -1,7 +1,7 @@
// Helpers to create/read key/value mappings in sqlite db.
'use strict';
require('core-js/shim');
-var Promise = require('prfun');
+var Promise = require('./promise.js');
var sqlite3 = require('sqlite3');
diff --git a/lib/index.js b/lib/index.js
index 88f8cdf..630e096 100644
--- a/lib/index.js
+++ b/lib/index.js
@@ -3,7 +3,7 @@
'use strict';
require('core-js/shim');
-var Promise = require('prfun');
+var Promise = require('./promise.js');
var json = require('../package.json');
diff --git a/lib/p.js b/lib/p.js
index 17f0841..ee4e703 100644
--- a/lib/p.js
+++ b/lib/p.js
@@ -1,7 +1,7 @@
// Helpers for promises.
'use strict';
require('core-js/shim');
-var Promise = require('prfun');
+var Promise = require('./promise.js');
var spawn = require('child_process').spawn;
diff --git a/lib/promise.js b/lib/promise.js
new file mode 100644
index 000..2733cf7
--- /dev/null
+++ b/lib/promise.js
@@ -0,0 +1 @@
+module.exports = require('prfun/wrap')(require('babybird'));
diff --git a/lib/standalone.js b/lib/standalone.js
index e14388a..4beefeb 100644
--- a/lib/standalone.js
+++ b/lib/standalone.js
@@ -1,6 +1,6 @@
'use strict';
require('core-js/shim');
-var Promise = require('prfun');
+var Promise = require('./promise.js');
var main = require('./');
var bundler = require('mw-ocg-bundler');
diff --git a/package.json b/package.json
index ac25037..55ab442 100644
--- a/package.json
+++ b/package.json
@@ -23,23 +23,24 @@
],
"license": "GPL-2.0",
"dependencies": {
-"commander": "~2.8.1",
-"core-js": "~0.9.18",
-"domino": "~1.0.19",
+"babybird": "~0.0.1",
+"commander": "~2.9.0",
+"core-js": "~2.3.0",
+"domino": "~1.0.24",
"linewrap": "~0.2.1",
-"prfun": "~2.1.1",
+"prfun": "~2.1.4",
"readable-stream": "~1.0.33",
"sqlite3": "~3.1.0",
-"tmp": "~0.0.27"
+"tmp": "~0.0.28"
},
"optionalDependencies": {
- "mw-ocg-bundler": "~1.3.0"
+ "mw-ocg-bundler": "~1.3.1"
},
"devDependencies": {
-"jscs": "~2.5.0",
-"jshint": "~2.8.0",
-"mocha": "~2.3.3",
-"npm-travis": "~1.0.0"
+"jscs": "~2.11.0",
+"jshint": "~2.9.2",
+"mocha": "~2.4.5",
+"npm-travis": "~1.0.2"
},
"bin": {
"mw-ocg-texter": "./bin/mw-ocg-texter"
diff --git a/test/samples.js b/test/samples.js
index 05cc171..f82db8f 100644
--- a/test/samples.js
+++ b/test/samples.js
@@ -1,7 +1,7 @@
/* global describe, it */
'use strict';
require('core-js/shim');
-var Promise = require('prfun');
+var Promise = require('../lib/promise.js');
var assert = require('assert');
var fs = require('fs');
--
To view, visit https://gerrit.wikimedia.org/r/287260
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: If9d826632bd83df912e670ab2da975afed4bc3c3
Gerrit-PatchSet: 1
Gerrit-Project:
mediawiki/extensions/Collection/OfflineContentGenerator/text_renderer
Gerrit-Branch: master
Gerrit-Owner: Cscott
___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] Update npm dependencies; switch to `babybird` Promise implem... - change (mediawiki...latex_renderer)
Cscott has uploaded a new change for review.
https://gerrit.wikimedia.org/r/287259
Change subject: Update npm dependencies; switch to `babybird` Promise
implementation.
..
Update npm dependencies; switch to `babybird` Promise implementation.
Change-Id: Ied0210a1f2cee28dacbdc4a61185a36d0a8e9fb3
---
M .jscsrc
M bin/mw-ocg-latexer
M lib/db.js
M lib/index.js
M lib/p.js
A lib/promise.js
M package.json
M test/samples.js
8 files changed, 18 insertions(+), 15 deletions(-)
git pull
ssh://gerrit.wikimedia.org:29418/mediawiki/extensions/Collection/OfflineContentGenerator/latex_renderer
refs/changes/59/287259/1
diff --git a/.jscsrc b/.jscsrc
index 4621905..01196e4 100644
--- a/.jscsrc
+++ b/.jscsrc
@@ -12,6 +12,7 @@
"try",
"catch"
],
+ "requireEarlyReturn": false,
"validateIndentation": "\t",
"maximumLineLength": null,
"disallowMultipleVarDecl": null,
diff --git a/bin/mw-ocg-latexer b/bin/mw-ocg-latexer
index e0ea6a6..1cb3bca 100755
--- a/bin/mw-ocg-latexer
+++ b/bin/mw-ocg-latexer
@@ -1,6 +1,6 @@
#!/usr/bin/env node
require('core-js/shim');
-var Promise = require('prfun');
+var Promise = require('../lib/promise.js');
if (/v0\.[0-7]\./.test(process.version)) {
console.error('Node version', process.version, 'is not supported.');
diff --git a/lib/db.js b/lib/db.js
index a2570c0..11f27de 100644
--- a/lib/db.js
+++ b/lib/db.js
@@ -1,7 +1,7 @@
// Helpers to create/read key/value mappings in sqlite db.
'use strict';
require('core-js/shim');
-var Promise = require('prfun');
+var Promise = require('./promise.js');
var sqlite3 = require('sqlite3');
diff --git a/lib/index.js b/lib/index.js
index 83ef366..5eacdb8 100644
--- a/lib/index.js
+++ b/lib/index.js
@@ -2,7 +2,7 @@
// -
'use strict';
require('core-js/shim');
-var Promise = require('prfun');
+var Promise = require('./promise.js');
var json = require('../package.json');
@@ -1371,7 +1371,7 @@
//
// This code is largely asynchronous. It chains promises together
// to manage the concurrency without callback hell. It uses the
-// ES6 Promise api, from the npm `core-js` and `prfun` packages.
+// ES6 Promise api, from the npm `babybird` and `prfun` packages.
// The `prfun` package has a nice implementation of guards, which is
// an easy way to limit the maximum parallelism of a task to ensure
// we don't spam the host with hundreds of tasks at once.
diff --git a/lib/p.js b/lib/p.js
index 17f0841..ee4e703 100644
--- a/lib/p.js
+++ b/lib/p.js
@@ -1,7 +1,7 @@
// Helpers for promises.
'use strict';
require('core-js/shim');
-var Promise = require('prfun');
+var Promise = require('./promise.js');
var spawn = require('child_process').spawn;
diff --git a/lib/promise.js b/lib/promise.js
new file mode 100644
index 000..2733cf7
--- /dev/null
+++ b/lib/promise.js
@@ -0,0 +1 @@
+module.exports = require('prfun/wrap')(require('babybird'));
diff --git a/package.json b/package.json
index 8943364..f7d53db 100644
--- a/package.json
+++ b/package.json
@@ -27,22 +27,23 @@
"node": ">=0.8"
},
"dependencies": {
-"commander": "~2.8.1",
-"core-js": "~0.9.18",
-"domino": "~1.0.19",
+"babybird": "~0.0.1",
+"commander": "~2.9.0",
+"core-js": "~2.3.0",
+"domino": "~1.0.24",
"gammalatex": "~0.0.8",
"icu-bidi": "~0.1.5",
-"prfun": "~2.1.1",
+"prfun": "~2.1.4",
"readable-stream": "~1.0.33",
"sqlite3": "~3.1.0",
"texvcjs": "~0.3.0",
-"tmp": "~0.0.27"
+"tmp": "~0.0.28"
},
"devDependencies": {
-"jscs": "~2.5.0",
-"jshint": "~2.8.0",
-"mocha": "~2.3.3",
-"npm-travis": "~1.0.0"
+"jscs": "~2.11.0",
+"jshint": "~2.9.2",
+"mocha": "~2.4.5",
+"npm-travis": "~1.0.2"
},
"bin": {
"mw-ocg-latexer": "./bin/mw-ocg-latexer"
diff --git a/test/samples.js b/test/samples.js
index f08d61b..085f077 100644
--- a/test/samples.js
+++ b/test/samples.js
@@ -1,7 +1,7 @@
/* global describe, it */
'use strict';
require('core-js/shim');
-var Promise = require('prfun');
+var Promise = require('../lib/promise.js');
var assert = require('assert');
var fs = require('fs');
--
To view, visit https://gerrit.wikimedia.org/r/287259
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: Ied0210a1f2cee28dacbdc4a61185a36d0a8e9fb3
Gerrit-PatchSet: 1
Gerrit-Project:
mediawiki/extensions/Collection/OfflineContentGenerator/latex_renderer
Gerrit-Branch: master
Gerrit-Owner: Cscott
___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] Update npm dependencies; switch to `babybird` Promise implem... - change (mediawiki...OfflineContentGenerator)
Cscott has uploaded a new change for review.
https://gerrit.wikimedia.org/r/287258
Change subject: Update npm dependencies; switch to `babybird` Promise
implementation.
..
Update npm dependencies; switch to `babybird` Promise implementation.
Change-Id: I9330a7801c3ef9dd6f3c6ce5cd5cd47acf3e7ceb
---
M lib/RedisWrapper.js
A lib/promise.js
M lib/threads/backend.js
M lib/threads/frontend.js
M lib/threads/gc.js
M mw-ocg-service.js
M package.json
M scripts/clear-host-cache.js
M scripts/clear-queue.js
M scripts/clear-time-range.js
M scripts/loadtest.js
M scripts/run-garbage-collect.js
12 files changed, 23 insertions(+), 19 deletions(-)
git pull
ssh://gerrit.wikimedia.org:29418/mediawiki/extensions/Collection/OfflineContentGenerator
refs/changes/58/287258/1
diff --git a/lib/RedisWrapper.js b/lib/RedisWrapper.js
index 4425c4a..1467a82 100644
--- a/lib/RedisWrapper.js
+++ b/lib/RedisWrapper.js
@@ -1,7 +1,7 @@
'use strict';
require('core-js/shim');
-var Promise = require('prfun');
+var Promise = require('./promise.js');
var util = require('util');
var events = require('events');
diff --git a/lib/promise.js b/lib/promise.js
new file mode 100644
index 000..2733cf7
--- /dev/null
+++ b/lib/promise.js
@@ -0,0 +1 @@
+module.exports = require('prfun/wrap')(require('babybird'));
diff --git a/lib/threads/backend.js b/lib/threads/backend.js
index e94081e..6019ada 100644
--- a/lib/threads/backend.js
+++ b/lib/threads/backend.js
@@ -23,7 +23,7 @@
*/
require('core-js/shim');
-var Promise = require('prfun');
+var Promise = require('../promise.js');
var childProcess = require('child_process');
var eh = require('../errorhelper.js');
diff --git a/lib/threads/frontend.js b/lib/threads/frontend.js
index 2968428..25894e0 100644
--- a/lib/threads/frontend.js
+++ b/lib/threads/frontend.js
@@ -23,7 +23,7 @@
*/
require('core-js/shim');
-var Promise = require('prfun');
+var Promise = require('../promise.js');
var Busboy = require('busboy');
var childProcess = require('child_process');
diff --git a/lib/threads/gc.js b/lib/threads/gc.js
index 59a7e2f..1436351 100644
--- a/lib/threads/gc.js
+++ b/lib/threads/gc.js
@@ -31,7 +31,7 @@
*/
require('core-js/shim');
-var Promise = require('prfun');
+var Promise = require('../promise.js');
var fs = require('fs');
var path = require('path');
diff --git a/mw-ocg-service.js b/mw-ocg-service.js
index 2aeaa6f..69e5d1f 100755
--- a/mw-ocg-service.js
+++ b/mw-ocg-service.js
@@ -33,7 +33,7 @@
*/
require('core-js/shim');
-var Promise = require('prfun');
+var Promise = require('./lib/promise.js');
var cli = require('./lib/cli.js');
var cluster = require('cluster');
diff --git a/package.json b/package.json
index 4715730..5b97188 100644
--- a/package.json
+++ b/package.json
@@ -8,12 +8,13 @@
},
"bin": "./mw-ocg-service.js",
"scripts": {
-"test": "npm run lint-no-0.8",
"lint": "npm run jshint && npm run jscs",
"lint-no-0.8": "node -e
'process.exit(/v0[.][0-8][.]/.test(process.version) ? 0 : 1)' || npm run lint",
"jshint": "jshint .",
"jscs": "jscs .",
-"jscs-fix": "jscs --fix ."
+"jscs-fix": "jscs --fix .",
+"test": "npm run lint-no-0.8",
+"travis": "npm-travis --remote gerrit
wikimedia/mediawiki-extensions-Collection-OfflineContentGenerator"
},
"repository": {
"type": "git",
@@ -24,14 +25,15 @@
],
"license": "GPL-2.0",
"dependencies": {
+"babybird": "~0.0.1",
"bunyan": "~1.4.0",
"busboy": "0.0.12",
-"commander": "~2.8.1",
-"core-js": "~0.9.18",
+"commander": "~2.9.0",
+"core-js": "~2.3.0",
"mime": "~1.2.11",
"mkdirp": "~0.5.0",
"mv": "~2.1.1",
-"prfun": "~2.1.1",
+"prfun": "~2.1.4",
"redis": "~0.12.1",
"rimraf": "~2.4.3",
"sprintf-js": "0.0.7"
@@ -43,9 +45,10 @@
"hiredis": "~0.4.0"
},
"devDependencies": {
-"jscs": "~2.5.0",
-"jshint": "~2.8.0",
-"request": "~2.64.0",
-"mw-ocg-bundler":
"wikimedia/mediawiki-extensions-Collection-OfflineContentGenerator-bundler#master"
+"jscs": "~2.11.0",
+"jshint": "~2.9.2",
+"mw-ocg-bundler":
"wikimedia/mediawiki-extensions-Collection-OfflineContentGenerator-bundler#master",
+"npm-travis": "~1.0.2",
+"request": "~2.72.0"
}
}
diff --git a/scripts/clear-host-cache.js b/scripts/clear-host-cache.js
index e1cf4b7..87caee2 100755
--- a/scripts/clear-host-cache.js
+++ b/scripts/clear-host-cache.js
@@ -44,7 +44,7 @@
*/
require('core-js/shim');
-var Promise = require('prfun');
+var Promise = require('../lib/promise.js');
var cli = require('../lib/cli.js');
var commander = require('commander');
diff --git a/scripts/clear-queue.js b/scripts/clear-queue.js
index 65c6733..5ee2088 100755
--- a/scripts/clear-queue.js
+++ b/scripts/clear-queue.js
@@ -28,7 +28,7 @@
*/
require('core-js/shim');
-var Promise = require('prfun');
+var Promise = require('../lib/promise.js');
[MediaWiki-commits] [Gerrit] Update npm dependencies; switch to `babybird` Promise implem... - change (mediawiki...bundler)
Cscott has uploaded a new change for review.
https://gerrit.wikimedia.org/r/287142
Change subject: Update npm dependencies; switch to `babybird` Promise
implementation.
..
Update npm dependencies; switch to `babybird` Promise implementation.
Change-Id: I7c0d6fcdd15f534a5654bf0c73a34fd575228859
---
M .jscsrc
M CHANGELOG.md
M bin/mw-ocg-bundler
M lib/api.js
M lib/attribution.js
M lib/authors.js
M lib/db.js
M lib/html.js
M lib/image.js
M lib/index.js
M lib/metabook.js
M lib/modules.js
M lib/p.js
M lib/parsoid.js
A lib/promise.js
M lib/retry-request.js
M lib/revisions.js
M lib/siteinfo.js
M package.json
M test/samples.js
20 files changed, 30 insertions(+), 26 deletions(-)
git pull
ssh://gerrit.wikimedia.org:29418/mediawiki/extensions/Collection/OfflineContentGenerator/bundler
refs/changes/42/287142/1
diff --git a/.jscsrc b/.jscsrc
index 19f37d9..e1fa233 100644
--- a/.jscsrc
+++ b/.jscsrc
@@ -3,6 +3,7 @@
// allow some slop to account for legacy code
"validateQuoteMarks": { "mark": "'", "escape": true },
"requireCapitalizedComments": { "allExcept": ["jshint", "global"] },
+ "requireEarlyReturn": false,
"validateIndentation": "\t",
"maximumLineLength": null,
"disallowMultipleVarDecl": null,
diff --git a/CHANGELOG.md b/CHANGELOG.md
index e530216..c3e4df7 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -1,4 +1,5 @@
# mw-ocg-bundler x.x.x (not yet released)
+* Update npm dependencies; switch to `babybird` Promise implementation.
# mw-ocg-bundler 1.3.1 (2016-03-31)
* Use underscores in titles sent to the REST v1/Parsoid v3 API.
diff --git a/bin/mw-ocg-bundler b/bin/mw-ocg-bundler
index 0e91017..dc84661 100755
--- a/bin/mw-ocg-bundler
+++ b/bin/mw-ocg-bundler
@@ -1,7 +1,7 @@
#!/usr/bin/env node
'use strict';
require('core-js/shim'); // For Map, endsWith, etc.
-var Promise = require('prfun');
+var Promise = require('../lib/promise.js');
var program = require('commander');
var bundler = require('../');
diff --git a/lib/api.js b/lib/api.js
index 2fe2f43..b984927 100644
--- a/lib/api.js
+++ b/lib/api.js
@@ -1,7 +1,7 @@
// Helpers for making mediawiki api requests.
'use strict';
require('core-js/shim');
-var Promise = require('prfun');
+var Promise = require('./promise.js');
var headers = require('./headers');
var querystring = require('querystring');
diff --git a/lib/attribution.js b/lib/attribution.js
index 4c13dc2..7ded566 100644
--- a/lib/attribution.js
+++ b/lib/attribution.js
@@ -1,6 +1,6 @@
'use strict';
require('core-js/shim'); // Map/Set/Promise support
-var Promise = require('prfun');
+var Promise = require('./promise.js');
var util = require('util');
diff --git a/lib/authors.js b/lib/authors.js
index 4d6ea7f..296ee9b 100644
--- a/lib/authors.js
+++ b/lib/authors.js
@@ -1,7 +1,7 @@
// Obtain authorship information for wiki articles.
'use strict';
require('core-js/shim');
-var Promise = require('prfun');
+var Promise = require('./promise.js');
var util = require('util');
diff --git a/lib/db.js b/lib/db.js
index a2570c0..11f27de 100644
--- a/lib/db.js
+++ b/lib/db.js
@@ -1,7 +1,7 @@
// Helpers to create/read key/value mappings in sqlite db.
'use strict';
require('core-js/shim');
-var Promise = require('prfun');
+var Promise = require('./promise.js');
var sqlite3 = require('sqlite3');
diff --git a/lib/html.js b/lib/html.js
index 4433321..a8120cc 100644
--- a/lib/html.js
+++ b/lib/html.js
@@ -1,7 +1,7 @@
// Generate contents of html.db
'use strict';
require('core-js/shim');
-var Promise = require('prfun');
+var Promise = require('./promise.js');
var util = require('util');
diff --git a/lib/image.js b/lib/image.js
index ef0884d..6457cfb 100644
--- a/lib/image.js
+++ b/lib/image.js
@@ -1,7 +1,7 @@
// Fetch images and metadata about them.
'use strict';
require('core-js/shim');
-var Promise = require('prfun');
+var Promise = require('./promise.js');
var fs = require('fs');
var headers = require('./headers');
diff --git a/lib/index.js b/lib/index.js
index 5b661fd..b3907b5 100644
--- a/lib/index.js
+++ b/lib/index.js
@@ -1,6 +1,6 @@
'use strict';
require('core-js/shim'); // Map/Set/Promise support
-var Promise = require('prfun');
+var Promise = require('./promise.js');
var json = require('../package.json');
diff --git a/lib/metabook.js b/lib/metabook.js
index ba3f64a..d70b0f1 100644
--- a/lib/metabook.js
+++ b/lib/metabook.js
@@ -1,7 +1,7 @@
/** Create a new metabook structure, or fixup a broken/incomplete one. */
'use strict';
require('core-js/shim');
-var Promise = require('prfun');
+var Promise = require('./promise.js');
var url = require('url');
diff --git a/lib/modules.js b/lib/modules.js
index 4d6ab7f..4e6836b 100644
--- a/lib/modules.js
+++ b/lib/modules.js
@@ -1,7 +1,7 @@
// Fetch css and js modules for the articles
'use strict';
require('core-js/shim');
-var Promise = require('prfun');
+var Promise = requir
[MediaWiki-commits] [Gerrit] WIP: Warn when editors use the deprecated hack. - change (mediawiki/core)
Cscott has uploaded a new change for review.
https://gerrit.wikimedia.org/r/286928
Change subject: WIP: Warn when editors use the deprecated hack.
..
WIP: Warn when editors use the deprecated hack.
Pages using self-closing tags in violation of the HTML5 parsing specification
will have their templates/pages added to a new tracking category. After these
uses are fixed, we will change the sanitizer to be consistent with the
HTML5 parsing spec.
Note that we already don't allow the hack if tidy is disabled; it
is converted to "". It may be worth making the no-tidy and with-tidy
behavior consistent.
Change-Id: Ie1cf3aa40d5483bf395ece539f0240b694ff04ab
---
M includes/Sanitizer.php
1 file changed, 11 insertions(+), 1 deletion(-)
git pull ssh://gerrit.wikimedia.org:29418/mediawiki/core
refs/changes/28/286928/1
diff --git a/includes/Sanitizer.php b/includes/Sanitizer.php
index d321e9f..f9eaede 100644
--- a/includes/Sanitizer.php
+++ b/includes/Sanitizer.php
@@ -453,7 +453,7 @@
* @return string
*/
public static function removeHTMLtags( $text, $processCallback = null,
- $args = [], $extratags = [], $removetags = []
+ $args = [], $extratags = [], $removetags = [], $warnCallback =
null
) {
extract( self::getRecognizedTagData( $extratags, $removetags )
);
@@ -604,6 +604,16 @@
call_user_func_array(
$processCallback, [ &$params, $args ] );
}
+ if ( $brace == '/>' && !(
isset( $htmlsingle[$t] ) || isset( $htmlsingleonly[$t] ) ) ) {
+ // Eventually we'll
just remove the self-closing
+ // slash, in order to
be consistent with HTML5
+ // semantics.
+ // $brace = '>';
+ // For now, let's just
warn authors to clean up.
+ if ( is_callable(
$warnCallback ) ) {
+
call_user_func_array( $warnCallback, [ 'deprecated-self-close-category' ] );
+ }
+ }
if ( !Sanitizer::validateTag(
$params, $t ) ) {
$badtag = true;
}
--
To view, visit https://gerrit.wikimedia.org/r/286928
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: Ie1cf3aa40d5483bf395ece539f0240b694ff04ab
Gerrit-PatchSet: 1
Gerrit-Project: mediawiki/core
Gerrit-Branch: master
Gerrit-Owner: Cscott
___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] Merge branch 'master' into wmf-deploy - change (mediawiki...ocg-collection)
Cscott has submitted this change and it was merged. Change subject: Merge branch 'master' into wmf-deploy .. Merge branch 'master' into wmf-deploy Change-Id: Ic4fce9e087502c22d697cc200fcb8d4bcc8cc0f4 --- 0 files changed, 0 insertions(+), 0 deletions(-) Approvals: Cscott: Verified; Looks good to me, approved -- To view, visit https://gerrit.wikimedia.org/r/286526 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: merged Gerrit-Change-Id: Ic4fce9e087502c22d697cc200fcb8d4bcc8cc0f4 Gerrit-PatchSet: 1 Gerrit-Project: mediawiki/services/ocg-collection Gerrit-Branch: wmf-deploy Gerrit-Owner: Cscott Gerrit-Reviewer: Cscott ___ MediaWiki-commits mailing list MediaWiki-commits@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] Updating to latest masters - change (mediawiki...ocg-collection)
Cscott has submitted this change and it was merged. Change subject: Updating to latest masters .. Updating to latest masters Deployed new bundler (d1b9c6b1e73766d056a3a80773da1e2cd38bd851): * Fix `--domainname` option when bundling a collection from the command line. Deployed new latexer (77b0b9d45bf15021a7b2e79a8bee3f528e2e5943): * Add `sawNonSpace` internal flag to avoid problems when forcing a newline. * Skip images inside headings. Change-Id: I67599b38089e34cf4178420b688a6a47811654aa --- M mw-ocg-bundler M mw-ocg-latexer 2 files changed, 0 insertions(+), 0 deletions(-) Approvals: Cscott: Verified; Looks good to me, approved diff --git a/mw-ocg-bundler b/mw-ocg-bundler index c4f4c4b..d1b9c6b 16 --- a/mw-ocg-bundler +++ b/mw-ocg-bundler -Subproject commit c4f4c4b212aada9646dbc113b6db352f51408f75 +Subproject commit d1b9c6b1e73766d056a3a80773da1e2cd38bd851 diff --git a/mw-ocg-latexer b/mw-ocg-latexer index 3ff99a5..77b0b9d 16 --- a/mw-ocg-latexer +++ b/mw-ocg-latexer -Subproject commit 3ff99a570acf7b8f3a8af3c47f02b02ac5aad01c +Subproject commit 77b0b9d45bf15021a7b2e79a8bee3f528e2e5943 -- To view, visit https://gerrit.wikimedia.org/r/286525 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: merged Gerrit-Change-Id: I67599b38089e34cf4178420b688a6a47811654aa Gerrit-PatchSet: 1 Gerrit-Project: mediawiki/services/ocg-collection Gerrit-Branch: master Gerrit-Owner: Cscott Gerrit-Reviewer: Cscott ___ MediaWiki-commits mailing list MediaWiki-commits@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] Merge branch 'master' into wmf-deploy - change (mediawiki...ocg-collection)
Cscott has uploaded a new change for review. https://gerrit.wikimedia.org/r/286526 Change subject: Merge branch 'master' into wmf-deploy .. Merge branch 'master' into wmf-deploy Change-Id: Ic4fce9e087502c22d697cc200fcb8d4bcc8cc0f4 --- 0 files changed, 0 insertions(+), 0 deletions(-) git pull ssh://gerrit.wikimedia.org:29418/mediawiki/services/ocg-collection refs/changes/26/286526/1 -- To view, visit https://gerrit.wikimedia.org/r/286526 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: newchange Gerrit-Change-Id: Ic4fce9e087502c22d697cc200fcb8d4bcc8cc0f4 Gerrit-PatchSet: 1 Gerrit-Project: mediawiki/services/ocg-collection Gerrit-Branch: wmf-deploy Gerrit-Owner: Cscott ___ MediaWiki-commits mailing list MediaWiki-commits@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] Updating to latest masters - change (mediawiki...ocg-collection)
Cscott has uploaded a new change for review. https://gerrit.wikimedia.org/r/286525 Change subject: Updating to latest masters .. Updating to latest masters Deployed new bundler (d1b9c6b1e73766d056a3a80773da1e2cd38bd851): * Fix `--domainname` option when bundling a collection from the command line. Deployed new latexer (77b0b9d45bf15021a7b2e79a8bee3f528e2e5943): * Add `sawNonSpace` internal flag to avoid problems when forcing a newline. * Skip images inside headings. Change-Id: I67599b38089e34cf4178420b688a6a47811654aa --- M mw-ocg-bundler M mw-ocg-latexer 2 files changed, 0 insertions(+), 0 deletions(-) git pull ssh://gerrit.wikimedia.org:29418/mediawiki/services/ocg-collection refs/changes/25/286525/1 diff --git a/mw-ocg-bundler b/mw-ocg-bundler index c4f4c4b..d1b9c6b 16 --- a/mw-ocg-bundler +++ b/mw-ocg-bundler -Subproject commit c4f4c4b212aada9646dbc113b6db352f51408f75 +Subproject commit d1b9c6b1e73766d056a3a80773da1e2cd38bd851 diff --git a/mw-ocg-latexer b/mw-ocg-latexer index 3ff99a5..77b0b9d 16 --- a/mw-ocg-latexer +++ b/mw-ocg-latexer -Subproject commit 3ff99a570acf7b8f3a8af3c47f02b02ac5aad01c +Subproject commit 77b0b9d45bf15021a7b2e79a8bee3f528e2e5943 -- To view, visit https://gerrit.wikimedia.org/r/286525 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I67599b38089e34cf4178420b688a6a47811654aa Gerrit-PatchSet: 1 Gerrit-Project: mediawiki/services/ocg-collection Gerrit-Branch: master Gerrit-Owner: Cscott ___ MediaWiki-commits mailing list MediaWiki-commits@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] Skip images inside headings. - change (mediawiki...latex_renderer)
Cscott has uploaded a new change for review.
https://gerrit.wikimedia.org/r/286188
Change subject: Skip images inside headings.
..
Skip images inside headings.
Change-Id: Id6d75fa5c9769618ef1ab74ed3bd060d2171f9b8
---
M lib/index.js
1 file changed, 4 insertions(+), 0 deletions(-)
git pull
ssh://gerrit.wikimedia.org:29418/mediawiki/extensions/Collection/OfflineContentGenerator/latex_renderer
refs/changes/88/286188/1
diff --git a/lib/index.js b/lib/index.js
index 35c8f34..e840f29 100644
--- a/lib/index.js
+++ b/lib/index.js
@@ -1163,6 +1163,10 @@
console.error('Skipping', resource, '(float)');
return;
}
+ if (this.inHeading) { // XXX Work around issues with inline images.
+ console.error('Skipping', resource, '(heading)');
+ return;
+ }
// Find page number for PDFs/DjVus (this is a parsoid workaround).
if (/[.](pdf|djvu)$/i.test(resource)) {
var page = '1';
--
To view, visit https://gerrit.wikimedia.org/r/286188
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: Id6d75fa5c9769618ef1ab74ed3bd060d2171f9b8
Gerrit-PatchSet: 1
Gerrit-Project:
mediawiki/extensions/Collection/OfflineContentGenerator/latex_renderer
Gerrit-Branch: master
Gerrit-Owner: Cscott
___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] Add `sawNonSpace` internal flag to avoid problems when using... - change (mediawiki...latex_renderer)
Cscott has uploaded a new change for review.
https://gerrit.wikimedia.org/r/286187
Change subject: Add `sawNonSpace` internal flag to avoid problems when using
`\\`.
..
Add `sawNonSpace` internal flag to avoid problems when using `\\`.
Often the `\newline` command errors out with
"! LaTeX Error: There's no line here to end."
if the only preceding content has been whitespace.
Change-Id: Ia3402913d55d9882046656dea763451cb800ee0b
---
M lib/index.js
A samples/firstaid.zip
M test/samples.js
3 files changed, 8 insertions(+), 3 deletions(-)
git pull
ssh://gerrit.wikimedia.org:29418/mediawiki/extensions/Collection/OfflineContentGenerator/latex_renderer
refs/changes/87/286187/1
diff --git a/lib/index.js b/lib/index.js
index 1e9db0d..35c8f34 100644
--- a/lib/index.js
+++ b/lib/index.js
@@ -292,6 +292,7 @@
this.stack = []; // Active inline decorations.
this.pos = 0;
this.newEnv = this.newLine = this.newPara = this.startPara = true;
+ this.sawNonSpace = false;
// Protect ToC content from certain markup.
this.inToc = 0;
// Allow list-type environments to prevent a default-direction change,
@@ -468,6 +469,7 @@
*/
Formatter.prototype.resetSOL = function() {
this.newEnv = this.newLine = this.newPara = true;
+ this.sawNonSpace = false;
};
/** Flush the formatter buffers and indicate that this is a good place to
* change the text directionality, if necessary. */
@@ -488,15 +490,17 @@
this.envBreak();
this._writeRaw('\n');
this.newPara = this.newLine = true;
+ this.sawNonSpace = false;
// This is a good place to change the xetex default bidi context dir.
this.dirBreak(); this.startPara = true;
};
/** Add a hard line break (only allowed within a paragraph). */
Formatter.prototype.lineBreak = function() {
- if (this.newLine) { return; }
+ if (this.newLine || !this.sawNonSpace) { return; }
this.envBreak();
this._writeRaw('\n');
this.newLine = true;
+ this.sawNonSpace = false;
};
// Internal: bookkeeping for decorations.
Formatter.prototype._addDecoration = function(d) {
@@ -514,6 +518,7 @@
text = text.replace(/\s+/g, ' '); // Remove newlines.
this.buffer.push(text);
this.pos += text.length;
+ if (/\S/.test(text)) { this.sawNonSpace = true; }
};
/**
* Add some decorated text. If `text` is omitted, this is a raw or block
@@ -583,7 +588,7 @@
});
this.restoreInlineStack(stack);
this.envBreak();
- this.newLine = this.newPara = true;
+ this.newLine = this.newPara = true; this.sawNonSpace = false;
};
// Helpers for directionality.
Formatter.prototype.switchDir = function(dir, opts) {
diff --git a/samples/firstaid.zip b/samples/firstaid.zip
new file mode 100644
index 000..9313d17
--- /dev/null
+++ b/samples/firstaid.zip
Binary files differ
diff --git a/test/samples.js b/test/samples.js
index 0fa118e..f08d61b 100644
--- a/test/samples.js
+++ b/test/samples.js
@@ -34,7 +34,7 @@
describe('Basic crash test', function() {
var hasXeLaTeX = checkExecutable('xelatex');
var hasJpegtran = checkExecutable('jpegtran');
- ['tao.zip', 'hurricanes.zip', 'malayalam.zip', 'multiwiki.zip',
'papier.zip', 'titlecrash.zip', 'us.zip', 'jabug.zip', 'bug68854.zip',
'1988.zip', 'set.zip', 'bug71185.zip', 'url-in-toc.zip', 'allah.zip',
'tibetan.zip', 'lao.zip',
'khmer.zip','pashto.zip','nepali.zip','newari.zip','maithili.zip'].forEach(function(bundle)
{
+ ['tao.zip', 'hurricanes.zip', 'malayalam.zip', 'multiwiki.zip',
'papier.zip', 'titlecrash.zip', 'us.zip', 'jabug.zip', 'bug68854.zip',
'1988.zip', 'set.zip', 'bug71185.zip', 'url-in-toc.zip', 'allah.zip',
'tibetan.zip', 'lao.zip',
'khmer.zip','pashto.zip','nepali.zip','newari.zip','maithili.zip','firstaid.zip'].forEach(function(bundle)
{
describe(bundle, function() {
var dest = hasXeLaTeX ? 'pdf' : 'tex';
it('should compile to ' + dest, function() {
--
To view, visit https://gerrit.wikimedia.org/r/286187
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: Ia3402913d55d9882046656dea763451cb800ee0b
Gerrit-PatchSet: 1
Gerrit-Project:
mediawiki/extensions/Collection/OfflineContentGenerator/latex_renderer
Gerrit-Branch: master
Gerrit-Owner: Cscott
___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] Fix JSAPI. - change (mediawiki...parsoid)
Cscott has uploaded a new change for review.
https://gerrit.wikimedia.org/r/286175
Change subject: Fix JSAPI.
..
Fix JSAPI.
Follow-up to 39e10aea.
Change-Id: Ic30a47bcf1ec18267ee310b9aa9ab88e8f31207c
---
M lib/html2wt/WikitextSerializer.js
M lib/jsapi.js
M tests/mocha/jsapi.js
3 files changed, 34 insertions(+), 22 deletions(-)
git pull ssh://gerrit.wikimedia.org:29418/mediawiki/services/parsoid
refs/changes/75/286175/1
diff --git a/lib/html2wt/WikitextSerializer.js
b/lib/html2wt/WikitextSerializer.js
index 53e57bd..3be7c0e 100644
--- a/lib/html2wt/WikitextSerializer.js
+++ b/lib/html2wt/WikitextSerializer.js
@@ -1251,6 +1251,7 @@
*/
WSP.serializeDOM = Promise.method(function(body, selserMode) {
console.assert(DU.isBody(body), 'Expected a body node.');
+ // XXX: Can we come up with a better assertion mesage here?
console.assert(this.env.page.editedDoc, 'Should be set.');
this.logType = selserMode ? "trace/selser" : "trace/wts";
diff --git a/lib/jsapi.js b/lib/jsapi.js
index 8d8fece..00296bc 100644
--- a/lib/jsapi.js
+++ b/lib/jsapi.js
@@ -19,6 +19,14 @@
var Promise = require('./utils/promise.js');
var WikitextSerializer =
require('./html2wt/WikitextSerializer.js').WikitextSerializer;
+// jsapi stores node data directly in DOM attributes, for easy visibility.
+var getDataMw = function(node) {
+ return DU.getJSONAttribute(node, 'data-mw', {});
+};
+var setDataMw = function(node, val) {
+ DU.setJSONAttribute(node, 'data-mw', val);
+};
+
// WTS helper
var wts = function(env, nodes) {
var body;
@@ -32,7 +40,10 @@
body.appendChild(nodes[i].cloneNode(true));
}
}
- return (new WikitextSerializer({ env: env })).serializeDOM(body);
+ var serializer = new WikitextSerializer({ env: env });
+ DU.visitDOM(body, DU.loadDataAttribs, true);
+ env.page.editedDoc = body.ownerDocument;
+ return serializer.serializeDOM(body);
};
// toString helper
@@ -189,7 +200,7 @@
// each Transclusion node could represent multiple templates.
var parent = this;
var result = [];
- DU.getDataMw(node).parts.forEach(function(part, i) {
+ getDataMw(node).parts.forEach(function(part, i) {
if (part.template) {
result.push(new PTemplate(parent.pdoc, parent,
node, i));
}
@@ -526,8 +537,8 @@
get: function() { return this.node.ownerDocument; },
},
dataMw: {
- get: function() { return DU.getDataMw(this.node); },
- set: function(v) { DU.setDataMw(this.node, v); this.update(); },
+ get: function() { return getDataMw(this.node); },
+ set: function(v) { setDataMw(this.node, v); this.update(); },
},
/**
* Internal helper: enumerate all PNodeLists contained within this node.
diff --git a/tests/mocha/jsapi.js b/tests/mocha/jsapi.js
index cae631c..65dc63b 100644
--- a/tests/mocha/jsapi.js
+++ b/tests/mocha/jsapi.js
@@ -37,7 +37,7 @@
pdoc.document.should.have.property('outerHTML');
});
});
- it.skip('filters out templates', function() {
+ it('filters out templates', function() {
var text = "I has a template! {{foo|bar|baz|eggs=spam}} See
it?\n";
var pdoc, templates, template;
return Parsoid.parse(text, { pdoc: true }).then(function(_pdoc)
{
@@ -68,7 +68,7 @@
wt.should.equal('spam');
});
});
- it.skip('filters templates, recursively', function() {
+ it('filters templates, recursively', function() {
var text = "{{foo|{{bar}}={{baz|{{spam}}";
return Parsoid.parse(text, { pdoc: true }).then(function(pdoc) {
var templates = pdoc.filterTemplates();
@@ -77,7 +77,7 @@
templates.length.should.equal(3);
});
});
- it.skip('filters templates, non-recursively', function() {
+ it('filters templates, non-recursively', function() {
var text = "{{foo|this {{includes a|template";
var foo;
return Parsoid.parse(text, { pdoc: true }).then(function(pdoc) {
@@ -94,7 +94,7 @@
wt.should.equal('template');
});
});
- it.skip('is easy to mutate templates', function() {
+ it('is easy to mutate templates', function() {
var text = "{{cleanup}} '''Foo''' is a [[bar]].
{{uncategorized}}";
return Parsoid.parse(text, { pdoc: true }).then(function(pdoc) {
pdoc.filterTemplates().forEach(function(template) {
@@ -113,7 +113,7 @@
});
describe('Further examples o
[MediaWiki-commits] [Gerrit] Fix debugging instructions in README. - change (mediawiki...latex_renderer)
Cscott has uploaded a new change for review.
https://gerrit.wikimedia.org/r/286074
Change subject: Fix debugging instructions in README.
..
Fix debugging instructions in README.
Change-Id: I0da19a4fa036116cf7c8516f233bd73620057c03
---
M README.md
M bin/mw-ocg-latexer
2 files changed, 2 insertions(+), 2 deletions(-)
git pull
ssh://gerrit.wikimedia.org:29418/mediawiki/extensions/Collection/OfflineContentGenerator/latex_renderer
refs/changes/74/286074/1
diff --git a/README.md b/README.md
index 87d65c0..a429daa 100644
--- a/README.md
+++ b/README.md
@@ -67,7 +67,7 @@
For debugging, preserving the XeTeX output is often useful:
```
-bin/mw-ocg-latexer -o out.tex us.zip
+bin/mw-ocg-latexer -l -D -o out.tex us.zip
TEXINPUTS=tex/: xelatex out.tex
```
diff --git a/bin/mw-ocg-latexer b/bin/mw-ocg-latexer
index 011eb81..e0ea6a6 100755
--- a/bin/mw-ocg-latexer
+++ b/bin/mw-ocg-latexer
@@ -35,7 +35,7 @@
.option('-l, --latex',
'Output LaTeX source instead of PDF')
.option('-D, --debug',
- 'Turn on debugging features (eg, full stack traces on
exceptions)')
+ 'Turn on debugging features (eg, preserve temp dir,
full stack traces on exceptions)')
.option('-T, --temporary-directory ',
'Use for temporaries, not $TMPDIR or /tmp', null);
--
To view, visit https://gerrit.wikimedia.org/r/286074
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: I0da19a4fa036116cf7c8516f233bd73620057c03
Gerrit-PatchSet: 1
Gerrit-Project:
mediawiki/extensions/Collection/OfflineContentGenerator/latex_renderer
Gerrit-Branch: master
Gerrit-Owner: Cscott
___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] Fix `--domainname` option when bundling a collection from th... - change (mediawiki...bundler)
Cscott has uploaded a new change for review.
https://gerrit.wikimedia.org/r/286073
Change subject: Fix `--domainname` option when bundling a collection from the
command line.
..
Fix `--domainname` option when bundling a collection from the command line.
Change-Id: I70c7c4e628f3e0482699f07a4e1ddd13aca236e1
---
M lib/metabook.js
1 file changed, 4 insertions(+), 2 deletions(-)
git pull
ssh://gerrit.wikimedia.org:29418/mediawiki/extensions/Collection/OfflineContentGenerator/bundler
refs/changes/73/286073/1
diff --git a/lib/metabook.js b/lib/metabook.js
index 300884d..ba3f64a 100644
--- a/lib/metabook.js
+++ b/lib/metabook.js
@@ -199,9 +199,11 @@
// Return a promise for a metabook from the given collection.
var metabookFromCollection = function(collection, options, log) {
+ // Work around commander issues with a flag named 'domain'
+ var domain = options.domain || options.domainname;
// First do the prefix-guessing, etc.
return metabookFromArticles(
- [{ prefix: options.prefix, domain: options.domain, title:
collection }],
+ [{ prefix: options.prefix, domain: domain, title: collection }],
options
).then(function(m) {
// Use this to fetch the source of the collection page.
@@ -223,7 +225,7 @@
// confirm rel="mw:WikiLink".
var title =
parsoidResult._resolve(links[i].getAttribute('href'));
articles.push({
- prefix: options.prefix, domain: options.domain,
title: title,
+ prefix: options.prefix, domain: domain, title:
title,
});
}
return metabookFromArticles(articles, options);
--
To view, visit https://gerrit.wikimedia.org/r/286073
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: I70c7c4e628f3e0482699f07a4e1ddd13aca236e1
Gerrit-PatchSet: 1
Gerrit-Project: mediawiki/extensions/Collection/OfflineContentGenerator/bundler
Gerrit-Branch: master
Gerrit-Owner: Cscott
___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] Decommission ocg1003. - change (operations/puppet)
Cscott has uploaded a new change for review.
https://gerrit.wikimedia.org/r/286070
Change subject: Decommission ocg1003.
..
Decommission ocg1003.
Uses the new decommission flag landed in T120077.
Bug: T84723
Change-Id: Ie167559b8d2714448f2cf39bcbe48ca618613938
---
M manifests/site.pp
M modules/ocg/manifests/init.pp
M modules/ocg/templates/mw-ocg-service.js.erb
3 files changed, 8 insertions(+), 0 deletions(-)
git pull ssh://gerrit.wikimedia.org:29418/operations/puppet
refs/changes/70/286070/1
diff --git a/manifests/site.pp b/manifests/site.pp
index b3393ec..a240843 100644
--- a/manifests/site.pp
+++ b/manifests/site.pp
@@ -2359,6 +2359,9 @@
# Offline Content Generator
node /^ocg100[123]\.eqiad\.wmnet$/ {
+if ($::fqdn == 'ocg1003.eqiad.wmnet') {
+ $ocg_decommission = true
+}
role ocg
}
diff --git a/modules/ocg/manifests/init.pp b/modules/ocg/manifests/init.pp
index 502fbf2..5fcf138 100644
--- a/modules/ocg/manifests/init.pp
+++ b/modules/ocg/manifests/init.pp
@@ -12,6 +12,7 @@
class ocg (
$host_name = $::fqdn,
+$decommission = $::ocg_decommission,
$service_port = 8000,
$redis_host = 'localhost',
$redis_port = 6379,
diff --git a/modules/ocg/templates/mw-ocg-service.js.erb
b/modules/ocg/templates/mw-ocg-service.js.erb
index 76411ec..3fe90b4 100644
--- a/modules/ocg/templates/mw-ocg-service.js.erb
+++ b/modules/ocg/templates/mw-ocg-service.js.erb
@@ -13,6 +13,10 @@
config.coordinator.hostname = "<%= @host_name %>";
+ <%- if @decommission == true %>
+ config.coordinator.decommission = "<%= @host_name %>";
+ <%- end -%>
+
config.redis.host = "<%= @redis_host %>";
config.redis.port = <%= @redis_port %>;
config.redis.password = "<%= @redis_password %>";
--
To view, visit https://gerrit.wikimedia.org/r/286070
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: Ie167559b8d2714448f2cf39bcbe48ca618613938
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Cscott
___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] Use FQDN for OCG hostnames. - change (operations/puppet)
Cscott has uploaded a new change for review. https://gerrit.wikimedia.org/r/286068 Change subject: Use FQDN for OCG hostnames. .. Use FQDN for OCG hostnames. Bug: T133864 Change-Id: Ib8a75b8194fc0832d87df918ac9fe09ffb40a67e --- M modules/ocg/manifests/init.pp 1 file changed, 1 insertion(+), 1 deletion(-) git pull ssh://gerrit.wikimedia.org:29418/operations/puppet refs/changes/68/286068/1 diff --git a/modules/ocg/manifests/init.pp b/modules/ocg/manifests/init.pp index 646220d..502fbf2 100644 --- a/modules/ocg/manifests/init.pp +++ b/modules/ocg/manifests/init.pp @@ -11,7 +11,7 @@ # class ocg ( -$host_name = $::hostname, +$host_name = $::fqdn, $service_port = 8000, $redis_host = 'localhost', $redis_port = 6379, -- To view, visit https://gerrit.wikimedia.org/r/286068 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: newchange Gerrit-Change-Id: Ib8a75b8194fc0832d87df918ac9fe09ffb40a67e Gerrit-PatchSet: 1 Gerrit-Project: operations/puppet Gerrit-Branch: production Gerrit-Owner: Cscott ___ MediaWiki-commits mailing list MediaWiki-commits@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] Merge branch 'master' into wmf-deploy - change (mediawiki...ocg-collection)
Cscott has submitted this change and it was merged. Change subject: Merge branch 'master' into wmf-deploy .. Merge branch 'master' into wmf-deploy Change-Id: Id9fcf4d356a07ca7c646ac9f5411b4b80016146d --- 0 files changed, 0 insertions(+), 0 deletions(-) Approvals: Cscott: Verified; Looks good to me, approved -- To view, visit https://gerrit.wikimedia.org/r/285719 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: merged Gerrit-Change-Id: Id9fcf4d356a07ca7c646ac9f5411b4b80016146d Gerrit-PatchSet: 1 Gerrit-Project: mediawiki/services/ocg-collection Gerrit-Branch: wmf-deploy Gerrit-Owner: Cscott Gerrit-Reviewer: Cscott ___ MediaWiki-commits mailing list MediaWiki-commits@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] Updating to latest masters - change (mediawiki...ocg-collection)
Cscott has submitted this change and it was merged. Change subject: Updating to latest masters .. Updating to latest masters Deployed new service (fc936c8ad7893d606ff67e5fa2fe4f62e755e5fe): * Allow decommissioning OCG hosts. (T120077) * Periodically check for configuration file changes. (Not enabled in default configuration.) * Improve graceful shutdown. Change-Id: Id729c63b7878b55f289915abf81e7df982ad6465 --- M mw-ocg-service 1 file changed, 0 insertions(+), 0 deletions(-) Approvals: Cscott: Verified; Looks good to me, approved diff --git a/mw-ocg-service b/mw-ocg-service index 4442920..fc936c8 16 --- a/mw-ocg-service +++ b/mw-ocg-service -Subproject commit 4442920b712dad67e3ca7c7bf6af4a37a794405d +Subproject commit fc936c8ad7893d606ff67e5fa2fe4f62e755e5fe -- To view, visit https://gerrit.wikimedia.org/r/285717 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: merged Gerrit-Change-Id: Id729c63b7878b55f289915abf81e7df982ad6465 Gerrit-PatchSet: 1 Gerrit-Project: mediawiki/services/ocg-collection Gerrit-Branch: master Gerrit-Owner: Cscott Gerrit-Reviewer: Arlolra Gerrit-Reviewer: Cscott Gerrit-Reviewer: Subramanya Sastry ___ MediaWiki-commits mailing list MediaWiki-commits@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] Merge branch 'master' into wmf-deploy - change (mediawiki...ocg-collection)
Cscott has uploaded a new change for review. https://gerrit.wikimedia.org/r/285719 Change subject: Merge branch 'master' into wmf-deploy .. Merge branch 'master' into wmf-deploy Change-Id: Id9fcf4d356a07ca7c646ac9f5411b4b80016146d --- 0 files changed, 0 insertions(+), 0 deletions(-) git pull ssh://gerrit.wikimedia.org:29418/mediawiki/services/ocg-collection refs/changes/19/285719/1 -- To view, visit https://gerrit.wikimedia.org/r/285719 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: newchange Gerrit-Change-Id: Id9fcf4d356a07ca7c646ac9f5411b4b80016146d Gerrit-PatchSet: 1 Gerrit-Project: mediawiki/services/ocg-collection Gerrit-Branch: wmf-deploy Gerrit-Owner: Cscott ___ MediaWiki-commits mailing list MediaWiki-commits@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] Updating to latest masters - change (mediawiki...ocg-collection)
Cscott has uploaded a new change for review. https://gerrit.wikimedia.org/r/285717 Change subject: Updating to latest masters .. Updating to latest masters Deployed new service (fc936c8ad7893d606ff67e5fa2fe4f62e755e5fe): * Allow decommissioning OCG hosts. (T120077) * Periodically check for configuration file changes. (Not enabled in default configuration.) * Improve graceful shutdown. Change-Id: Id729c63b7878b55f289915abf81e7df982ad6465 --- M mw-ocg-service 1 file changed, 0 insertions(+), 0 deletions(-) git pull ssh://gerrit.wikimedia.org:29418/mediawiki/services/ocg-collection refs/changes/17/285717/1 diff --git a/mw-ocg-service b/mw-ocg-service index 4442920..fc936c8 16 --- a/mw-ocg-service +++ b/mw-ocg-service -Subproject commit 4442920b712dad67e3ca7c7bf6af4a37a794405d +Subproject commit fc936c8ad7893d606ff67e5fa2fe4f62e755e5fe -- To view, visit https://gerrit.wikimedia.org/r/285717 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: newchange Gerrit-Change-Id: Id729c63b7878b55f289915abf81e7df982ad6465 Gerrit-PatchSet: 1 Gerrit-Project: mediawiki/services/ocg-collection Gerrit-Branch: master Gerrit-Owner: Cscott ___ MediaWiki-commits mailing list MediaWiki-commits@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] Merge branch 'master' into wmf-deploy - change (mediawiki...ocg-collection)
Cscott has submitted this change and it was merged. Change subject: Merge branch 'master' into wmf-deploy .. Merge branch 'master' into wmf-deploy Change-Id: I12bc0acc8da662a5eb24c2a0f6a2d2a48aa8c597 --- 0 files changed, 0 insertions(+), 0 deletions(-) Approvals: Cscott: Verified; Looks good to me, approved -- To view, visit https://gerrit.wikimedia.org/r/285277 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: merged Gerrit-Change-Id: I12bc0acc8da662a5eb24c2a0f6a2d2a48aa8c597 Gerrit-PatchSet: 1 Gerrit-Project: mediawiki/services/ocg-collection Gerrit-Branch: wmf-deploy Gerrit-Owner: Cscott Gerrit-Reviewer: Arlolra Gerrit-Reviewer: Cscott Gerrit-Reviewer: Subramanya Sastry ___ MediaWiki-commits mailing list MediaWiki-commits@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] Updating to latest masters - change (mediawiki...ocg-collection)
Cscott has submitted this change and it was merged. Change subject: Updating to latest masters .. Updating to latest masters Deployed new service (4442920b712dad67e3ca7c7bf6af4a37a794405d): * Update README. Deployed new bundler (c4f4c4b212aada9646dbc113b6db352f51408f75): * Don't fallback to rest.wikimedia.org. * Use domain for status messages if prefix is not available. * T124762: Resolve titles even when `$wgArticlePath` involves a query string. * T130685: Send `Accept` header to Parsoid/RESTBase to get `data-mw` attributes inline. Deployed new latexer (3ff99a570acf7b8f3a8af3c47f02b02ac5aad01c): * Update README. Deployed new texter (6b6ef440cb03c16d79c76e123044ee8e48b9c6b8): * Use released version 0.3.3. (No code changes.) Change-Id: I940db60c9ab2a606c4f94e5123cc4fe43926575b --- M mw-ocg-bundler M mw-ocg-latexer M mw-ocg-service M mw-ocg-texter 4 files changed, 0 insertions(+), 0 deletions(-) Approvals: Cscott: Verified; Looks good to me, approved diff --git a/mw-ocg-bundler b/mw-ocg-bundler index 1fec608..c4f4c4b 16 --- a/mw-ocg-bundler +++ b/mw-ocg-bundler -Subproject commit 1fec608a4d55e7bddb115122dd0db0fd4bc88570 +Subproject commit c4f4c4b212aada9646dbc113b6db352f51408f75 diff --git a/mw-ocg-latexer b/mw-ocg-latexer index 2879356..3ff99a5 16 --- a/mw-ocg-latexer +++ b/mw-ocg-latexer -Subproject commit 28793564325d1f5c941ce2d3be0b6cd288ae714f +Subproject commit 3ff99a570acf7b8f3a8af3c47f02b02ac5aad01c diff --git a/mw-ocg-service b/mw-ocg-service index 8a96266..4442920 16 --- a/mw-ocg-service +++ b/mw-ocg-service -Subproject commit 8a9626602a907ef6c2cb825119292933245efede +Subproject commit 4442920b712dad67e3ca7c7bf6af4a37a794405d diff --git a/mw-ocg-texter b/mw-ocg-texter index 73f7b86..6b6ef44 16 --- a/mw-ocg-texter +++ b/mw-ocg-texter -Subproject commit 73f7b86871abff978c818ce3941c52072b0e53f5 +Subproject commit 6b6ef440cb03c16d79c76e123044ee8e48b9c6b8 -- To view, visit https://gerrit.wikimedia.org/r/285270 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: merged Gerrit-Change-Id: I940db60c9ab2a606c4f94e5123cc4fe43926575b Gerrit-PatchSet: 1 Gerrit-Project: mediawiki/services/ocg-collection Gerrit-Branch: master Gerrit-Owner: Cscott Gerrit-Reviewer: Arlolra Gerrit-Reviewer: Cscott Gerrit-Reviewer: Subramanya Sastry ___ MediaWiki-commits mailing list MediaWiki-commits@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] Merge branch 'master' into wmf-deploy - change (mediawiki...ocg-collection)
Cscott has uploaded a new change for review. https://gerrit.wikimedia.org/r/285277 Change subject: Merge branch 'master' into wmf-deploy .. Merge branch 'master' into wmf-deploy Change-Id: I12bc0acc8da662a5eb24c2a0f6a2d2a48aa8c597 --- 0 files changed, 0 insertions(+), 0 deletions(-) git pull ssh://gerrit.wikimedia.org:29418/mediawiki/services/ocg-collection refs/changes/77/285277/1 -- To view, visit https://gerrit.wikimedia.org/r/285277 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I12bc0acc8da662a5eb24c2a0f6a2d2a48aa8c597 Gerrit-PatchSet: 1 Gerrit-Project: mediawiki/services/ocg-collection Gerrit-Branch: wmf-deploy Gerrit-Owner: Cscott ___ MediaWiki-commits mailing list MediaWiki-commits@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] Updating to latest masters - change (mediawiki...ocg-collection)
Cscott has uploaded a new change for review. https://gerrit.wikimedia.org/r/285270 Change subject: Updating to latest masters .. Updating to latest masters Deployed new service (4442920b712dad67e3ca7c7bf6af4a37a794405d): * Update README. Deployed new bundler (c4f4c4b212aada9646dbc113b6db352f51408f75): * Don't fallback to rest.wikimedia.org. * Use domain for status messages if prefix is not available. * T124762: Resolve titles even when `$wgArticlePath` involves a query string. * T130685: Send `Accept` header to Parsoid/RESTBase to get `data-mw` attributes inline. Deployed new latexer (3ff99a570acf7b8f3a8af3c47f02b02ac5aad01c): * Update README. Deployed new texter (6b6ef440cb03c16d79c76e123044ee8e48b9c6b8): * Use released version 0.3.3. (No code changes.) Change-Id: I940db60c9ab2a606c4f94e5123cc4fe43926575b --- M mw-ocg-bundler M mw-ocg-latexer M mw-ocg-service M mw-ocg-texter 4 files changed, 0 insertions(+), 0 deletions(-) git pull ssh://gerrit.wikimedia.org:29418/mediawiki/services/ocg-collection refs/changes/70/285270/1 diff --git a/mw-ocg-bundler b/mw-ocg-bundler index 1fec608..c4f4c4b 16 --- a/mw-ocg-bundler +++ b/mw-ocg-bundler -Subproject commit 1fec608a4d55e7bddb115122dd0db0fd4bc88570 +Subproject commit c4f4c4b212aada9646dbc113b6db352f51408f75 diff --git a/mw-ocg-latexer b/mw-ocg-latexer index 2879356..3ff99a5 16 --- a/mw-ocg-latexer +++ b/mw-ocg-latexer -Subproject commit 28793564325d1f5c941ce2d3be0b6cd288ae714f +Subproject commit 3ff99a570acf7b8f3a8af3c47f02b02ac5aad01c diff --git a/mw-ocg-service b/mw-ocg-service index 8a96266..4442920 16 --- a/mw-ocg-service +++ b/mw-ocg-service -Subproject commit 8a9626602a907ef6c2cb825119292933245efede +Subproject commit 4442920b712dad67e3ca7c7bf6af4a37a794405d diff --git a/mw-ocg-texter b/mw-ocg-texter index 73f7b86..6b6ef44 16 --- a/mw-ocg-texter +++ b/mw-ocg-texter -Subproject commit 73f7b86871abff978c818ce3941c52072b0e53f5 +Subproject commit 6b6ef440cb03c16d79c76e123044ee8e48b9c6b8 -- To view, visit https://gerrit.wikimedia.org/r/285270 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I940db60c9ab2a606c4f94e5123cc4fe43926575b Gerrit-PatchSet: 1 Gerrit-Project: mediawiki/services/ocg-collection Gerrit-Branch: master Gerrit-Owner: Cscott ___ MediaWiki-commits mailing list MediaWiki-commits@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] Add autoload mechanism for user extension code. - change (mediawiki...parsoid)
Cscott has uploaded a new change for review.
https://gerrit.wikimedia.org/r/284738
Change subject: Add autoload mechanism for user extension code.
..
Add autoload mechanism for user extension code.
This parallels the autoload mechanism in mediawiki core. In fact, small
wikis can use the same extensions directory for both core and Parsoid.
Bug: T133320
Change-Id: I8e05d5bfdff873f28a58dead68aaca0e4823cf32
---
M .gitignore
M lib/config/ParsoidConfig.js
2 files changed, 26 insertions(+), 1 deletion(-)
git pull ssh://gerrit.wikimedia.org:29418/mediawiki/services/parsoid
refs/changes/38/284738/1
diff --git a/.gitignore b/.gitignore
index 6abe976..4de2a73 100644
--- a/.gitignore
+++ b/.gitignore
@@ -22,3 +22,5 @@
coverage/
contrib/
node_modules
+
+extensions
diff --git a/lib/config/ParsoidConfig.js b/lib/config/ParsoidConfig.js
index 36ba1f1..9adee69 100644
--- a/lib/config/ParsoidConfig.js
+++ b/lib/config/ParsoidConfig.js
@@ -526,9 +526,32 @@
}
prefix = apiConf.prefix;
- // Give them cite by default.
+ // Give them some default extensions.
if (!Array.isArray(apiConf.extensions)) {
+ // Native support for certain extensions (Cite, etc)
apiConf.extensions = Util.clone(this.defaultNativeExtensions);
+ var grabExtensions = function(dir) {
+ var base = path.join(__dirname, '..', '..',
'extensions');
+ if (dir) { base = path.resolve(base, dir); }
+ try {
+ if (!fs.statSync(base).isDirectory()) { return;
/* not a dir */}
+ } catch (e) { return; /* no file there */ }
+ fs.readdirSync(base).forEach(function(d) {
+ var p = path.join(base, d, 'parsoid'), st;
+ try {
+ st = fs.statSync(p);
+ } catch (e) { return; /* no file there */}
+ if (!st.isDirectory()) { return; /* not a dir
*/}
+ // Make sure that exceptions here are visible
to user.
+ apiConf.extensions.push(require(p));
+ });
+ };
+ /* Include user extensions */
+ grabExtension();
+ /* Include wiki-specific user extensions */
+ // User can specify an alternate directory here, so they can
point
+ // directly at their mediawiki core install if they wish.
+ grabExtension(apiConf.extdir || apiConf.domain);
}
if (this.reverseMwApiMap.has(apiConf.domain)) {
--
To view, visit https://gerrit.wikimedia.org/r/284738
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: I8e05d5bfdff873f28a58dead68aaca0e4823cf32
Gerrit-PatchSet: 1
Gerrit-Project: mediawiki/services/parsoid
Gerrit-Branch: master
Gerrit-Owner: Cscott
___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] Allow decommissioning OCG hosts. - change (mediawiki...OfflineContentGenerator)
Cscott has uploaded a new change for review.
https://gerrit.wikimedia.org/r/284601
Change subject: Allow decommissioning OCG hosts.
..
Allow decommissioning OCG hosts.
A decommissioned host will answer frontend requests but not take new
backend jobs from the redis queue. This allows us to remove its
cached jobs from redis (either proactively or by timing them out),
after which it should receive no further frontend requests and can
be shut down.
For cluster failover, the "backup" hosts on the secondary cluster can
all be listed as decommissioned using a regexp until they are needed.
Bug: T120077
Change-Id: I5b8388ac914a2129f8346ee3f4932455fa14c8e4
---
M defaults.js
M mw-ocg-service.js
2 files changed, 17 insertions(+), 0 deletions(-)
git pull
ssh://gerrit.wikimedia.org:29418/mediawiki/extensions/Collection/OfflineContentGenerator
refs/changes/01/284601/1
diff --git a/defaults.js b/defaults.js
index a97637a..3355ec3 100644
--- a/defaults.js
+++ b/defaults.js
@@ -24,6 +24,11 @@
/** Public hostname of this instance for HTTP GET requests for
locally stored content. */
hostname: null,
+ /** Array of strings or regular expressions naming hosts which
+* should be "decommissioned" --- that is, they should not
+* launch backend threads. */
+ decommission: [],
+
/** How often (in seconds) to check for configuration changes.
* We gracefully shutdown when we get a config change;
presumably
* upstart or some other service runner will restart us.
Defaults
diff --git a/mw-ocg-service.js b/mw-ocg-service.js
index e140ee6..c702c48 100755
--- a/mw-ocg-service.js
+++ b/mw-ocg-service.js
@@ -182,6 +182,18 @@
if (autoThreads === 'auto') {
autoThreads = os.cpus().length;
}
+ // Check: is this host decommissioned? If so, don't launch any
+ // background threads.
+ var host = config.coordinator.hostname || os.hostname();
+ var decommission = config.coordinator.decommission;
+ if (!Array.isArray(decommission)) { decommission = [ decommission ]; }
+ if (decommission.some(function(s) {
+ // s could be a string or a regular expression
+ return host===s || (s.test && s.test(host));
+ })) {
+ console.warn('%s decommissioned; no backend threads launched.',
host);
+ autoThreads = 0;
+ }
for (i = 0; i < autoThreads; i++) {
spawnWorker('backend');
}
--
To view, visit https://gerrit.wikimedia.org/r/284601
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: I5b8388ac914a2129f8346ee3f4932455fa14c8e4
Gerrit-PatchSet: 1
Gerrit-Project: mediawiki/extensions/Collection/OfflineContentGenerator
Gerrit-Branch: master
Gerrit-Owner: Cscott
___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] Improve graceful shutdown. - change (mediawiki...OfflineContentGenerator)
Cscott has uploaded a new change for review.
https://gerrit.wikimedia.org/r/284598
Change subject: Improve graceful shutdown.
..
Improve graceful shutdown.
During a graceful shutdown we don't halt background render tasks or
garbage collections which are in progress; instead waiting politely
for these to finish before shutting down. We also shut down the
front end tasks *last* to minimize downtime for those fetching
cached resources.
Change-Id: Id128e7ad3ba06d4da6302c2c42b8f2c32e58de9b
---
M lib/RedisWrapper.js
M lib/threads/backend.js
M lib/threads/frontend.js
M lib/threads/gc.js
M mw-ocg-service.js
5 files changed, 125 insertions(+), 70 deletions(-)
git pull
ssh://gerrit.wikimedia.org:29418/mediawiki/extensions/Collection/OfflineContentGenerator
refs/changes/98/284598/1
diff --git a/lib/RedisWrapper.js b/lib/RedisWrapper.js
index 36d9b12..323a5a6 100644
--- a/lib/RedisWrapper.js
+++ b/lib/RedisWrapper.js
@@ -86,7 +86,8 @@
/**
* Gracefully closes the connection. Emits 'closed' when complete.
*/
-RedisWrapper.prototype.close = function() {
+RedisWrapper.prototype.close = function(cb) {
+ if (cb) { this.once('closed', cb); }
if (this.client && this.client.connected) {
this.client.quit();
} else {
diff --git a/lib/threads/backend.js b/lib/threads/backend.js
index e3a60d1..8723c09 100644
--- a/lib/threads/backend.js
+++ b/lib/threads/backend.js
@@ -47,6 +47,7 @@
var config = null;
var redisClient = null;
+var stopping = false;
/* === Public Exported Functions === */
/**
@@ -120,16 +121,13 @@
* Starts the backend server
*/
function startBackend() {
- var loop = false;
redisClient.on('closed', function() {
- if (!loop) {
- loop = true;
- console.error(
- 'Backend connection to redis died
unexpectedly.',
- { channel: 'backend.error.fatal' }
- );
- stopBackend(process.exit);
- }
+ if (stopping) return;
+ console.error(
+ 'Backend connection to redis died unexpectedly.',
+ { channel: 'backend.error.fatal' }
+ );
+ stopBackend(process.exit);
});
redisClient.on('opened', getNewItemFromQueue);
redisClient.connect();
@@ -145,8 +143,8 @@
* @param {callback} callbackFunc - Function to call when server successfully
closed
*/
function stopBackend(callbackFunc) {
- redisClient.close();
- callbackFunc();
+ stopping = callbackFunc || function() { };
+ // now we have to wait until the backend is done with its job.
}
/* === Private Functions === */
@@ -155,43 +153,51 @@
* then starts the promise chain to bundle, render, and cleanup.
*/
function getNewItemFromQueue() {
- redisClient.blpop(config.redis.job_queue_name, 0)
+ // Before checking the redis queue, see if this thread should shut down
+ if (stopping) {
+ console.info(
+ 'Backend thread halting as requested.', { channel:
'backend' }
+ );
+ return redisClient.close(stopping);
+ }
+ redisClient.blpop(config.redis.job_queue_name, 3)
.then(function(result) {
// The return, if there is one, will be
[,]
if (!result) {
- throw new BackendError('Redis returned nil when
picking up new job from queue.');
- } else {
- console.info(
- 'Got new job "%s", attempting to get
status details and launching',
- result[1],
- {
- channel: 'backend',
- job: { id: result[1] },
- }
- );
- return
redisClient.hget(config.redis.status_set_name, result[1]);
+ // Timeout. Check our host status, and then
retry.
+ return;
}
- })
- .catch(function(err) {
+ console.info(
+ 'Got new job "%s", attempting to get status
details and launching',
+ result[1],
+ {
+ channel: 'backend',
+ job: { id: result[1] },
+ }
+ );
+ return redisC
[MediaWiki-commits] [Gerrit] Periodically check for configuration file changes. - change (mediawiki...OfflineContentGenerator)
Cscott has uploaded a new change for review.
https://gerrit.wikimedia.org/r/284599
Change subject: Periodically check for configuration file changes.
..
Periodically check for configuration file changes.
This allows a service to gracefully restart when its configuration changes.
Change-Id: I586c2c39b8efcacbd50e884374aaeae2890b4bb9
---
M defaults.js
M lib/cli.js
M mw-ocg-service.js
3 files changed, 61 insertions(+), 3 deletions(-)
git pull
ssh://gerrit.wikimedia.org:29418/mediawiki/extensions/Collection/OfflineContentGenerator
refs/changes/99/284599/1
diff --git a/defaults.js b/defaults.js
index edfc8ed..a97637a 100644
--- a/defaults.js
+++ b/defaults.js
@@ -23,6 +23,12 @@
/** Public hostname of this instance for HTTP GET requests for
locally stored content. */
hostname: null,
+
+ /** How often (in seconds) to check for configuration changes.
+* We gracefully shutdown when we get a config change;
presumably
+* upstart or some other service runner will restart us.
Defaults
+* to zero which means, "never check for configuration
changes". */
+ checkTime: 0,
},
/** Configuration for the frontend HTTP server thread. You can choose
to serve
* content via a local socket, or an IP address. If both are null the
server will
diff --git a/lib/cli.js b/lib/cli.js
index 7972cdb..77d07a8 100644
--- a/lib/cli.js
+++ b/lib/cli.js
@@ -18,9 +18,15 @@
// Parse configuration files, with optional command-line override.
var parseConfig = exports.parseConfig = function(commanderConfig) {
+ var deps = [], latest = 0, req = function(f) {
+ f = path.resolve(__dirname, f);
+ deps.push('+' + f); // record filenames for all config files
parsed.
+ latest = Math.max(latest, fs.statSync(f).ctime.getTime());
+ return require(f);
+ };
/* === Configuration Options & File === */
- var config = require('../defaults.js'), configPath = '..';
+ var config = req('../defaults.js'), configPath = '..';
// Local configuration overrides.
while (config.config) {
var configFile = relativeTo(configPath, config.config);
@@ -28,9 +34,10 @@
try {
fs.statSync(configFile);
} catch (e) {
+ deps.push('-' + configFile);
break; // File not present.
}
- config = require(configFile)(config) || config;
+ config = req(configFile)(config) || config;
configPath = path.dirname(configFile);
}
@@ -41,7 +48,7 @@
// to the current working directory instead of relative
to the path of this
// file.
commanderConfig = relativeTo(process.cwd(),
commanderConfig);
- config = require(commanderConfig)(config) || config;
+ config = req(commanderConfig)(config) || config;
}
} catch (err) {
console.error('Could not open configuration file %s! %s',
commanderConfig, err);
@@ -54,9 +61,38 @@
"Failed jobs shouldn't be gc'd before the lockout time expires."
);
+ // Record all dependent files and the latest ctime.
+ config.config_deps = deps;
+ config.config_time = latest;
return config;
};
+// Check to see if any configuration files have changed.
+var configChanged = exports.configChanged = function(config) {
+ var deps = config.config_deps, latest = config.config_time, i;
+ for (i=0; i latest) {
+ // Newer file, or an expected file disappeared.
+ return true;
+ }
+ } else {
+ if (s!==null) {
+ // A file appeared!
+ return true;
+ }
+ }
+ }
+ // No changes.
+ return false;
+};
+
// Set up logging.
var setupLogging = exports.setupLogging = function(config, forceStdout) {
var bunyan = require('bunyan');
diff --git a/mw-ocg-service.js b/mw-ocg-service.js
index 9f99d66..e140ee6 100755
--- a/mw-ocg-service.js
+++ b/mw-ocg-service.js
@@ -71,8 +71,10 @@
var i;
/* --- Thread management --- */
+ var checkTimer = 0;
var gracefulShutdown = function gracefulShutdown() {
respawnWorkers = false;
+ if (checkTimer) clearTimeout(checkTimer);
console.info('Beginning graceful shutdown');
var shutdownSome = function(which) {
@@ -111,6 +113,7 @@
var immediateShutdown = function immediateShutdown() {
respawnWorkers = false
[MediaWiki-commits] [Gerrit] Send Accept header to Parsoid/RESTBase to get data-mw attrib... - change (mediawiki...bundler)
Cscott has uploaded a new change for review.
https://gerrit.wikimedia.org/r/284356
Change subject: Send Accept header to Parsoid/RESTBase to get data-mw
attributes inline.
..
Send Accept header to Parsoid/RESTBase to get data-mw attributes inline.
Change-Id: I336eee1339f686446a6bfdd04b52981cc69e3d39
---
M lib/parsoid.js
1 file changed, 8 insertions(+), 2 deletions(-)
git pull
ssh://gerrit.wikimedia.org:29418/mediawiki/extensions/Collection/OfflineContentGenerator/bundler
refs/changes/56/284356/1
diff --git a/lib/parsoid.js b/lib/parsoid.js
index 06f6a53..936a5bd 100644
--- a/lib/parsoid.js
+++ b/lib/parsoid.js
@@ -17,6 +17,12 @@
// Limit the # of concurrent requests to parsoid.
var PARSOID_REQUEST_LIMIT = 5;
+// Add `Accept` header to get data-mw attributes inline.
+var pheaders = {};
+Object.keys(headers).forEach(function(k) { pheaders[k] = headers[k]; });
+pheaders['Accept'] =
+ 'text/html; charset=utf-8; profile="mediawiki.org/specs/html/1.2.1"';
+
// Escape special regexp characters in a string.
// Used to build a regexp matching a literal string.
var escapeRegExp = function(s) {
@@ -229,7 +235,7 @@
}
var apiURL = findApiUrl('GET', this.apiVersion, this.wikis[wiki],
title, revid);
- request({ url: apiURL.url, encoding: 'utf8', headers: headers, pool:
false, log: this.log }, function(error, response, body) {
+ request({ url: apiURL.url, encoding: 'utf8', headers: pheaders, pool:
false, log: this.log }, function(error, response, body) {
if (error || response.statusCode !== 200) {
deferred.reject(new Error('Error fetching ' +
apiURL.api + ' result: ' + apiURL.url + ' ' + (error || response.statusCode)));
} else {
@@ -255,7 +261,7 @@
url: apiURL.url,
method: 'POST',
encoding: 'utf8',
- headers: headers,
+ headers: pheaders,
pool: false,
log: this.log,
form: (
--
To view, visit https://gerrit.wikimedia.org/r/284356
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: I336eee1339f686446a6bfdd04b52981cc69e3d39
Gerrit-PatchSet: 1
Gerrit-Project: mediawiki/extensions/Collection/OfflineContentGenerator/bundler
Gerrit-Branch: master
Gerrit-Owner: Cscott
___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] Resolve titles even when $wgArticlePath involves a query str... - change (mediawiki...bundler)
Cscott has uploaded a new change for review.
https://gerrit.wikimedia.org/r/284232
Change subject: Resolve titles even when $wgArticlePath involves a query string.
..
Resolve titles even when $wgArticlePath involves a query string.
Change-Id: Idf9018f54ee741f3db6ec4f54d8087a8d5a17d84
---
M lib/parsoid.js
1 file changed, 13 insertions(+), 2 deletions(-)
git pull
ssh://gerrit.wikimedia.org:29418/mediawiki/extensions/Collection/OfflineContentGenerator/bundler
refs/changes/32/284232/1
diff --git a/lib/parsoid.js b/lib/parsoid.js
index a48c006..06f6a53 100644
--- a/lib/parsoid.js
+++ b/lib/parsoid.js
@@ -37,7 +37,8 @@
this.document = domino.createDocument(text);
// Create a regexp that matches the article path in order to resolve
URLs
// into titles.
- var m = /^([\s\S]*)\$1([\s\S]*)$/.exec(siteinfo.general.articlepath);
+ this.articlepath = siteinfo.general.articlepath;
+ var m = /^([\s\S]*)\$1([\s\S]*)$/.exec(this.articlepath);
this._resolveRE = new RegExp(
'^' + escapeRegExp(m[1]) + '([\\s\\S]*)' + escapeRegExp(m[2]) +
'$'
);
@@ -45,11 +46,21 @@
// Resolve an article title.
ParsoidResult.prototype._resolve = function(href) {
+ if (/\?/.test(this.articlepath)) {
+ // Fall back to old-style stripping
+ return decodeURIComponent(href.replace(/^(..?\/)*/, ''));
+ }
+ // Resolve to an absolute URL
var path = url.parse(url.resolve(this.getBaseHref(), href), false,
true).
pathname;
// Now remove the articlepath.
var m = this._resolveRE.exec(path);
- if (!m) { throw new Error('Bad article title: ' + href); }
+ if (!m) {
+ throw new Error(
+ '$wgArticlePath is set incorrectly: ' +
+ this.articlepath + ' ' + this.getBaseHref() + ' ' + href
+ );
+ }
return decodeURIComponent(m[1]);
};
--
To view, visit https://gerrit.wikimedia.org/r/284232
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: Idf9018f54ee741f3db6ec4f54d8087a8d5a17d84
Gerrit-PatchSet: 1
Gerrit-Project: mediawiki/extensions/Collection/OfflineContentGenerator/bundler
Gerrit-Branch: master
Gerrit-Owner: Cscott
___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] Note issues with hyperlinks when $wgUsePathInfo is false. - change (mediawiki...latex_renderer)
Cscott has uploaded a new change for review. https://gerrit.wikimedia.org/r/284231 Change subject: Note issues with hyperlinks when $wgUsePathInfo is false. .. Note issues with hyperlinks when $wgUsePathInfo is false. Parsoid generates a tag, which doesn't support article paths involving query strings. Change-Id: Ieb5cd20af0a099495567c3a8bd094109e3ef6492 --- M README.md 1 file changed, 4 insertions(+), 0 deletions(-) git pull ssh://gerrit.wikimedia.org:29418/mediawiki/extensions/Collection/OfflineContentGenerator/latex_renderer refs/changes/31/284231/1 diff --git a/README.md b/README.md index 4a2488d..87d65c0 100644 --- a/README.md +++ b/README.md @@ -42,6 +42,9 @@ In Ubuntu 12.04, the `lmodern` package must also be installed manually. +Hyperlinks in generated PDFs will not resolve correctly unless the +target wiki has [$wgUsePathInfo] enabled. + ## Generating bundles You may wish to install the [mw-ocg-bundler] npm package to create bundles @@ -91,6 +94,7 @@ [mw-ocg-bundler]: https://github.com/wikimedia/mediawiki-extensions-Collection-OfflineContentGenerator-bundler [XeLaTeX]: https://en.wikipedia.org/wiki/XeTeX +[$wgUsePathInfo]: https://www.mediawiki.org/wiki/Manual:$wgUsePathInfo [NPM1]: https://nodei.co/npm/mw-ocg-latexer.png [NPM2]: https://nodei.co/npm/mw-ocg-latexer/ -- To view, visit https://gerrit.wikimedia.org/r/284231 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: newchange Gerrit-Change-Id: Ieb5cd20af0a099495567c3a8bd094109e3ef6492 Gerrit-PatchSet: 1 Gerrit-Project: mediawiki/extensions/Collection/OfflineContentGenerator/latex_renderer Gerrit-Branch: master Gerrit-Owner: Cscott ___ MediaWiki-commits mailing list MediaWiki-commits@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] Use domain for status messages if prefix is not available. - change (mediawiki...bundler)
Cscott has uploaded a new change for review.
https://gerrit.wikimedia.org/r/284207
Change subject: Use domain for status messages if prefix is not available.
..
Use domain for status messages if prefix is not available.
Change-Id: Ie1b01716e1c851ba19fc93a79ee0f9d5617b58b0
---
M lib/authors.js
M lib/html.js
M lib/index.js
M lib/parsoid.js
M lib/revisions.js
M lib/siteinfo.js
6 files changed, 18 insertions(+), 7 deletions(-)
git pull
ssh://gerrit.wikimedia.org:29418/mediawiki/extensions/Collection/OfflineContentGenerator/bundler
refs/changes/07/284207/1
diff --git a/lib/authors.js b/lib/authors.js
index 4fdd5c0..4d6ea7f 100644
--- a/lib/authors.js
+++ b/lib/authors.js
@@ -22,7 +22,9 @@
Authors.prototype.fetchMetadata = Promise.guard(AUTHORS_REQUEST_LIMIT,
function(wiki, title, revid, status /* Optional. */) {
if (status) {
status.report(null, util.format(
- '%s:%s [authors, %s]', this.wikis[wiki].prefix, title,
+ '%s:%s [authors, %s]',
+ this.wikis[wiki].prefix || this.wikis[wiki].domain,
+ title,
revid ? ('revision ' + revid) : 'latest revision'
));
}
diff --git a/lib/html.js b/lib/html.js
index 0a8b76c..4433321 100644
--- a/lib/html.js
+++ b/lib/html.js
@@ -26,7 +26,9 @@
// This is inside the guard, so if we launch lots of fetches in
// parallel, we won't report them all at once.
status.report(null, util.format(
- '%s:%s [PHP, %s]', this.wikis[wiki].prefix, title,
+ '%s:%s [PHP, %s]',
+ this.wikis[wiki].prefix || this.wikis[wiki].domain,
+ title,
revid ? ('revision ' + revid) : 'latest revision'
));
}
diff --git a/lib/index.js b/lib/index.js
index eeedf9e..5b661fd 100644
--- a/lib/index.js
+++ b/lib/index.js
@@ -225,7 +225,8 @@
}
status.report(null, util.format(
'%s:%s [complete]',
- metabook.wikis[item.wiki].prefix,
item.title
+ metabook.wikis[item.wiki].prefix ||
+ metabook.wikis[item.wiki].domain,
item.title
));
});
};
diff --git a/lib/parsoid.js b/lib/parsoid.js
index 98952ff..a48c006 100644
--- a/lib/parsoid.js
+++ b/lib/parsoid.js
@@ -183,7 +183,7 @@
}.bind(this));
}
- var prefix = this.wikis[wiki].prefix;
+ var prefix = this.wikis[wiki].prefix || this.wikis[wiki].domain;
if (status) {
// This is inside the guard, so if we launch lots of fetches in
// parallel, we won't report them all at once.
@@ -230,7 +230,7 @@
var parse = function(siteinfo, wiki, wikitext, title, status) {
wiki = wiki || 0;
- var prefix = this.wikis[wiki].prefix;
+ var prefix = this.wikis[wiki].prefix || this.wikis[wiki].domain;
return new Promise(function(resolve, reject) {
if (status) {
diff --git a/lib/revisions.js b/lib/revisions.js
index b97b5b7..a8b4d3a 100644
--- a/lib/revisions.js
+++ b/lib/revisions.js
@@ -100,7 +100,10 @@
null;
var fetchAndWriteOne = Promise.guard(REVISION_REQUEST_LIMIT,
function(t) {
- status.report(null, this.wikis[t.wiki].prefix + ':' + t.title);
+ status.report(null, util.format(
+ '%s:%s',
+ this.wikis[t.wiki].prefix || this.wikis[t.wiki].domain,
+ t.title));
return this.fetchOne(t.wiki, t.canontitle || t.title, t.revid).
then(function(data) {
return data === null ? null : writeOne(data,
revStream, revDb, options);
diff --git a/lib/siteinfo.js b/lib/siteinfo.js
index 60eebf1..d8b75d9 100644
--- a/lib/siteinfo.js
+++ b/lib/siteinfo.js
@@ -54,7 +54,10 @@
var db = new Db(path.join(outdir, 'siteinfo.db'));
return Promise.all(wikis.map(function(_, wiki) {
return this.fetch(wiki).then(function(siteinfo) {
- status.report(null, wikis[wiki].prefix + ' siteinfo');
+ status.report(null, util.format(
+ '%s siteinfo',
+ wikis[wiki].prefix || wikis[wiki].domain
+ ));
// Only write siteinfo.json for wiki 0 (backward
compat).
var p = (wiki === 0 && options.compat) ?
writeSiteInfo(outdir, wiki, siteinfo, options) :
--
To view, visit https://gerrit.wikimedia.
[MediaWiki-commits] [Gerrit] Don't fallback to rest.wikimedia.org. - change (mediawiki...bundler)
Cscott has uploaded a new change for review.
https://gerrit.wikimedia.org/r/284202
Change subject: Don't fallback to rest.wikimedia.org.
..
Don't fallback to rest.wikimedia.org.
The rest.wikimedia.org domain is deprecated (we prefer to use the
/api/rest_v1/ urls), and this "helpful" default
ends up breaking things for users who are running OCG on their own
local wikis and don't have a restbase instance set up.
Also remove the domain of the old public parsoid service, which has been
shut down.
Change-Id: I04c0dea82773f5fb5c8f229daad4d9212744592e
---
M bin/mw-ocg-bundler
M lib/metabook.js
2 files changed, 2 insertions(+), 4 deletions(-)
git pull
ssh://gerrit.wikimedia.org:29418/mediawiki/extensions/Collection/OfflineContentGenerator/bundler
refs/changes/02/284202/1
diff --git a/bin/mw-ocg-bundler b/bin/mw-ocg-bundler
index a7aabd9..0e91017 100755
--- a/bin/mw-ocg-bundler
+++ b/bin/mw-ocg-bundler
@@ -26,7 +26,7 @@
.option('-h, --domainname ',
'Wiki hostname to use to resolve the title (Parsoid
v2/v3 API, RESTbase API) [en.wikipedia.org]', null)
.option('-a, --parsoid-api ',
- 'Parsoid API root',
'http://parsoid-lb.eqiad.wikimedia.org/')
+ 'Parsoid API root')
.option('--restbase-api ',
'RESTbase API root
[http://rest.wikimedia.org//v1/]')
.option('--php-api ',
@@ -240,8 +240,6 @@
restbaseApi: program.restbaseApi ?
// Be user-friendly: strip trailing /page/html/ if
present.
program.restbaseApi.replace(/\/page\/html\/?$/, '/') :
- program.domainname ?
- 'http://rest.wikimedia.org/' + program.domainname +
'/v1/' :
undefined,
parsoidApi: program.parsoidApi,
apiVersion: program.apiVersion,
diff --git a/lib/metabook.js b/lib/metabook.js
index 3bc9c4d..300884d 100644
--- a/lib/metabook.js
+++ b/lib/metabook.js
@@ -35,7 +35,7 @@
script_extension: '.php',
format: 'nuwiki',
// Our extra fields (filled in below).
- restbase: undefined,
+ restbase1: undefined,
parsoid: undefined,
prefix: undefined,
// The filerepos is a link to commons, etc; it looks something
like:
--
To view, visit https://gerrit.wikimedia.org/r/284202
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: I04c0dea82773f5fb5c8f229daad4d9212744592e
Gerrit-PatchSet: 1
Gerrit-Project: mediawiki/extensions/Collection/OfflineContentGenerator/bundler
Gerrit-Branch: master
Gerrit-Owner: Cscott
___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] Document how to use the bundler to point at a local wiki. - change (mediawiki...bundler)
Cscott has uploaded a new change for review. https://gerrit.wikimedia.org/r/284191 Change subject: Document how to use the bundler to point at a local wiki. .. Document how to use the bundler to point at a local wiki. Change-Id: I392a50769f2bb27be758f96c2af05ccdf751cf9e --- M README.md 1 file changed, 13 insertions(+), 0 deletions(-) git pull ssh://gerrit.wikimedia.org:29418/mediawiki/extensions/Collection/OfflineContentGenerator/bundler refs/changes/91/284191/1 diff --git a/README.md b/README.md index 9557a43..303fa5c 100644 --- a/README.md +++ b/README.md @@ -43,6 +43,18 @@ For non-interactive use feel free to remove the `-v` flag. +If you are running a local mediawiki instance, use appropriate `-h`, +`--parsoid-api`, and `--php-api` options to point at your local wiki and local +Parsoid installation: +``` +bin/mw-ocg-bundler -v -o bundle.zip -h localhost --parsoid-api http://localhost:8142 --php-api http://localhost/api.php "Main Page" +``` + +Note that the argument to `-h` must match the "domain" you've +configured in Parsoid's `localsettings.js` and MediaWiki's +`$wgVirtualRestConfig`; it doesn't necessarily need to be a valid +DNS domain. See [Visual Editor's configuration guide] for more details. + For other options, see: ``` bin/mw-ocg-bundler --help @@ -62,6 +74,7 @@ [mw-ocg-latexer]: https://github.com/wikimedia/mediawiki-extensions-Collection-OfflineContentGenerator-latex_renderer [mw-ocg-texter]: https://github.com/cscott/mw-ocg-texter [XeLaTeX]:https://en.wikipedia.org/wiki/XeTeX +[Visual Editor's configuration guide]: https://www.mediawiki.org/wiki/Extension:VisualEditor#Linking_with_Parsoid [NPM1]: https://nodei.co/npm/mw-ocg-bundler.png [NPM2]: https://nodei.co/npm/mw-ocg-bundler/ -- To view, visit https://gerrit.wikimedia.org/r/284191 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I392a50769f2bb27be758f96c2af05ccdf751cf9e Gerrit-PatchSet: 1 Gerrit-Project: mediawiki/extensions/Collection/OfflineContentGenerator/bundler Gerrit-Branch: master Gerrit-Owner: Cscott ___ MediaWiki-commits mailing list MediaWiki-commits@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] Update README to use Parsoid v3 API. - change (mediawiki...OfflineContentGenerator)
Cscott has uploaded a new change for review.
https://gerrit.wikimedia.org/r/284188
Change subject: Update README to use Parsoid v3 API.
..
Update README to use Parsoid v3 API.
Change-Id: Ie1742ba018c268b923ef21689d3820d5cd88ccfd
---
M README.md
1 file changed, 24 insertions(+), 12 deletions(-)
git pull
ssh://gerrit.wikimedia.org:29418/mediawiki/extensions/Collection/OfflineContentGenerator
refs/changes/88/284188/1
diff --git a/README.md b/README.md
index fb9a01c..aba24b6 100644
--- a/README.md
+++ b/README.md
@@ -30,33 +30,44 @@
If you have installed [VisualEditor], no additional configuration will
be necessary: OCG will use the VisualEditor configuration to find an
-appropriate RESTBase or Parsoid service and prefix.
+appropriate RESTBase or Parsoid service and prefix. No special
+`localsettings.js` file will be required.
+
+## Installing a standalone OCG without Visual Editor
In the absence of VisualEditor, you will still need to install
-[Parsoid], and then configure OCG to use it. You will launch
-`mw-ocg-service` as `./mw-ocg-service.js -c localsettings.js` and
+[Parsoid], and then configure OCG to use it. Use the instructions in
+the [Visual Editor configuration guide].
+
+You will also have to launch `mw-ocg-service` as
+`./mw-ocg-service.js -c localsettings.js` and
create a `localsettings.js` file containing:
```javascript
// for mw-ocg-service
module.exports = function(config) {
- // change the port here if you are running parsoid on a different port
+ // URL to the Parsoid instance
+ // Use port 8142 if you use the Debian package
config.backend.bundler.parsoid_api = "http://localhost:8000";;
- // the prefix here should match $wgDBname in your LocalSettings.php
- config.backend.bundler.parsoid_prefix = "localhost";
- // Use the Parsoid "v1" API
- config.backend.bundler.additionalArgs = [ '--api-version=parsoid1' ];
+ // Use the Parsoid "v3" API
+ // The "domainname" should match the "domain" in the setMwApi
+ // call in Parsoid's localsettings.js and the "domain" in mediawiki's
+ // $wgVirtualRestConfig in LocalSettings.php
+ config.backend.bundler.additionalArgs = [ '--domainname=localhost' ];
}
```
+As described in the [Visual Editor configuration guide] mentioned above,
Parsoid would in turn be configured with its own `localsettings.js`
containing:
```javascript
// for Parsoid
exports.setup = function( parsoidConfig ) {
- // first argument here should match $wgDBname in your LocalSettings.php
- parsoidConfig.setInterwiki( 'localhost',
'http://localhost/path/to/your/mediawiki/api.php' );
- // optional:
- parsoidConfig.serverPort = 8000;
+ parsoidConfig.setMwApi({
+uri: 'http://path/to/your/mediawiki/api.php',
+domain: 'localhost',
+// "prefix" should match $wgDBname in your LocalSettings.php
+prefix: 'localhost'
+ });
};
```
@@ -143,6 +154,7 @@
[mw-ocg-bundler]:
https://github.com/wikimedia/mediawiki-extensions-Collection-OfflineContentGenerator-bundler
[mw-ocg-latexer]:
https://github.com/wikimedia/mediawiki-extensions-Collection-OfflineContentGenerator-latex_renderer
[mw-ocg-texter]:
https://github.com/wikimedia/mediawiki-extensions-Collection-OfflineContentGenerator-text_renderer
+[Visual Editor configuration guide]:
https://www.mediawiki.org/wiki/Extension:VisualEditor#Linking_with_Parsoid
[1]:
https://david-dm.org/wikimedia/mediawiki-extensions-Collection-OfflineContentGenerator.png
[2]:
https://david-dm.org/wikimedia/mediawiki-extensions-Collection-OfflineContentGenerator
--
To view, visit https://gerrit.wikimedia.org/r/284188
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: Ie1742ba018c268b923ef21689d3820d5cd88ccfd
Gerrit-PatchSet: 1
Gerrit-Project: mediawiki/extensions/Collection/OfflineContentGenerator
Gerrit-Branch: master
Gerrit-Owner: Cscott
___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] WIP: Hook up Balancer as a Tidy implementation. - change (mediawiki/core)
Cscott has uploaded a new change for review.
https://gerrit.wikimedia.org/r/282733
Change subject: WIP: Hook up Balancer as a Tidy implementation.
..
WIP: Hook up Balancer as a Tidy implementation.
Change-Id: I4fd433fd6f1847061b0bf4b3e249c918720d4fae
---
M autoload.php
M includes/DefaultSettings.php
M includes/parser/MWTidy.php
A includes/tidy/Html5Internal.php
4 files changed, 23 insertions(+), 0 deletions(-)
git pull ssh://gerrit.wikimedia.org:29418/mediawiki/core
refs/changes/33/282733/1
diff --git a/autoload.php b/autoload.php
index e686594..5cc15db 100644
--- a/autoload.php
+++ b/autoload.php
@@ -819,6 +819,7 @@
'MediaWiki\\Tidy\\BalanceStack' => __DIR__ .
'/includes/tidy/Balancer.php',
'MediaWiki\\Tidy\\Balancer' => __DIR__ . '/includes/tidy/Balancer.php',
'MediaWiki\\Tidy\\Html5Depurate' => __DIR__ .
'/includes/tidy/Html5Depurate.php',
+ 'MediaWiki\\Tidy\\Html5Internal' => __DIR__ .
'/includes/tidy/Html5Internal.php',
'MediaWiki\\Tidy\\RaggettBase' => __DIR__ .
'/includes/tidy/RaggettBase.php',
'MediaWiki\\Tidy\\RaggettExternal' => __DIR__ .
'/includes/tidy/RaggettExternal.php',
'MediaWiki\\Tidy\\RaggettInternalHHVM' => __DIR__ .
'/includes/tidy/RaggettInternalHHVM.php',
diff --git a/includes/DefaultSettings.php b/includes/DefaultSettings.php
index d9d04fd..9f81960 100644
--- a/includes/DefaultSettings.php
+++ b/includes/DefaultSettings.php
@@ -4144,6 +4144,8 @@
*- RaggettInternalHHVM: Use the limited-functionality HHVM extension
*- RaggettInternalPHP: Use the PECL extension
*- RaggettExternal: Shell out to an external binary (tidyBin)
+ *- Html5Depurate: Use external Depurate service
+ *- Html5Internal: Use the built-in HTML5 balancer
*
* - tidyConfigFile: Path to configuration file for any of the Raggett drivers
* - debugComment: True to add a comment to the output with warning messages
diff --git a/includes/parser/MWTidy.php b/includes/parser/MWTidy.php
index bdf3efb..f281c25 100644
--- a/includes/parser/MWTidy.php
+++ b/includes/parser/MWTidy.php
@@ -132,6 +132,9 @@
case 'Html5Depurate':
self::$instance = new
MediaWiki\Tidy\Html5Depurate( $config );
break;
+ case 'Html5Internal':
+ self::$instance = new
MediaWiki\Tidy\Html5Internal( $config );
+ break;
default:
throw new MWException( "Invalid tidy
driver: \"{$config['driver']}\"" );
}
diff --git a/includes/tidy/Html5Internal.php b/includes/tidy/Html5Internal.php
new file mode 100644
index 000..eed9046
--- /dev/null
+++ b/includes/tidy/Html5Internal.php
@@ -0,0 +1,17 @@
+ true,
+ ] );
+ $this->balancer = new Balancer( $this->config['strict'] );
+ }
+
+ public function tidy( $text ) {
+ return $this->balancer->balance( $text );
+ }
+}
--
To view, visit https://gerrit.wikimedia.org/r/282733
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: I4fd433fd6f1847061b0bf4b3e249c918720d4fae
Gerrit-PatchSet: 1
Gerrit-Project: mediawiki/core
Gerrit-Branch: master
Gerrit-Owner: Cscott
___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] Bump src to a5be1cdc (deploy-20160405 branch) for deploy - change (mediawiki...deploy)
Cscott has uploaded a new change for review. https://gerrit.wikimedia.org/r/281805 Change subject: Bump src to a5be1cdc (deploy-20160405 branch) for deploy .. Bump src to a5be1cdc (deploy-20160405 branch) for deploy This is a cherry-pick of Iefc0f9fbb55a9c5ca10246871d027e569ea86167 to keep up with core's deploy of 1.27.0-wmf.20, which reorders the alias list in core (T116020). Change-Id: If526570a1bc7c8fa697ff49b83b14d5360b748f8 --- M src 1 file changed, 0 insertions(+), 0 deletions(-) git pull ssh://gerrit.wikimedia.org:29418/mediawiki/services/parsoid/deploy refs/changes/05/281805/1 diff --git a/src b/src index 579ec3e..a5be1cd 16 --- a/src +++ b/src -Subproject commit 579ec3e63e0181eaa3965cbe7aea67bc507e884e +Subproject commit a5be1cdc67eefbe691166e5c581c5c4df5cf66f8 -- To view, visit https://gerrit.wikimedia.org/r/281805 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: newchange Gerrit-Change-Id: If526570a1bc7c8fa697ff49b83b14d5360b748f8 Gerrit-PatchSet: 1 Gerrit-Project: mediawiki/services/parsoid/deploy Gerrit-Branch: master Gerrit-Owner: Cscott ___ MediaWiki-commits mailing list MediaWiki-commits@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] Improve comment to localizers in MessagesEn.php - change (mediawiki/core)
Cscott has uploaded a new change for review. https://gerrit.wikimedia.org/r/281701 Change subject: Improve comment to localizers in MessagesEn.php .. Improve comment to localizers in MessagesEn.php Follow up to Ia670512e0cb375335873e7f9a08b638bbe039e45. Bug: T116020 Change-Id: I8634585e52ce5339dbfcd5b19c5d1b6c9ef5687e --- M languages/messages/MessagesEn.php 1 file changed, 9 insertions(+), 4 deletions(-) git pull ssh://gerrit.wikimedia.org:29418/mediawiki/core refs/changes/01/281701/1 diff --git a/languages/messages/MessagesEn.php b/languages/messages/MessagesEn.php index 23f2bb0..674be13 100644 --- a/languages/messages/MessagesEn.php +++ b/languages/messages/MessagesEn.php @@ -198,10 +198,15 @@ * CASE is 0 to match all case variants, 1 for case-sensitive * * Note to localisers: - * - Include the English magic words as synonyms. This allows people from other - * that do no speak the language to contribute more easily. - * - Order the aliases so that common aliases are occur before more rarely - * used aliases. Tools are expected to use the first alias. + * - Include the English magic words as synonyms. This allows people from + * other wikis that do not speak the language to contribute more easily. + * - The first alias listed MUST be the preferred alias in that language. + * Tools (like Visual Editor) are expected to use the first listed alias + * when editing or creating new content. + * - Order the other aliases so that common aliases occur before more rarely + * used aliases. The aliases SHOULD be sorted by the following convention: + * 1. Local first, English last, then + * 2. Most common first, least common last. * * This array can be modified at runtime with the LanguageGetMagic hook */ -- To view, visit https://gerrit.wikimedia.org/r/281701 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I8634585e52ce5339dbfcd5b19c5d1b6c9ef5687e Gerrit-PatchSet: 1 Gerrit-Project: mediawiki/core Gerrit-Branch: master Gerrit-Owner: Cscott ___ MediaWiki-commits mailing list MediaWiki-commits@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] Bump version after release. - change (mediawiki...bundler)
Cscott has uploaded a new change for review.
https://gerrit.wikimedia.org/r/280723
Change subject: Bump version after release.
..
Bump version after release.
Change-Id: Iafc05e3c28356b4447965e708c9dcf7ae1de9c9b
---
M CHANGELOG.md
M package.json
2 files changed, 3 insertions(+), 1 deletion(-)
git pull
ssh://gerrit.wikimedia.org:29418/mediawiki/extensions/Collection/OfflineContentGenerator/bundler
refs/changes/23/280723/1
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 6b13963..e530216 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -1,3 +1,5 @@
+# mw-ocg-bundler x.x.x (not yet released)
+
# mw-ocg-bundler 1.3.1 (2016-03-31)
* Use underscores in titles sent to the REST v1/Parsoid v3 API.
* Use `body_only` instead of `bodyOnly` in REST v1/Parsoid v3 API.
diff --git a/package.json b/package.json
index eae8b96..e8d4840 100644
--- a/package.json
+++ b/package.json
@@ -1,6 +1,6 @@
{
"name": "mw-ocg-bundler",
- "version": "1.3.1",
+ "version": "1.3.1-git",
"description": "Generates resource bundles from mediawiki articles",
"main": "lib/index.js",
"repository": {
--
To view, visit https://gerrit.wikimedia.org/r/280723
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: Iafc05e3c28356b4447965e708c9dcf7ae1de9c9b
Gerrit-PatchSet: 1
Gerrit-Project: mediawiki/extensions/Collection/OfflineContentGenerator/bundler
Gerrit-Branch: master
Gerrit-Owner: Cscott
___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] Release 1.3.1. - change (mediawiki...bundler)
Cscott has uploaded a new change for review.
https://gerrit.wikimedia.org/r/280722
Change subject: Release 1.3.1.
..
Release 1.3.1.
Change-Id: Id0bf2d77c370ad07d2e13291c8a449601e37c56f
---
M CHANGELOG.md
M package.json
2 files changed, 11 insertions(+), 2 deletions(-)
git pull
ssh://gerrit.wikimedia.org:29418/mediawiki/extensions/Collection/OfflineContentGenerator/bundler
refs/changes/22/280722/1
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 23e14c0..6b13963 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -1,4 +1,13 @@
-# mw-ocg-bundler x.x.x (not yet released)
+# mw-ocg-bundler 1.3.1 (2016-03-31)
+* Use underscores in titles sent to the REST v1/Parsoid v3 API.
+* Use `body_only` instead of `bodyOnly` in REST v1/Parsoid v3 API.
+* Support downloading the set of CSS/JS modules required.
+* Add username to User-Agent header.
+* Deprecate `parsoid-lb.eqiad.wikimedia.org`.
+* Handle protocol-relative API urls.
+* Turn on jscs and clean up source code.
+* Update commander, request, sqlite3, jshint, and mocha dependencies.
+* Rename `--domain` option to `--domainname` or `-h`.
# mw-ocg-bundler 1.3.0 (2015-09-14)
* Add `--php-api` option to allow specification of PHP API endpoint.
diff --git a/package.json b/package.json
index 1d3a92d..eae8b96 100644
--- a/package.json
+++ b/package.json
@@ -1,6 +1,6 @@
{
"name": "mw-ocg-bundler",
- "version": "1.3.0-git",
+ "version": "1.3.1",
"description": "Generates resource bundles from mediawiki articles",
"main": "lib/index.js",
"repository": {
--
To view, visit https://gerrit.wikimedia.org/r/280722
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: Id0bf2d77c370ad07d2e13291c8a449601e37c56f
Gerrit-PatchSet: 1
Gerrit-Project: mediawiki/extensions/Collection/OfflineContentGenerator/bundler
Gerrit-Branch: master
Gerrit-Owner: Cscott
___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] Release 0.3.3. - change (mediawiki...text_renderer)
Cscott has uploaded a new change for review.
https://gerrit.wikimedia.org/r/280720
Change subject: Release 0.3.3.
..
Release 0.3.3.
Change-Id: I8b2440b658bd1264b6a4351f50ce252a1dd2a9b2
---
M CHANGELOG.md
M package.json
2 files changed, 4 insertions(+), 2 deletions(-)
git pull
ssh://gerrit.wikimedia.org:29418/mediawiki/extensions/Collection/OfflineContentGenerator/text_renderer
refs/changes/20/280720/1
diff --git a/CHANGELOG.md b/CHANGELOG.md
index a4162b7..f269764 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -1,4 +1,6 @@
-# mw-ocg-texter x.x.x (not yet released)
+# mw-ocg-texter 0.3.3 (2016-03-31)
+* Update commander, sqlite3, jshint, mocha, and readable-stream dependencies.
+* Turn on jscs and clean up source code.
# mw-ocg-texter 0.3.2 (2015-09-15)
* Improve layout of preformatted content.
diff --git a/package.json b/package.json
index 4fe9de8..1a523f3 100644
--- a/package.json
+++ b/package.json
@@ -1,6 +1,6 @@
{
"name": "mw-ocg-texter",
- "version": "0.3.2-git",
+ "version": "0.3.3",
"description": "Converts mediawiki collection bundles (as generated by
mw-ocg-bundler) to plaintext",
"main": "lib/index.js",
"scripts": {
--
To view, visit https://gerrit.wikimedia.org/r/280720
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: I8b2440b658bd1264b6a4351f50ce252a1dd2a9b2
Gerrit-PatchSet: 1
Gerrit-Project:
mediawiki/extensions/Collection/OfflineContentGenerator/text_renderer
Gerrit-Branch: master
Gerrit-Owner: Cscott
___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] Bump version after release. - change (mediawiki...text_renderer)
Cscott has uploaded a new change for review.
https://gerrit.wikimedia.org/r/280721
Change subject: Bump version after release.
..
Bump version after release.
Change-Id: Id1b6478018d894b81af1452b54783aba4e1bd1aa
---
M CHANGELOG.md
M package.json
2 files changed, 3 insertions(+), 1 deletion(-)
git pull
ssh://gerrit.wikimedia.org:29418/mediawiki/extensions/Collection/OfflineContentGenerator/text_renderer
refs/changes/21/280721/1
diff --git a/CHANGELOG.md b/CHANGELOG.md
index f269764..578722d 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -1,3 +1,5 @@
+# mw-ocg-texter x.x.x (not yet released)
+
# mw-ocg-texter 0.3.3 (2016-03-31)
* Update commander, sqlite3, jshint, mocha, and readable-stream dependencies.
* Turn on jscs and clean up source code.
diff --git a/package.json b/package.json
index 1a523f3..ac25037 100644
--- a/package.json
+++ b/package.json
@@ -1,6 +1,6 @@
{
"name": "mw-ocg-texter",
- "version": "0.3.3",
+ "version": "0.3.3-git",
"description": "Converts mediawiki collection bundles (as generated by
mw-ocg-bundler) to plaintext",
"main": "lib/index.js",
"scripts": {
--
To view, visit https://gerrit.wikimedia.org/r/280721
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: Id1b6478018d894b81af1452b54783aba4e1bd1aa
Gerrit-PatchSet: 1
Gerrit-Project:
mediawiki/extensions/Collection/OfflineContentGenerator/text_renderer
Gerrit-Branch: master
Gerrit-Owner: Cscott
___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] WIP: Add {{#balance}} to opt-in to balanced templates - change (mediawiki/core)
Cscott has uploaded a new change for review.
https://gerrit.wikimedia.org/r/279670
Change subject: WIP: Add {{#balance}} to opt-in to balanced templates
..
WIP: Add {{#balance}} to opt-in to balanced templates
Allow templates to opt-in to balancing, which means (roughly) that the
output of the template will not contain any unclosed tags nor trigger
the HTML adoption agency algorithm when included in a page. This
allows high-performance incremental update of page contents after
templates are modified, and ensures the safe editing of the template
using HTML-based tools such as VisualEditor or jsapi.
The RFC for {{#balance}} proposes three balance types: block, inline,
and table. As a first step, this patch implements only the block
type, but the ground is laid for the future implementation of other
balance types.
Bug: T114445
Change-Id: I7216f6cffe1589e49336a53629b32b3264d7bc43
---
M includes/parser/CoreParserFunctions.php
M includes/parser/Parser.php
M includes/parser/Preprocessor.php
M includes/parser/Preprocessor_DOM.php
M includes/parser/Preprocessor_Hash.php
M languages/i18n/en.json
M languages/i18n/qqq.json
M languages/messages/MessagesEn.php
8 files changed, 99 insertions(+), 0 deletions(-)
git pull ssh://gerrit.wikimedia.org:29418/mediawiki/core
refs/changes/70/279670/1
diff --git a/includes/parser/CoreParserFunctions.php
b/includes/parser/CoreParserFunctions.php
index a55ddf3..6b57e0d 100644
--- a/includes/parser/CoreParserFunctions.php
+++ b/includes/parser/CoreParserFunctions.php
@@ -70,6 +70,7 @@
$parser->setFunctionHook( 'speciale', [ __CLASS__, 'speciale' ]
);
$parser->setFunctionHook( 'tag', [ __CLASS__, 'tagObj' ],
Parser::SFH_OBJECT_ARGS );
$parser->setFunctionHook( 'formatdate', [ __CLASS__,
'formatDate' ] );
+ $parser->setFunctionHook( 'balance', [ __CLASS__, 'balanceObj'
], Parser::SFH_OBJECT_ARGS );
if ( $wgAllowDisplayTitle ) {
$parser->setFunctionHook(
@@ -1047,6 +1048,30 @@
}
/**
+* Parser function to mark the current template as balanced.
+* @param Parser $parser
+* @param PPFrame $frame
+* @param PPNode[] $args
+* @return string
+*/
+ public static function balanceObj( $parser, $frame, $args ) {
+ if ( !count( $args ) ) {
+ return '';
+ }
+ $type = strtolower( trim( $frame->expand( array_shift( $args )
) ) );
+ if ( $type === '' ) {
+ $type = "block"; # default {{#balance}} type is "block"
+ }
+ if ( $type === 'none' || $type === 'block' ) {
+ $frame->setBalanceType( $type );
+ return '';
+ }
+ return '' .
+ wfMessage( 'bad-balance', $type
)->inContentLanguage()->text() .
+ '';
+ }
+
+ /**
* Fetched the current revision of the given title and return this.
* Will increment the expensive function count and
* add a template link to get the value refreshed on changes.
diff --git a/includes/parser/Parser.php b/includes/parser/Parser.php
index 5ee0c5a..b2a3fc7 100644
--- a/includes/parser/Parser.php
+++ b/includes/parser/Parser.php
@@ -3446,6 +3446,8 @@
$isChildObj = false;
// $text is a DOM node needing expansion in the current frame
$isLocalObj = false;
+ // $text should be balanced before/after (unless is "none")
+ $balanceType = "none";
# Title object, where $text came from
$title = false;
@@ -3526,6 +3528,10 @@
# Parser functions
if ( !$found ) {
$colonPos = strpos( $part1, ':' );
+ if ( $part1 === '#balance' ) {
+ # Allow the #balance function to omit the
trailing colon.
+ $colonPos = strlen( $part1 );
+ }
if ( $colonPos !== false ) {
$func = substr( $part1, 0, $colonPos );
$funcArgs = [ trim( substr( $part1, $colonPos +
1 ) ) ];
@@ -3678,6 +3684,8 @@
# Uncached expansion
$text = $newFrame->expand( $text );
}
+ # Does the child template need balancing?
+ $balanceType = $newFrame->getBalanceType();
}
if ( $isLocalObj && $nowiki ) {
$text = $frame->expand( $text, PPFrame::RECOVER_ORIG );
diff --git a/includes/parser/Preprocessor.php b/includes/parser/Preprocessor.php
index 368d79e..015bf2a 100644
--- a/inclu
[MediaWiki-commits] [Gerrit] Updating to latest masters - change (mediawiki...ocg-collection)
Cscott has submitted this change and it was merged. Change subject: Updating to latest masters .. Updating to latest masters Deployed new bundler (1fec608a4d55e7bddb115122dd0db0fd4bc88570): * Use underscores in titles sent to the RESTBase / Parsoid API. Change-Id: I26e2f21fad03368347159e9e83f2897e64f4006d --- M mw-ocg-bundler 1 file changed, 0 insertions(+), 0 deletions(-) Approvals: Cscott: Verified; Looks good to me, approved diff --git a/mw-ocg-bundler b/mw-ocg-bundler index 6491a01..1fec608 16 --- a/mw-ocg-bundler +++ b/mw-ocg-bundler -Subproject commit 6491a016302de2db86a8ee14238ce5f25707df74 +Subproject commit 1fec608a4d55e7bddb115122dd0db0fd4bc88570 -- To view, visit https://gerrit.wikimedia.org/r/278038 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: merged Gerrit-Change-Id: I26e2f21fad03368347159e9e83f2897e64f4006d Gerrit-PatchSet: 1 Gerrit-Project: mediawiki/services/ocg-collection Gerrit-Branch: master Gerrit-Owner: Cscott Gerrit-Reviewer: Cscott ___ MediaWiki-commits mailing list MediaWiki-commits@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] Updating to latest masters - change (mediawiki...ocg-collection)
Cscott has uploaded a new change for review. https://gerrit.wikimedia.org/r/278038 Change subject: Updating to latest masters .. Updating to latest masters Deployed new bundler (1fec608a4d55e7bddb115122dd0db0fd4bc88570): * Use underscores in titles sent to the RESTBase / Parsoid API. Change-Id: I26e2f21fad03368347159e9e83f2897e64f4006d --- M mw-ocg-bundler 1 file changed, 0 insertions(+), 0 deletions(-) git pull ssh://gerrit.wikimedia.org:29418/mediawiki/services/ocg-collection refs/changes/38/278038/1 diff --git a/mw-ocg-bundler b/mw-ocg-bundler index 6491a01..1fec608 16 --- a/mw-ocg-bundler +++ b/mw-ocg-bundler -Subproject commit 6491a016302de2db86a8ee14238ce5f25707df74 +Subproject commit 1fec608a4d55e7bddb115122dd0db0fd4bc88570 -- To view, visit https://gerrit.wikimedia.org/r/278038 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I26e2f21fad03368347159e9e83f2897e64f4006d Gerrit-PatchSet: 1 Gerrit-Project: mediawiki/services/ocg-collection Gerrit-Branch: master Gerrit-Owner: Cscott ___ MediaWiki-commits mailing list MediaWiki-commits@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] Merge branch 'master' into wmf-deploy - change (mediawiki...ocg-collection)
Cscott has submitted this change and it was merged. Change subject: Merge branch 'master' into wmf-deploy .. Merge branch 'master' into wmf-deploy Change-Id: I2c093791aa23a56d4333f8cafc8593c90a908e41 --- 0 files changed, 0 insertions(+), 0 deletions(-) Approvals: Cscott: Verified; Looks good to me, approved -- To view, visit https://gerrit.wikimedia.org/r/278040 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: merged Gerrit-Change-Id: I2c093791aa23a56d4333f8cafc8593c90a908e41 Gerrit-PatchSet: 1 Gerrit-Project: mediawiki/services/ocg-collection Gerrit-Branch: wmf-deploy Gerrit-Owner: Cscott Gerrit-Reviewer: Cscott ___ MediaWiki-commits mailing list MediaWiki-commits@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] Merge branch 'master' into wmf-deploy - change (mediawiki...ocg-collection)
Cscott has uploaded a new change for review. https://gerrit.wikimedia.org/r/278040 Change subject: Merge branch 'master' into wmf-deploy .. Merge branch 'master' into wmf-deploy Change-Id: I2c093791aa23a56d4333f8cafc8593c90a908e41 --- 0 files changed, 0 insertions(+), 0 deletions(-) git pull ssh://gerrit.wikimedia.org:29418/mediawiki/services/ocg-collection refs/changes/40/278040/1 -- To view, visit https://gerrit.wikimedia.org/r/278040 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I2c093791aa23a56d4333f8cafc8593c90a908e41 Gerrit-PatchSet: 1 Gerrit-Project: mediawiki/services/ocg-collection Gerrit-Branch: wmf-deploy Gerrit-Owner: Cscott ___ MediaWiki-commits mailing list MediaWiki-commits@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] Suppress duplicate prefix/domain warnings for parserTests. - change (mediawiki...parsoid)
Cscott has uploaded a new change for review.
https://gerrit.wikimedia.org/r/266594
Change subject: Suppress duplicate prefix/domain warnings for parserTests.
..
Suppress duplicate prefix/domain warnings for parserTests.
Change-Id: Id0e66d3dd9cdeb066a4f4267dcd31aed181cda58
---
M bin/parserTests.js
1 file changed, 3 insertions(+), 1 deletion(-)
git pull ssh://gerrit.wikimedia.org:29418/mediawiki/services/parsoid
refs/changes/94/266594/1
diff --git a/bin/parserTests.js b/bin/parserTests.js
index 64ff483..4f77e50 100755
--- a/bin/parserTests.js
+++ b/bin/parserTests.js
@@ -1655,7 +1655,8 @@
var extensions =
parsoidConfig.defaultNativeExtensions.concat(ParserHook);
// Send all requests to the mock API server.
- parsoidConfig.mwApiMap.forEach(function(apiConf) {
+
Array.from(parsoidConfig.mwApiMap.values()).forEach(function(apiConf) {
+ parsoidConfig.removeMwApi(apiConf);
parsoidConfig.setMwApi({
prefix: apiConf.prefix,
domain: apiConf.domain,
@@ -1666,6 +1667,7 @@
// This isn't part of the sitematrix but the
// "Check noCommafy in formatNum" test depends on it.
+ parsoidConfig.removeMwApi({ domain: 'be-tarask.wikipedia.org'
});
parsoidConfig.setMwApi({
prefix: 'be-taraskwiki',
domain: 'be-tarask.wikipedia.org',
--
To view, visit https://gerrit.wikimedia.org/r/266594
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: Id0e66d3dd9cdeb066a4f4267dcd31aed181cda58
Gerrit-PatchSet: 1
Gerrit-Project: mediawiki/services/parsoid
Gerrit-Branch: master
Gerrit-Owner: Cscott
___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] Add missing `.deployment-prep` to redis server hostname. - change (operations/puppet)
Cscott has uploaded a new change for review.
https://gerrit.wikimedia.org/r/266539
Change subject: Add missing `.deployment-prep` to redis server hostname.
..
Add missing `.deployment-prep` to redis server hostname.
Change-Id: I2861a41b2bbda3a71459d51faaeba32ad136661a
---
M hieradata/labs/deployment-prep/common.yaml
1 file changed, 3 insertions(+), 3 deletions(-)
git pull ssh://gerrit.wikimedia.org:29418/operations/puppet
refs/changes/39/266539/1
diff --git a/hieradata/labs/deployment-prep/common.yaml
b/hieradata/labs/deployment-prep/common.yaml
index 2751397..1efecb7 100644
--- a/hieradata/labs/deployment-prep/common.yaml
+++ b/hieradata/labs/deployment-prep/common.yaml
@@ -2,7 +2,7 @@
base::remote_syslog::enable: true
base::remote_syslog::central_host:
"deployment-logstash2.deployment-prep.eqiad.wmflabs:10514"
"ocg::temp_dir": /mnt/tmpfs
-"ocg::redis_host": deployment-redis01.eqiad.wmflabs
+"ocg::redis_host": deployment-redis01.deployment-prep.eqiad.wmflabs
"ocg::statsd_host": labmon1001.eqiad.wmnet
"ocg::graylog_host": deployment-logstash2.deployment-prep.eqiad.wmflabs
"ocg::redis_password": '%{::passwords::redis::main_password}'
@@ -121,9 +121,9 @@
hhvm:
pcre_cache_type: lru
"mediawiki::jobrunner::queue_servers":
-- deployment-redis01.eqiad.wmflabs
+- deployment-redis01.deployment-prep.eqiad.wmflabs
"mediawiki::jobrunner::aggr_servers":
-- deployment-redis01.eqiad.wmflabs
+- deployment-redis01.deployment-prep.eqiad.wmflabs
"mediawiki::jobrunner::statsd_server": labmon1001.eqiad.wmnet
"mediawiki::jobrunner::runners_basic": 5
"mediawiki::jobrunner::runners_gwt": 1
--
To view, visit https://gerrit.wikimedia.org/r/266539
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: I2861a41b2bbda3a71459d51faaeba32ad136661a
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Cscott
___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
