Hello Ottomata,
I'd like you to do a code review. Please visit
https://gerrit.wikimedia.org/r/148700
to review the following change.
Change subject: Decrease throttle for webrequest log verification
......................................................................
Decrease throttle for webrequest log verification
The cluster default is unnecessarily high for this job, and only
bought us jobs that unnecessarily check hdfs again and again.
Change-Id: I254e96c4c92c3e842581570c54fc378e609747de
---
M oozie/webrequest/partition/add/coordinator.xml
1 file changed, 11 insertions(+), 0 deletions(-)
git pull ssh://gerrit.wikimedia.org:29418/analytics/refinery
refs/changes/00/148700/1
diff --git a/oozie/webrequest/partition/add/coordinator.xml
b/oozie/webrequest/partition/add/coordinator.xml
index db3577c..ab050ab 100644
--- a/oozie/webrequest/partition/add/coordinator.xml
+++ b/oozie/webrequest/partition/add/coordinator.xml
@@ -56,6 +56,17 @@
even-though the concurrency is low.
-->
<concurrency>2</concurrency>
+
+
+ <!--
+ Since we expect only one incarnation per hourly dataset, the
+ default throttle of 12 is way to high, and there is not need
+ to keep that many materialized jobs around.
+
+ By resorting to 2, we keep the hdfs checks on the datasets
+ low, while still being able to easily feed the concurrency.
+ -->
+ <throttle>2</throttle>
</controls>
<datasets>
--
To view, visit https://gerrit.wikimedia.org/r/148700
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: I254e96c4c92c3e842581570c54fc378e609747de
Gerrit-PatchSet: 1
Gerrit-Project: analytics/refinery
Gerrit-Branch: master
Gerrit-Owner: QChris <christ...@quelltextlich.at>
Gerrit-Reviewer: Ottomata <o...@wikimedia.org>
_______________________________________________
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
