Hoo man has uploaded a new change for review. (
https://gerrit.wikimedia.org/r/354494 )
Change subject: dumpWikidata: Make the minimum shard size depend on the number
of shards
......................................................................
dumpWikidata: Make the minimum shard size depend on the number of shards
So that the number of shards can easily be changed w/o having to re-calculate
these numbers.
Change-Id: I2f178f1e071b4eaa414a5d50911cd4bca1749b4b
---
M modules/snapshot/files/cron/dumpwikidatajson.sh
M modules/snapshot/files/cron/dumpwikidatardf.sh
2 files changed, 5 insertions(+), 3 deletions(-)
git pull ssh://gerrit.wikimedia.org:29418/operations/puppet
refs/changes/94/354494/1
diff --git a/modules/snapshot/files/cron/dumpwikidatajson.sh
b/modules/snapshot/files/cron/dumpwikidatajson.sh
index cac00e2..7827d3a 100644
--- a/modules/snapshot/files/cron/dumpwikidatajson.sh
+++ b/modules/snapshot/files/cron/dumpwikidatajson.sh
@@ -75,7 +75,7 @@
exit 1
fi
fileSize=`stat --printf="%s" $tempFile`
- if [ $fileSize -lt 2100000000 ]; then
+ if [ $fileSize -lt `expr 10500000000 / $shards` ]; then
echo "File size of $tempFile is only $fileSize. Aborting." >>
$mainLogFile
exit 1
fi
diff --git a/modules/snapshot/files/cron/dumpwikidatardf.sh
b/modules/snapshot/files/cron/dumpwikidatardf.sh
index f5c3b2a..1640893 100644
--- a/modules/snapshot/files/cron/dumpwikidatardf.sh
+++ b/modules/snapshot/files/cron/dumpwikidatardf.sh
@@ -12,8 +12,6 @@
declare -A dumpNameToFlavor
dumpNameToFlavor=(["all"]="full-dump" ["truthy"]="truthy-dump")
-# Sanity check: Minimal size we expect each shard of a certain dump to have
-dumpNameToMinSize=(["all"]=2500000000 ["truthy"]=1500000000)
dumpName=$1
@@ -43,6 +41,10 @@
shards=5
+declare -A dumpNameToMinSize
+# Sanity check: Minimal size we expect each shard of a certain dump to have
+dumpNameToMinSize=(["all"]=`expr 12500000000 / $shards` ["truthy"]=`expr
7500000000 / $shards`)
+
# Try to create the dump (up to three times).
retries=0
--
To view, visit https://gerrit.wikimedia.org/r/354494
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: I2f178f1e071b4eaa414a5d50911cd4bca1749b4b
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Hoo man <[email protected]>
_______________________________________________
MediaWiki-commits mailing list
[email protected]
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
