Ottomata has submitted this change and it was merged. (
https://gerrit.wikimedia.org/r/335437 )
Change subject: Drop wdqs_extract partitions older than 90 days
......................................................................
Drop wdqs_extract partitions older than 90 days
Amended to run once per day
Bug: T146915
Change-Id: I5d29490c0e8e0314a131d12a758bdfb3d2d8735f
---
M modules/role/manifests/analytics_cluster/refinery/data/drop.pp
1 file changed, 11 insertions(+), 1 deletion(-)
Approvals:
Ottomata: Looks good to me, approved
jenkins-bot: Verified
diff --git a/modules/role/manifests/analytics_cluster/refinery/data/drop.pp
b/modules/role/manifests/analytics_cluster/refinery/data/drop.pp
index 1cacd8e..24bfb80 100644
--- a/modules/role/manifests/analytics_cluster/refinery/data/drop.pp
+++ b/modules/role/manifests/analytics_cluster/refinery/data/drop.pp
@@ -7,6 +7,7 @@
$webrequest_log_file =
"${role::analytics_cluster::refinery::log_dir}/drop-webrequest-partitions.log"
$eventlogging_log_file =
"${role::analytics_cluster::refinery::log_dir}/drop-eventlogging-partitions.log"
+ $wdqs_extract_log_file =
"${role::analytics_cluster::refinery::log_dir}/drop-wdqs-extract-partitions.log"
# Keep this many days of raw webrequest data.
$raw_retention_days = 31
@@ -34,4 +35,13 @@
minute => '15',
hour => '*/4',
}
-}
\ No newline at end of file
+
+ # keep this many days of wdqs_extract data
+ $wdqs_extract_retention_days = 90
+ cron {'refinery-drop-wdqs-extract-partitions':
+ command => "export
PYTHONPATH=\${PYTHONPATH}:${role::analytics_cluster::refinery::path}/python &&
${role::analytics_cluster::refinery::path}/bin/refinery-drop-hourly-partitions
-d ${wdqs_extract_retention_days} -D wmf -t wdqs_extract -l
/wmf/data/wmf/wdqs_extract >> ${wdqs_extract_log_file} 2>&1",
+ user => 'hdfs',
+ minute => '0',
+ hour => '1',
+ }
+}
--
To view, visit https://gerrit.wikimedia.org/r/335437
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: merged
Gerrit-Change-Id: I5d29490c0e8e0314a131d12a758bdfb3d2d8735f
Gerrit-PatchSet: 6
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Nschaaf <[email protected]>
Gerrit-Reviewer: Joal <[email protected]>
Gerrit-Reviewer: Nschaaf <[email protected]>
Gerrit-Reviewer: Nuria <[email protected]>
Gerrit-Reviewer: Ottomata <[email protected]>
Gerrit-Reviewer: jenkins-bot <>
_______________________________________________
MediaWiki-commits mailing list
[email protected]
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
