Nuria has uploaded a new change for review.
https://gerrit.wikimedia.org/r/297643
Change subject: Seeting up log retention policy in yarn
......................................................................
Seeting up log retention policy in yarn
We were retaining logs indefinitely what was filling up
the cluster with terabytes of logs. Setting log retention to
10 days which seems sufficient for troubleshooting of long jobs.
Bug: T139178
Change-Id: Iddabc37bd621136706f108b04ddf3ce116b0945e
---
M templates/hadoop/yarn-site.xml.erb
1 file changed, 13 insertions(+), 0 deletions(-)
git pull ssh://gerrit.wikimedia.org:29418/operations/puppet/cdh
refs/changes/43/297643/1
diff --git a/templates/hadoop/yarn-site.xml.erb
b/templates/hadoop/yarn-site.xml.erb
index 1a9094d..046ca6d 100644
--- a/templates/hadoop/yarn-site.xml.erb
+++ b/templates/hadoop/yarn-site.xml.erb
@@ -257,6 +257,19 @@
$HADOOP_YARN_HOME/*,$HADOOP_YARN_HOME/lib/*
</value>
</property>
+
+ <property>
+ <name>yarn.log-aggregation.retain-seconds</name>
+ <value>1080000</value>
+ <description>
+ How long (in secs) to keep aggregation logs before deleting them. -1
disables.
+ Be careful, if you set this too small you will spam the name node. Set
to 12 days.
+ If yarn.log-aggregation.retain-check-interval-seconds is not set
+ or set to 0 or a negative value (default) then the check interval is
+ one-tenth of the aggregated log retention time. So we will be checking
+ whether logs are old about daily.
+ </description>
+ </property>
<% if @yarn_site_extra_properties -%>
<% @yarn_site_extra_properties.sort.map do |key, value| -%>
--
To view, visit https://gerrit.wikimedia.org/r/297643
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: Iddabc37bd621136706f108b04ddf3ce116b0945e
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet/cdh
Gerrit-Branch: master
Gerrit-Owner: Nuria <nu...@wikimedia.org>
_______________________________________________
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
