Ottomata has submitted this change and it was merged. Change subject: Only using hive-partitioner to create partitions on pagecounts table ......................................................................
Only using hive-partitioner to create partitions on pagecounts table We will soon be using Oozie instead of this custom python script to automatically create hive partitions on top of webrequest data Change-Id: I6acac58c0892431e7abacad3cb1d8f33066ede32 See: https://gerrit.wikimedia.org/r/#/c/131208/ --- M manifests/role/analytics/kraken.pp 1 file changed, 7 insertions(+), 1 deletion(-) Approvals: Ottomata: Verified; Looks good to me, approved jenkins-bot: Verified diff --git a/manifests/role/analytics/kraken.pp b/manifests/role/analytics/kraken.pp index 54abd83..fea1abc 100644 --- a/manifests/role/analytics/kraken.pp +++ b/manifests/role/analytics/kraken.pp @@ -1,4 +1,6 @@ # kraken.pp - role classes dealing with Kraken data analysis. +# +# NOTE! 'kraken' will be renamed soon. # == Class role::analytics::kraken # Kraken refers to the Analytics codebase used to generate @@ -113,6 +115,10 @@ $datadir = $role::analytics::kraken::external_data_hdfs_dir $database = 'wmf' + # We are only using hive-partition to add partitions to the pagecounts table. + # The webrequest table is using Oozie. + $tables = 'pagecounts' + # Note: I'm not worried about logrotate yet. # This generates just a few lines per hour. $log_file = "${role::analytics::kraken::log_dir}/hive-partitioner.log" @@ -132,7 +138,7 @@ # cron job to automatically create hive partitions for any # newly imported data. cron { 'kraken-create-external-hive-partitions': - command => "${script} --database ${database} --hive-options='${hive_options}' ${datadir} >> ${log_file} 2>&1", + command => "${script} --database ${database} --tables ${tables} --hive-options='${hive_options}' ${datadir} >> ${log_file} 2>&1", user => 'hdfs', minute => 21, require => Exec["${script}-exists"], -- To view, visit https://gerrit.wikimedia.org/r/133225 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: merged Gerrit-Change-Id: I6acac58c0892431e7abacad3cb1d8f33066ede32 Gerrit-PatchSet: 3 Gerrit-Project: operations/puppet Gerrit-Branch: production Gerrit-Owner: Ottomata <o...@wikimedia.org> Gerrit-Reviewer: Ottomata <o...@wikimedia.org> Gerrit-Reviewer: jenkins-bot <> _______________________________________________ MediaWiki-commits mailing list MediaWiki-commits@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
