[
https://issues.apache.org/jira/browse/METRON-95?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15225432#comment-15225432
]
ASF GitHub Bot commented on METRON-95:
--------------------------------------
Github user cestella commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/68#discussion_r58473310
--- Diff: deployment/roles/metron_streaming/defaults/main.yml ---
@@ -34,3 +39,6 @@ storm_topologies:
- "{{ metron_directory }}/config/topologies/yaf/remote.yaml"
- "{{ metron_directory }}/config/topologies/pcap/remote.yaml"
- "{{ metron_directory }}/config/topologies/enrichment/remote.yaml"
+hdfs_retention_days: 30
+hdfs_bro_purge_cronjob: "yarn jar {{ metron_directory }}/lib/{{
metron_dataloads_jar }} org.apache.metron.dataloads.bulk.HDFSDataPruner -f {{
hdfs_url }} -g '/apps/metron/enrichment/indexed/bro_doc/*enrichment-*' -s
$(date -d '{{ hdfs_retention_days }} days ago' +%m/%d/%Y) -n 1 >>
/var/log/bro-purge/cron-bro-purge.log 2>&1"
--- End diff --
Not quite sure what shadow the command line options means. Yeah, some
people do explicitly call `yarn jar foo.jar main.class args`, but I suspect
they'd rather be calling `my_script_name.sh args` every single time. Before we
had this for the zookeeper config utility, I'd grep the ansible scripts every
time I needed to update the sourceconfigs in zookeeper. When I needed to
submit flux topologies (similar pattern) I'd have to grep ansible to remember
the Flux command (similarly obtuse as the yarn jar command). I don't find that
to be very good user experience. Another reason is that I can put
$METRON_HOME/bin on the path and have the script autocomplete, whereas I'm
never getting that main.class autocompleted. :)
> Create data purging script for HDFS files
> -----------------------------------------
>
> Key: METRON-95
> URL: https://issues.apache.org/jira/browse/METRON-95
> Project: Metron
> Issue Type: Sub-task
> Reporter: David M. Lyle
> Assignee: David M. Lyle
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
