Aren't the logs stored in date-based directories? I thought we used to deploy a daily cron with a hadoop fs -rmr based on today's date minus 30 days or so.
Jon On Wed, Dec 26, 2018 at 6:20 AM Farrukh Naveed Anjum < anjum.farr...@gmail.com> wrote: > what about data in HDFS and Other Logs ? Any idea to purne them ? > > On Wed, Dec 26, 2018 at 3:51 PM zeo...@gmail.com <zeo...@gmail.com> wrote: > >> There used to be something that was automatically deployed as a cron that >> did that. For ES I know it moved to curator >> https://github.com/apache/metron/blob/master/metron-platform/metron-data-management/README.md#pruning-data-from-elasticsearch >> >> I don't know if the HDFS job still exists or not, but it used to. >> >> Jon >> >> On Wed, Dec 26, 2018, 2:06 AM Farrukh Naveed Anjum < >> anjum.farr...@gmail.com> wrote: >> >>> Hi, >>> >>> How can write a data retention script for metron. So that safe hard disk >>> space. Like I want to Store only last N Days (Say 30 days) data in HDFS, >>> Elasticssearch and Logs in Metron. >>> >>> Right now, Metron eats up my all hard drive space on server. >>> >>> Any Idea / Help will be appreciated. >>> >>> >>> -- >>> With Regards >>> Farrukh Naveed Anjum >>> >> -- >> >> Jon Zeolla >> > > > -- > With Regards > Farrukh Naveed Anjum > -- Jon Zeolla
