We tried to use it, but we had the same issue. It was not documented. We tried to use it, and we had some issues. It also was not exactly what we wanted, so we decided to create something from scratch by using Elasticsearch Curator. We wanted to have an ability to manage different prune mechanism for different feeds. Having a hard threshold to remove index and Soft threshold to close that index. Maybe it can be a feature to add to the indexing JSON config file per feed.
Cheers, Ali On Thu, Nov 23, 2017 at 12:20 PM, Michael Miklavcic < [email protected]> wrote: > From what I can tell, the data pruner isn't documented anywhere, so I'm > curious if anybody is using this, and if so, how are you using it? > > - > https://github.com/apache/metron/blob/master/metron- > platform/metron-data-management/README.md > - > https://github.com/apache/metron/blob/master/metron- > platform/metron-data-management/src/main/java/org/ > apache/metron/dataloads/bulk/ElasticsearchDataPrunerRunner.java > - > https://github.com/apache/metron/blob/master/metron- > platform/metron-data-management/src/main/java/org/ > apache/metron/dataloads/bulk/DataPruner.java > > It looks to me that it allows you to specify the start date and a number of > days for lookback from the start date to purge along with a regex pattern > to match the index name. It also does not look like it has any built-in > scheduling semantics, so I assume this was a cron job. I think that about > covers it. Anything I've missed? > > I'm adding a quick doc write-up to METRON-939 ( > https://github.com/apache/metron/pull/840) for using Curator to prune > indices from Elasticsearch. It is desirable to make sure I've covered > existing use cases. > > Best, > Mike > -- A.Nazemian
