Sure. I will have a chat internally and come back to you shortly. It was a quick and dirty work actually just to fix this temporarily. However, it might be a good starting point.
On Thu, Nov 23, 2017 at 3:31 PM, Michael Miklavcic < [email protected]> wrote: > Thanks Ali, that's good feedback. Would you be willing to share any of your > Curator calls/config and use cases with the community? I'd love to add it > to a document around ES pruning in the short term, and maybe we could look > at how to build this into indexing at some point. > > Cheers, > Mike > > On Nov 22, 2017 8:53 PM, "Ali Nazemian" <[email protected]> wrote: > > > We tried to use it, but we had the same issue. It was not documented. We > > tried to use it, and we had some issues. It also was not exactly what we > > wanted, so we decided to create something from scratch by using > > Elasticsearch Curator. We wanted to have an ability to manage different > > prune mechanism for different feeds. Having a hard threshold to remove > > index and Soft threshold to close that index. Maybe it can be a feature > to > > add to the indexing JSON config file per feed. > > > > Cheers, > > Ali > > > > On Thu, Nov 23, 2017 at 12:20 PM, Michael Miklavcic < > > [email protected]> wrote: > > > > > From what I can tell, the data pruner isn't documented anywhere, so I'm > > > curious if anybody is using this, and if so, how are you using it? > > > > > > - > > > https://github.com/apache/metron/blob/master/metron- > > > platform/metron-data-management/README.md > > > - > > > https://github.com/apache/metron/blob/master/metron- > > > platform/metron-data-management/src/main/java/org/ > > > apache/metron/dataloads/bulk/ElasticsearchDataPrunerRunner.java > > > - > > > https://github.com/apache/metron/blob/master/metron- > > > platform/metron-data-management/src/main/java/org/ > > > apache/metron/dataloads/bulk/DataPruner.java > > > > > > It looks to me that it allows you to specify the start date and a > number > > of > > > days for lookback from the start date to purge along with a regex > pattern > > > to match the index name. It also does not look like it has any built-in > > > scheduling semantics, so I assume this was a cron job. I think that > about > > > covers it. Anything I've missed? > > > > > > I'm adding a quick doc write-up to METRON-939 ( > > > https://github.com/apache/metron/pull/840) for using Curator to prune > > > indices from Elasticsearch. It is desirable to make sure I've covered > > > existing use cases. > > > > > > Best, > > > Mike > > > > > > > > > > > -- > > A.Nazemian > > > -- A.Nazemian
