One thing to keep in mind, as we will be introducing Solr shortly, is to find if something similar to curator exists for Solr. But we'll cross that bridge when we get there
22.11.2017, 22:58, "Ali Nazemian" <alinazem...@gmail.com>: > Sure. I will have a chat internally and come back to you shortly. It was a > quick and dirty work actually just to fix this temporarily. However, it > might be a good starting point. > > On Thu, Nov 23, 2017 at 3:31 PM, Michael Miklavcic < > michael.miklav...@gmail.com> wrote: > >> Thanks Ali, that's good feedback. Would you be willing to share any of your >> Curator calls/config and use cases with the community? I'd love to add it >> to a document around ES pruning in the short term, and maybe we could look >> at how to build this into indexing at some point. >> >> Cheers, >> Mike >> >> On Nov 22, 2017 8:53 PM, "Ali Nazemian" <alinazem...@gmail.com> wrote: >> >> > We tried to use it, but we had the same issue. It was not documented. We >> > tried to use it, and we had some issues. It also was not exactly what we >> > wanted, so we decided to create something from scratch by using >> > Elasticsearch Curator. We wanted to have an ability to manage different >> > prune mechanism for different feeds. Having a hard threshold to remove >> > index and Soft threshold to close that index. Maybe it can be a feature >> to >> > add to the indexing JSON config file per feed. >> > >> > Cheers, >> > Ali >> > >> > On Thu, Nov 23, 2017 at 12:20 PM, Michael Miklavcic < >> > michael.miklav...@gmail.com> wrote: >> > >> > > From what I can tell, the data pruner isn't documented anywhere, so I'm >> > > curious if anybody is using this, and if so, how are you using it? >> > > >> > > - >> > > https://github.com/apache/metron/blob/master/metron- >> > > platform/metron-data-management/README.md >> > > - >> > > https://github.com/apache/metron/blob/master/metron- >> > > platform/metron-data-management/src/main/java/org/ >> > > apache/metron/dataloads/bulk/ElasticsearchDataPrunerRunner.java >> > > - >> > > https://github.com/apache/metron/blob/master/metron- >> > > platform/metron-data-management/src/main/java/org/ >> > > apache/metron/dataloads/bulk/DataPruner.java >> > > >> > > It looks to me that it allows you to specify the start date and a >> number >> > of >> > > days for lookback from the start date to purge along with a regex >> pattern >> > > to match the index name. It also does not look like it has any built-in >> > > scheduling semantics, so I assume this was a cron job. I think that >> about >> > > covers it. Anything I've missed? >> > > >> > > I'm adding a quick doc write-up to METRON-939 ( >> > > https://github.com/apache/metron/pull/840) for using Curator to prune >> > > indices from Elasticsearch. It is desirable to make sure I've covered >> > > existing use cases. >> > > >> > > Best, >> > > Mike >> > > >> > >> > >> > >> > -- >> > A.Nazemian >> > > > -- > A.Nazemian ------------------- Thank you, James Sirota PMC- Apache Metron jsirota AT apache DOT org
