Re: How to change Elasticsearch indexing policy

2017-07-14 Thread Ali Nazemian
Hi Simon, That's true for very large clusters. However, always having lots of shards and indices are not optimum for a small cluster. First, I think having hourly index cannot be one recipe for every cluster without considering the ingestion rate, cluster size and the type of query. Second, query

Re: How to change Elasticsearch indexing policy

2017-07-14 Thread Simon Elliston Ball
You could change the index data format. One word of caution here though; the last time I saw this done it caused huge problems with locking on ingest against people running queries on the current day’s data and tended to knock recent relevant indexes out of disk cache at the OS level. It might l

How to change Elasticsearch indexing policy

2017-07-14 Thread Ali Nazemian
Hi, I am investigating different tuning aspects, and I was wondering how I can change the policy of Elasticsearch indexing. Currently, as a default behaviour, events are stored in separate indices hourly. How can I change this behaviour? Is this a hard-coded design or I can change it through confi