Hello I have a use case that feels like a good fit for ElasticSearch except for one problem. I'm hoping someone might be able to suggest an approach for overcoming it using ElasticSearch.
I have a lot of time-series data from sensors. Extremely simplified, a reading looks a bit like this { "sensor_id": 12345678, "timestamp": 10203454354, "value": 5643 } I want to do things like calculate the average value for each sensor within date buckets for recent history. Thus far ElasticSearch seems like an excellent fit (using an approach similar to that described here: http://www.elasticsearch.com/guide/en/elasticsearch/guide/current/time-based.html) The problem is that I need the end user to be able to dynamically group sensors into 'categories' via a UI and then do aggregations and filtering based on that. ( eg 1: calculate the average value for each category of sensor within date buckets for recent history) ( eg 2: as above but filtered to only calculate for category A & B) If the user moves a particular sensor from one category to another, then the system should reflect that when calculating aggregations across previous readings. Some approaches I could take 1) re-index every time a user changes the category structure. This doesn't really seem feasible. 2) Resolve categories to sensor_ids in the application and use them to filter and bucket in ElasticSearch. Take the result from ElasticSearch and re-aggregate in the application. This seems problematic because A) There may be 1000s of sensor_ids in a category. The request payload could get quite large. B) It seems a shame to have to implement bucketing and aggregation in the app when I have ElasticSearch 3) Filter and Aggregate using a function that can map a sensor_id to a category for each reading. This would address problem B from approach 2, but a) the function would still be large if there are 1000s of sensor ids, and b) I am unsure of the performance implications of using functions this way. Has anyone done something like this with ElasticSearch? How? Cheers Perryn -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/1a2acbc4-e72e-488a-8ef3-36846d290b4c%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.