Re: Using elasticsearch as a realtime fire hose

2014-08-26 Thread Jilles van Gurp
You might want to look at developing a plugin for this or maybe using an existing one. This one for example might do partly what you need: https://github.com/derryx/elasticsearch-changes-plugin If you develop your own plugin, you should be able to tap into what is happening in the cluster at a

Re: Using elasticsearch as a realtime fire hose

2014-08-25 Thread Jim Alateras
What kind of events do you think of? Single new document indexed? Batch of docs indexed? Node-wide? Or cluster wide? event on whenever a document is added to an index cluster wide You mention Redis, for something like publish/subscribe pattern, you'd have to use a persistent

Using elasticsearch as a realtime fire hose

2014-08-23 Thread Jim Alateras
I was wondering whether there were any mechanisms to use ES as a realtime feed for downstream systems. I have a cluster that gathers observations from many sensors. I have a need to maintain a list of realtime counters in REDIS so I want to further process these observation once they hit the

Re: Using elasticsearch as a realtime fire hose

2014-08-23 Thread joergpra...@gmail.com
What kind of events do you think of? Single new document indexed? Batch of docs indexed? Node-wide? Or cluster wide? You mention Redis, for something like publish/subscribe pattern, you'd have to use a persistent connection and implement your own ES actions, which is possible with e.g. HTTP