Hi All, thanks for the recommendations. I really like staying close to the Apache.org product line - so I will likely go the route that puts kafka streams or spark into play.
On Wed, Apr 10, 2019 at 11:45 AM Robin Moffatt <[email protected]> wrote: > +1 for looking into Kafka Streams. You can build and maintain state within > your app, and expose it on a REST endpoint for your node/react app to > query. > There's an example of this here: > > https://github.com/confluentinc/kafka-streams-examples/blob/5.2.1-post/src/main/java/io/confluent/examples/streams/interactivequeries/kafkamusic/KafkaMusicExample.java > > Depending on the processing you want to do KSQL could also be useful, > writing the results of queries (e.g. spotting exceptions in the data) to a > Kafka topic that can be queried by your app. > > > -- > > Robin Moffatt | Developer Advocate | [email protected] | @rmoff > > > On Tue, 9 Apr 2019 at 21:26, Nick Torenvliet <[email protected]> > wrote: > > > Hi all, > > > > Just looking for some general guidance. > > > > We have a kafka -> druid pipeline we intend to use in an industrial > setting > > to monitor process data. > > > > Our kafka system recieves messages on a single topic. > > > > The messages are {"timestamp": yy:mm:ddThh:mm:ss.mmm, > "plant_equipment_id": > > "id_string", "sensorvalue": float} > > > > For our POC there are about 2000 unique plant_equipment ids, this will > > quickly grow to 20,000. > > > > The kafka topic streams into druid > > > > We are building some node.js/react browser based apps for analytics and > > real time stream monitoring. > > > > We are thinking that for visualizing historical data sets we will hit > druid > > for data. > > > > For real time streaming we are wondering what our best option is. > > > > One option is to just hit druid semi regularly and update the on screen > > visualization as data arrives from there. > > > > Another option is to stream subset of the topics (somehow) from kafka > using > > some streams interface. > > > > With all the stock ticker apps out there, I have to imagine this is a > > really common use case. > > > > Anyone have any thoughts as to what we are best to do? > > > > Nick > > >
