Hi guys,
I’d like to implement high-load linear scalable IoT
application using Apache Ignite. Sensors’ data comes from Apache Kafka topic
which is partitioned by sensor_id. As a primary storage the data is saved to
Cassandra. Application should allow to do the following:
* Streaming calculations and saving results to Cassandra
* Rapidly return the latest value for each sensor
* Getting the values of sensors that are changed during some time period
* Creation of data marts based on analytic queries (maybe via Spark queries
over Cassandra)
Data stream comes from Kafka and therefore it is automatically linearly
distributed between consumers in consumer group.
Here I have two questions:
* Is there a way to process and store the incoming (from Kafka) data on the
Ignite node, that receives the data? In other words, can I somehow synchronize
the partitioning of Kafka (which is done by message id) and partitioning of
Ignite (which is done by affinity functions), so that other nodes don’t bother
about sensors data that are being processed the Ignite node?
* As I need to rapidly return the latest value for some sensor via REST call,
what is the appropriate way of implementation of the HTTP load balancer, so
that the REST call is automatically balanced to the node, which contains the
data for the sensor?
--
Best Regards,
Mikhail
