Re: Processing huge amount of data from paged API

2018-01-21 Thread anonymous
The devices and device messages are retrieved using the APIs provided by company X (not the company's real name), which owns the IoT network. There is the option of setting HTTP POST callbacks for device messages, but we want to be able to run analytics on messages of ALL the devices of the

Processing huge amount of data from paged API

2018-01-21 Thread anonymous
Hello, I'm in an IoT company, and I have a use case for which I would like to know if Apache Spark could be helpful. It's a very broad question, and sorry if it's long winded. We have HTTP GET APIs to get two kinds of information: 1) The Device Messages API returns data about device messages (in