Hi, I have a continuous rest api stream which keeps spitting out data in form of json. I access the stream using python requests.get(url, stream=True, headers=headers).
I want to receive them using spark and do further processing. I am not sure which is best way to receive it in spark. What are the options i have. Some options i can think of 1. push data from rest api stream in to kakfa queue and use spark kafka streaming utilities to capture data and further process. 2. push data from rest api stream to a local socket and use spark socket stream utilities to capture data and further process. 3. is there any other way to receive it? thanks VR