Hi Guys, I have a random idea and it would be great to receive some input.
Can we have a HTTP2 Based receiver for Spark Streaming? I am wondering why not build micro services using Spark when needed? I can see it is not meant for that but I like to think it can be possible. To be more concrete, here is a scenario. It is a very common pattern in the industry that people build a service layer on top of a database (say Cassandra, HBASE or whatever). So all the service layer does is expose few service functions for the application via HTTP2 or RPC or some other protocol and the service layer is built simply by using the database driver underneath. After some point people will start thinking how to scale this service layer so they start introducing components service discovery, resource management and so on. I feel like I can get all this from Spark with added bonus such as leveraging the locality using the connectors written for a distributed database. Now I understand Spark Streaming may not be ready for this given its latency and mini-batch model but at some point I am confident that this latency issue will be addressed in which case I do see a path for this. what do you think? Thanks! kant