When you say “receive messages” you mean acting as a REST endpoint, right? If so, it might be better to use JMS (or Kafka) option for a few reasons:
The receiver will be deployed to any of the available executors, so your REST clients will need to be made aware of the IP where the receiver is running (or you have some other proxy doing that). In the event the app restarts or the receiver dies and is restarted the executor the receiver runs on may change. Now there’s an option to specify a preference for what hosts to run the receiver on by overriding “preferredLocation” if you still decide to go that route. There’s a potential loss of messages when you are deploying or if your receiver or streaming app dies. You would now have to worry about managing those messages on the client’s side during a deployment or outage. There’s also a security issue (may not be relevant in your case) in that you would be exposing your executors and cluster in order to receive these messages. Worst case would be if the clients are outside your enterprise network. My preference would be to use JMS or Kafka or some other messaging systems as a buffer between the two systems. Thanks, Silvio From: Pavol Loffay Date: Thursday, October 1, 2015 at 3:58 PM To: "user@spark.apache.org<mailto:user@spark.apache.org>" Subject: Java REST custom receiver Hello, is it possible to implement custom receiver [1] which will receive messages from REST calls? As REST classes in Java(jax-rs) are defined declarative and instantiated by application server I'm not use if it is possible. I have tried to implement custom receiver which is inject to REST class via CDI and then is passed to JavaStreamingContext. But there is problem receiver instance in REST class is not the same as in SparkContext (supervisor is null). Could anyone help me with this? I'm also using JMS in my app so data from REST can be sent to JMS and then received by spark JMS receiver. But I think there should be more straight forward solution. [1]: https://spark.apache.org/docs/latest/api/java/
