Re: Can't get SparkListener to work

will see the correct messages from your listener. Best Regards, Shixiong(Ryan) Zhu 2015-04-19 1:06 GMT+08:00 Praveen Balaji secondorderpolynom...@gmail.com : Thanks for the response, Archit. I get callbacks when I do not throw an exception from map. My use case, however, is to get

Re: Can't get SparkListener to work

...@gmail.com wrote: Hi Praveen, Can you try once removing throw exception in map. Do you still not get it.? On Apr 18, 2015 8:14 AM, Praveen Balaji secondorderpolynom...@gmail.com wrote: Thanks for the response, Imran. I probably chose the wrong methods for this email. I implemented all

Can't get SparkListener to work

I'm trying to create a simple SparkListener to get notified of error on executors. I do not get any call backs on my SparkListener. Here some simple code I'm executing in spark-shell. But I still don't get any callbacks on my listener. Am I doing something wrong? Thanks for any clue you can send

Re: Can't get SparkListener to work

Rashid iras...@cloudera.com wrote: when you start the spark-shell, its already too late to get the ApplicationStart event. Try listening for StageCompleted or JobEnd instead. On Fri, Apr 17, 2015 at 5:54 PM, Praveen Balaji secondorderpolynom...@gmail.com wrote: I'm trying to create a simple

Database operations on executor nodes

I was wondering what people generally do about doing database operations from executor nodes. I’m (at least for now) avoiding doing database updates from executor nodes to avoid proliferation of database connections on the cluster. The general pattern I adopt is to collect queries (or tuples)