Re: init / shutdown for complex map job?
Something like? val a = myRDD.mapPartitions(p = { //Do the init //Perform some operations //Shut it down? }) Thanks Best Regards On Sun, Dec 28, 2014 at 1:53 AM, Kevin Burton bur...@spinn3r.com wrote: I have a job where I want to map over all data in a cassandra database. I’m then selectively sending things to my own external system (ActiveMQ) if the item matches criteria. The problem is that I need to do some init and shutdown. Basically on init I need to create ActiveMQ connections and on shutdown I need to close them or daemon threads will be left running. What’s the best way to accomplish this. I could find it after I RTFMd…(but perhaps I missed it) -- Founder/CEO Spinn3r.com Location: *San Francisco, CA* blog: http://burtonator.wordpress.com … or check out my Google+ profile https://plus.google.com/102718274791889610666/posts http://spinn3r.com
Re: init / shutdown for complex map job?
You can't quite do cleanup in mapPartitions in that way. Here is a bit more explanation (farther down): http://blog.cloudera.com/blog/2014/09/how-to-translate-from-mapreduce-to-apache-spark/ On Dec 28, 2014 8:18 AM, Akhil Das ak...@sigmoidanalytics.com wrote: Something like? val a = myRDD.mapPartitions(p = { //Do the init //Perform some operations //Shut it down? }) Thanks Best Regards On Sun, Dec 28, 2014 at 1:53 AM, Kevin Burton bur...@spinn3r.com wrote: I have a job where I want to map over all data in a cassandra database. I’m then selectively sending things to my own external system (ActiveMQ) if the item matches criteria. The problem is that I need to do some init and shutdown. Basically on init I need to create ActiveMQ connections and on shutdown I need to close them or daemon threads will be left running. What’s the best way to accomplish this. I could find it after I RTFMd…(but perhaps I missed it) -- Founder/CEO Spinn3r.com Location: *San Francisco, CA* blog: http://burtonator.wordpress.com … or check out my Google+ profile https://plus.google.com/102718274791889610666/posts http://spinn3r.com
Re: init / shutdown for complex map job?
A follow-up to the blog cited below was hinted at, per But Wait, There's More ... To keep this post brief, the remainder will be left to a follow-up post. Is this follow-up pending? Is it sort of pending? Did the follow-up happen, but I just couldn't find it on the web? Regards, Ray. On Sun, 28 Dec 2014 08:54:13 + Sean Owen so...@cloudera.com wrote: You can't quite do cleanup in mapPartitions in that way. Here is a bit more explanation (farther down): http://blog.cloudera.com/blog/2014/09/how-to-translate-from-mapreduce-to-apache-spark/ On Dec 28, 2014 8:18 AM, Akhil Das ak...@sigmoidanalytics.com wrote: Something like? val a = myRDD.mapPartitions(p = { //Do the init //Perform some operations //Shut it down? }) - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
Re: init / shutdown for complex map job?
(Still pending, but believe it's in progress and being written by a colleague here.) On Sun, Dec 28, 2014 at 2:41 PM, Ray Melton rtmel...@gmail.com wrote: A follow-up to the blog cited below was hinted at, per But Wait, There's More ... To keep this post brief, the remainder will be left to a follow-up post. Is this follow-up pending? Is it sort of pending? Did the follow-up happen, but I just couldn't find it on the web? Regards, Ray. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
init / shutdown for complex map job?
I have a job where I want to map over all data in a cassandra database. I’m then selectively sending things to my own external system (ActiveMQ) if the item matches criteria. The problem is that I need to do some init and shutdown. Basically on init I need to create ActiveMQ connections and on shutdown I need to close them or daemon threads will be left running. What’s the best way to accomplish this. I could find it after I RTFMd…(but perhaps I missed it) -- Founder/CEO Spinn3r.com Location: *San Francisco, CA* blog: http://burtonator.wordpress.com … or check out my Google+ profile https://plus.google.com/102718274791889610666/posts http://spinn3r.com