Yes. I can do a just in time init… I can see that the first map was done.
However, I can’t see that the last map was done I think.. and the shutdown
is the key part. Without it all my daemon threads won’t properly exit and
I will not have all messages sent over the wire.
On Sun, Dec 28, 2014 at 12:18 AM, Akhil Das <ak...@sigmoidanalytics.com>
wrote:
> Something like?
>
> val a = myRDD.mapPartitions(p => {
>
>
>
> //Do the init
>
> //Perform some operations
>
> //Shut it down?
>
> })
>
>
>
> Thanks
> Best Regards
>
> On Sun, Dec 28, 2014 at 1:53 AM, Kevin Burton <bur...@spinn3r.com> wrote:
>
>> I have a job where I want to map over all data in a cassandra database.
>>
>> I’m then selectively sending things to my own external system (ActiveMQ)
>> if the item matches criteria.
>>
>> The problem is that I need to do some init and shutdown. Basically on
>> init I need to create ActiveMQ connections and on shutdown I need to close
>> them or daemon threads will be left running.
>>
>> What’s the best way to accomplish this. I could find it after I
>> RTFMd…(but perhaps I missed it)
>>
>> --
>>
>> Founder/CEO Spinn3r.com
>> Location: *San Francisco, CA*
>> blog: http://burtonator.wordpress.com
>> … or check out my Google+ profile
>> <https://plus.google.com/102718274791889610666/posts>
>> <http://spinn3r.com>
>>
>>
>
--
Founder/CEO Spinn3r.com
Location: *San Francisco, CA*
blog: http://burtonator.wordpress.com
… or check out my Google+ profile
<https://plus.google.com/102718274791889610666/posts>
<http://spinn3r.com>
