You can certainly create threads in a map transformation. We do this to do concurrent DB lookups during one stage for example. I would recommend, however, that you switch to mapPartitions from map as this allows you to create a fixed size thread pool to share across items on a partition as opposed to spawning a future per record in the RDD for example.
On Tue, Jul 21, 2015 at 4:11 AM, Shushant Arora <shushantaror...@gmail.com> wrote: > Hi > > Can I create user threads in executors. > I have a streaming app where after processing I have a requirement to push > events to external system . Each post request costs ~90-100 ms. > > To make post parllel, I can not use same thread because that is limited by > no of cores available in system , can I useuser therads in spark App? I > tried to create 2 thredas in a map tasks and it worked. > > Is there any upper limit on no of user threds in spark executor ? Is it a > good idea to create user threads in spark map task? > > Thanks > > -- *Richard Marscher* Software Engineer Localytics Localytics.com <http://localytics.com/> | Our Blog <http://localytics.com/blog> | Twitter <http://twitter.com/localytics> | Facebook <http://facebook.com/localytics> | LinkedIn <http://www.linkedin.com/company/1148792?trk=tyah>
