If you can post multiple items at a time, then use foreachPartition to post the whole partition in a single request.
On Tue, Jul 21, 2015 at 9:35 AM, Richard Marscher <rmarsc...@localytics.com> wrote: > You can certainly create threads in a map transformation. We do this to do > concurrent DB lookups during one stage for example. I would recommend, > however, that you switch to mapPartitions from map as this allows you to > create a fixed size thread pool to share across items on a partition as > opposed to spawning a future per record in the RDD for example. > > On Tue, Jul 21, 2015 at 4:11 AM, Shushant Arora <shushantaror...@gmail.com > > wrote: > >> Hi >> >> Can I create user threads in executors. >> I have a streaming app where after processing I have a requirement to >> push events to external system . Each post request costs ~90-100 ms. >> >> To make post parllel, I can not use same thread because that is limited >> by no of cores available in system , can I useuser therads in spark App? I >> tried to create 2 thredas in a map tasks and it worked. >> >> Is there any upper limit on no of user threds in spark executor ? Is it a >> good idea to create user threads in spark map task? >> >> Thanks >> >> > > > -- > *Richard Marscher* > Software Engineer > Localytics > Localytics.com <http://localytics.com/> | Our Blog > <http://localytics.com/blog> | Twitter <http://twitter.com/localytics> | > Facebook <http://facebook.com/localytics> | LinkedIn > <http://www.linkedin.com/company/1148792?trk=tyah> >
