I can post multiple items at a time. Data is being read from kafka and filtered after that its posted . Does foreachPartition load complete partition in memory or use an iterator of batch underhood? If compete batch is not loaded will using custim size of 100-200 request in one batch and post will help instead of whole partition ?
On Wed, Jul 22, 2015 at 12:18 AM, Tathagata Das <t...@databricks.com> wrote: > If you can post multiple items at a time, then use foreachPartition to > post the whole partition in a single request. > > On Tue, Jul 21, 2015 at 9:35 AM, Richard Marscher < > rmarsc...@localytics.com> wrote: > >> You can certainly create threads in a map transformation. We do this to >> do concurrent DB lookups during one stage for example. I would recommend, >> however, that you switch to mapPartitions from map as this allows you to >> create a fixed size thread pool to share across items on a partition as >> opposed to spawning a future per record in the RDD for example. >> >> On Tue, Jul 21, 2015 at 4:11 AM, Shushant Arora < >> shushantaror...@gmail.com> wrote: >> >>> Hi >>> >>> Can I create user threads in executors. >>> I have a streaming app where after processing I have a requirement to >>> push events to external system . Each post request costs ~90-100 ms. >>> >>> To make post parllel, I can not use same thread because that is limited >>> by no of cores available in system , can I useuser therads in spark App? I >>> tried to create 2 thredas in a map tasks and it worked. >>> >>> Is there any upper limit on no of user threds in spark executor ? Is it >>> a good idea to create user threads in spark map task? >>> >>> Thanks >>> >>> >> >> >> -- >> *Richard Marscher* >> Software Engineer >> Localytics >> Localytics.com <http://localytics.com/> | Our Blog >> <http://localytics.com/blog> | Twitter <http://twitter.com/localytics> | >> Facebook <http://facebook.com/localytics> | LinkedIn >> <http://www.linkedin.com/company/1148792?trk=tyah> >> > >
