can i process multiple batch in parallel in spark streaming
Hi when i run my spark streaming application .. following information show on application streaming UI. i am using spark 1.5.0 Batch Time Input Size Scheduling Delay (?) Processing Time (?) Status 2015/12/09 11:00:42 107 events - - queued 2015/12/09 11:00:41 103 events - - queued 2015/12/09 11:00:40 107 events - - queued 2015/12/09 11:00:39 105 events - - queued 2015/12/09 11:00:38 109 events - - queued 2015/12/09 11:00:37 106 events - - queued 2015/12/09 11:00:36 109 events - - queued 2015/12/09 11:00:35 113 events - - queued 2015/12/09 11:00:34 109 events - - queued 2015/12/09 11:00:33 107 events - - queued 2015/12/09 11:00:32 99 events 42 s- processing it seems batches push into queue and work like FIFO manner . is it possible all my Active batches start processing in parallel. Regards Prateek -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/can-i-process-multiple-batch-in-parallel-in-spark-streaming-tp25653.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
Re: can i process multiple batch in parallel in spark streaming
Hi Thanks In my scenario batches are independent .so is it safe to use in production environment ? Regards Prateek On Wed, Dec 9, 2015 at 11:39 AM, Ted Yu <yuzhih...@gmail.com> wrote: > Have you seen this thread ? > > http://search-hadoop.com/m/q3RTtgSGrobJ3Je > > On Wed, Dec 9, 2015 at 11:12 AM, prateek arora <prateek.arora...@gmail.com > > wrote: > >> Hi >> >> when i run my spark streaming application .. following information show on >> application streaming UI. >> i am using spark 1.5.0 >> >> >> Batch Time Input Size Scheduling Delay (?) Processing Time >> (?) >> Status >> 2015/12/09 11:00:42 107 events - - >>queued >> 2015/12/09 11:00:41 103 events - - >>queued >> 2015/12/09 11:00:40 107 events - - >>queued >> 2015/12/09 11:00:39 105 events - - >>queued >> 2015/12/09 11:00:38 109 events - - >>queued >> 2015/12/09 11:00:37 106 events - - >>queued >> 2015/12/09 11:00:36 109 events - - >>queued >> 2015/12/09 11:00:35 113 events - - >>queued >> 2015/12/09 11:00:34 109 events - - >>queued >> 2015/12/09 11:00:33 107 events - - >>queued >> 2015/12/09 11:00:32 99 events 42 s- >>processing >> >> >> >> it seems batches push into queue and work like FIFO manner . is it >> possible >> all my Active batches start processing in parallel. >> >> Regards >> Prateek >> >> >> >> -- >> View this message in context: >> http://apache-spark-user-list.1001560.n3.nabble.com/can-i-process-multiple-batch-in-parallel-in-spark-streaming-tp25653.html >> Sent from the Apache Spark User List mailing list archive at Nabble.com. >> >> - >> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org >> For additional commands, e-mail: user-h...@spark.apache.org >> >> >
Re: can i process multiple batch in parallel in spark streaming
Have you seen this thread ? http://search-hadoop.com/m/q3RTtgSGrobJ3Je On Wed, Dec 9, 2015 at 11:12 AM, prateek arora <prateek.arora...@gmail.com> wrote: > Hi > > when i run my spark streaming application .. following information show on > application streaming UI. > i am using spark 1.5.0 > > > Batch Time Input Size Scheduling Delay (?) Processing Time > (?) > Status > 2015/12/09 11:00:42 107 events - - > queued > 2015/12/09 11:00:41 103 events - - > queued > 2015/12/09 11:00:40 107 events - - > queued > 2015/12/09 11:00:39 105 events - - > queued > 2015/12/09 11:00:38 109 events - - > queued > 2015/12/09 11:00:37 106 events - - > queued > 2015/12/09 11:00:36 109 events - - > queued > 2015/12/09 11:00:35 113 events - - > queued > 2015/12/09 11:00:34 109 events - - > queued > 2015/12/09 11:00:33 107 events - - > queued > 2015/12/09 11:00:32 99 events 42 s- > processing > > > > it seems batches push into queue and work like FIFO manner . is it > possible > all my Active batches start processing in parallel. > > Regards > Prateek > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/can-i-process-multiple-batch-in-parallel-in-spark-streaming-tp25653.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > - > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > For additional commands, e-mail: user-h...@spark.apache.org > >
