Re: Serial batching with Spark Streaming

2015-06-20 Thread Tathagata Das
wrote: Is it possible to achieve serial batching with Spark Streaming? Example: I configure the Streaming Context for creating a batch every 3 seconds. Processing of the batch #2 takes longer than 3 seconds and creates a backlog of batches: batch #1 takes 2s batch #2 takes

Re: Serial batching with Spark Streaming

2015-06-20 Thread Michal Čizmazia
if you are setting any SparkConf configurations? On Wed, Jun 17, 2015 at 12:22 PM, Michal Čizmazia mici...@gmail.com wrote: Is it possible to achieve serial batching with Spark Streaming? Example: I configure the Streaming Context for creating a batch every 3 seconds. Processing

Re: Serial batching with Spark Streaming

2015-06-19 Thread Michal Čizmazia
1.4.0, could you show us a screenshot of the streaming tab, especially the list of batches? And could you also tell us if you are setting any SparkConf configurations? On Wed, Jun 17, 2015 at 12:22 PM, Michal Čizmazia mici...@gmail.com wrote: Is it possible to achieve serial batching

Re: Serial batching with Spark Streaming

2015-06-19 Thread Tathagata Das
wrote: Is it possible to achieve serial batching with Spark Streaming? Example: I configure the Streaming Context for creating a batch every 3 seconds. Processing of the batch #2 takes longer than 3 seconds and creates a backlog of batches: batch #1 takes 2s batch #2 takes 10s

Re: Serial batching with Spark Streaming

2015-06-19 Thread Michal Čizmazia
SparkConf configurations? On Wed, Jun 17, 2015 at 12:22 PM, Michal Čizmazia mici...@gmail.com wrote: Is it possible to achieve serial batching with Spark Streaming? Example: I configure the Streaming Context for creating a batch every 3 seconds. Processing of the batch #2 takes

Re: Serial batching with Spark Streaming

2015-06-18 Thread Binh Nguyen Van
tell us if you are setting any SparkConf configurations? On Wed, Jun 17, 2015 at 12:22 PM, Michal Čizmazia mici...@gmail.com wrote: Is it possible to achieve serial batching with Spark Streaming? Example: I configure the Streaming Context for creating a batch every 3 seconds

Re: Serial batching with Spark Streaming

2015-06-18 Thread Michal Čizmazia
configurations? On Wed, Jun 17, 2015 at 12:22 PM, Michal Čizmazia mici...@gmail.com wrote: Is it possible to achieve serial batching with Spark Streaming? Example: I configure the Streaming Context for creating a batch every 3 seconds. Processing of the batch #2 takes longer than 3 seconds

Re: Serial batching with Spark Streaming

2015-06-18 Thread Michal Čizmazia
you also tell us if you are setting any SparkConf configurations? On Wed, Jun 17, 2015 at 12:22 PM, Michal Čizmazia mici...@gmail.com wrote: Is it possible to achieve serial batching with Spark Streaming? Example: I configure the Streaming Context for creating a batch every 3

Serial batching with Spark Streaming

2015-06-17 Thread Michal Čizmazia
Is it possible to achieve serial batching with Spark Streaming? Example: I configure the Streaming Context for creating a batch every 3 seconds. Processing of the batch #2 takes longer than 3 seconds and creates a backlog of batches: batch #1 takes 2s batch #2 takes 10s batch #3 takes 2s batch

Re: Serial batching with Spark Streaming

2015-06-17 Thread Tathagata Das
, Jun 17, 2015 at 12:22 PM, Michal Čizmazia mici...@gmail.com wrote: Is it possible to achieve serial batching with Spark Streaming? Example: I configure the Streaming Context for creating a batch every 3 seconds. Processing of the batch #2 takes longer than 3 seconds and creates a backlog