[
https://issues.apache.org/jira/browse/SPARK-28603?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Raja updated SPARK-28603:
-------------------------
Attachment: Batch_Inconsistencies.JPG
> Spark Streaming application receives inconsistent input events per batch
> interval
> ---------------------------------------------------------------------------------
>
> Key: SPARK-28603
> URL: https://issues.apache.org/jira/browse/SPARK-28603
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 1.6.3
> Reporter: Raja
> Priority: Major
> Attachments: Batch_Inconsistencies.JPG
>
> Original Estimate: 72h
> Remaining Estimate: 72h
>
> We have a 2 sec batch interval for a Spark Streaming application. The Spark
> is configured to receive from RabbitMQ queue and batch interval was chosen
> based on the resources available in the Cluster and the processing time taken
> without causing scheduling delays. For each run we have defined the
> MaxReceiverRate, BlockInterval and BackPressure enabled to deliver consistent
> performance for each batch.
> For example, the MaxReceiverRate was given "75", BlockInterval = 50ms and
> backPressure enabled, we expect for 2 sec batch - 150 msgs should be
> delivered for a batch to process. Most of the time we are able to achieve
> this performance, but except for few cases, where few batches will receive
> "0" events and a following batch receives say 3000 msgs (> greater than the
> maxReceiverRate). we are not sure of this unexpected behavior of the batch
> sizing, because of which our application is causing great scheduling delays
> because of which the application processing is unable to catch up to the
> incoming msg rates.
>
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
