Raja created SPARK-28603:
----------------------------
Summary: Spark Streaming application receives inconsistent input
events per batch interval
Key: SPARK-28603
URL: https://issues.apache.org/jira/browse/SPARK-28603
Project: Spark
Issue Type: Bug
Components: Spark Core
Affects Versions: 1.6.3
Reporter: Raja
We have a 2 sec batch interval for a Spark Streaming application. The Spark is
configured to receive from RabbitMQ queue and batch interval was chosen based
on the resources available in the Cluster and the processing time taken without
causing scheduling delays. For each run we have defined the MaxReceiverRate,
BlockInterval and BackPressure enabled to deliver consistent performance for
each batch.
For example, the MaxReceiverRate was given "75", BlockInterval = 50ms and
backPressure enabled, we expect for 2 sec batch - 150 msgs should be delivered
for a batch to process. Most of the time we are able to achieve this
performance, but except for few cases, where few batches will receive "0"
events and a following batch receives say 3000 msgs (> greater than the
maxReceiverRate). we are not sure of this unexpected behavior of the batch
sizing, because of which our application is causing great scheduling delays
because of which the application processing is unable to catch up to the
incoming msg rates.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
