[
https://issues.apache.org/jira/browse/SPARK-28603?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16899817#comment-16899817
]
Raja edited comment on SPARK-28603 at 8/5/19 6:42 AM:
------------------------------------------------------
Hi Dongjoon,
We are using the Datastax enterprise with Spark integrated with the DSE
install. As per the Datastax documentation, the latest version of DSE 6.7
supports Apache Spark 2.2.3.4. Just wanted to confirm that Spark support will
not be able to support any issues for Apache Spark 2.2.3.4 as well right.
Regards,
Raja.
was (Author: raja.boyangari):
Hi Dongjoon,
We are using the Datastax enterprise with Spark integrated with the DSE
install. As per the Datastax documentation, the latest version of DSE supports
2.2.3.4. Just wanted to confirm that Spark support will not be able to support
any issues for 2.2.3.4 as well right.
Regards,
Raja.
> Spark Streaming application receives inconsistent input events per batch
> interval
> ---------------------------------------------------------------------------------
>
> Key: SPARK-28603
> URL: https://issues.apache.org/jira/browse/SPARK-28603
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 1.6.3
> Reporter: Raja
> Priority: Major
> Attachments: Batch_Inconsistencies.JPG
>
> Original Estimate: 72h
> Remaining Estimate: 72h
>
> We have a 2 sec batch interval for a Spark Streaming application. The Spark
> is configured to receive from RabbitMQ queue and batch interval was chosen
> based on the resources available in the Cluster and the processing time taken
> without causing scheduling delays. For each run we have defined the
> MaxReceiverRate, BlockInterval and BackPressure enabled to deliver consistent
> performance for each batch.
> For example, the MaxReceiverRate was given "75", BlockInterval = 50ms and
> backPressure enabled, we expect for 2 sec batch - 150 msgs should be
> delivered for a batch to process. Most of the time we are able to achieve
> this performance, but except for few cases, where few batches will receive
> "0" events and a following batch receives say 3000 msgs (> greater than the
> maxReceiverRate). we are not sure of this unexpected behavior of the batch
> sizing, because of which our application is causing great scheduling delays
> because of which the application processing is unable to catch up to the
> incoming msg rates.
>
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
