ive high loads for a sustained amount of
>> time, you want to implement some sort of autoscaling that adds nodes to
>> your cluster, and increases partitioning of the data. Autoscaling cannot
>> react fast enough for momentary spikes, but it can prevent your system from
&g
s to your
> cluster, and increases partitioning of the data. Autoscaling cannot react
> fast enough for momentary spikes, but it can prevent your system from being
> overwhelmed with sustained high loads
>
>
>
> *From: *Mich Talebzadeh
> *Date: *Friday, March 26, 2021 at 1:
enough for
momentary spikes, but it can prevent your system from being overwhelmed with
sustained high loads
From: Mich Talebzadeh
Date: Friday, March 26, 2021 at 1:44 PM
To: "user @spark"
Subject: [EXTERNAL] The trigger interval in spark structured streaming
CAUTION: This email
One thing I noticed is that when the trigger interval in foreachBatch is
set to something low (in this case 2 seconds, equivalent to the batch
interval that source sends data to Kafka topic (every 2 seconds)
trigger(processingTime='2 seconds')
Spark sends the warning that the queue is falling
