[
https://issues.apache.org/jira/browse/SPARK-17386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon resolved SPARK-17386.
----------------------------------
Resolution: Incomplete
> Default polling and trigger intervals cause excessive RPC calls
> ---------------------------------------------------------------
>
> Key: SPARK-17386
> URL: https://issues.apache.org/jira/browse/SPARK-17386
> Project: Spark
> Issue Type: Bug
> Components: DStreams
> Reporter: Frederick Reiss
> Priority: Minor
> Labels: bulk-closed
>
> The default trigger interval for a Structured Streaming query is
> {{ProcessingTime(0)}}, i.e. "trigger new microbatches as fast as possible".
> When the trigger is set to this default value, the scheduler in
> {{StreamExecution}} will sit in a loop calling {{getOffset()}} every 10 msec
> (the default value of STREAMING_POLLING_DELAY) on every {{Source}} until new
> data arrives.
> In test cases, where most of the sources are {{MemoryStream}} or
> {{TextSocketSource}}, this rapid polling leads to excessive CPU usage.
> In a production environment, this overhead could disrupt critical
> infrastructure. Most sources in Spark clusters will be {{FileStreamSource}}
> or the not-yet-written Kafka 0.10 Source. The {{getOffset()}} method of
> {{FileStreamSource}} performs a directory listing of an HDFS directory. If no
> data has arrived, Spark will list the directory's contents up to 100 times
> per second. This overhead could disrupt service to other systems using HDFS,
> including Spark itself. A similar situation will exist with the Kafka source,
> the {{getOffset()}} method of which will presumably call Kafka's
> {{Consumer.poll()}} method.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
