[ https://issues.apache.org/jira/browse/SPARK-15272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15570221#comment-15570221 ]
Cody Koeninger edited comment on SPARK-15272 at 10/12/16 11:33 PM: ------------------------------------------------------------------- Does the 0.10 consumer's handling of preferred locations http://spark.apache.org/docs/latest/streaming-kafka-0-10-integration.html#locationstrategies address this for you? was (Author: c...@koeninger.org): Checking to see if the 0.10 consumer's handling of preferred locations http://spark.apache.org/docs/latest/streaming-kafka-0-10-integration.html#locationstrategies addresses this. > DirectKafkaInputDStream doesn't work with window operation > ---------------------------------------------------------- > > Key: SPARK-15272 > URL: https://issues.apache.org/jira/browse/SPARK-15272 > Project: Spark > Issue Type: Bug > Components: Streaming > Affects Versions: 1.5.2 > Reporter: Lubomir Nerad > > Using Kafka direct {{DStream}} with simple window operation like: > {code:java} > kafkaDStream.window(Durations.milliseconds(10000), > Durations.milliseconds(1000)); > .print(); > {code} > with 1s batch duration either freezes after several seconds or lags terribly > (depending on cluster mode). > This happens when Kafka brokers are not part of the Spark cluster (they are > on different nodes). The {{KafkaRDD}} still reports them as preferred > locations. This doesn't seem to be problem in non-window scenarios but with > window it conflicts with delay scheduling algorithm implemented in > {{TaskSetManager}}. It either significantly delays (Yarn mode) or completely > drains (Spark mode) resource offers with {{TaskLocality.ANY}} which are > needed to process tasks with these Kafka broker aligned preferred locations. > When delay scheduling algorithm is switched off ({{spark.locality.wait=0}}), > the example works correctly. > I think that the {{KafkaRDD}} shouldn't report preferred locations if the > brokers don't correspond to worker nodes or allow the reporting of preferred > locations to be switched off. Also it would be good if delay scheduling > algorithm didn't drain / delay offers in the case, the tasks have unmatched > preferred locations. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org