[ https://issues.apache.org/jira/browse/BEAM-673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17047688#comment-17047688 ]
Ismaël Mejía edited comment on BEAM-673 at 2/28/20 2:29 PM: ------------------------------------------------------------ Now that we have moved into a DoFn world with (or without) SplittableDoFn I wonder if we should change this issue to be consistent better with he ParDo API. Have something similar have been proposed as part of the ongoing work on portability? A mechanism for transforms to suggest translations to the runner as proposed in BEAM-2085 ? [~kenn] [~robertwb] [~lcwik] ? was (Author: iemejia): Now that we have moved into a DoFn world with (or without) SplittableDoFn I wonder if we should change this issue to be consistent better with he ParDo API. Have something similar have been proposed as part of the ongoing work on portability? A mechanism for transforms to suggest translations to the runner as proposed in BEAM-2085 ? [~robertwb] [~lcwik] ? > Data locality for Read.Bounded > ------------------------------ > > Key: BEAM-673 > URL: https://issues.apache.org/jira/browse/BEAM-673 > Project: Beam > Issue Type: Improvement > Components: runner-spark > Reporter: Amit Sela > Assignee: Ismaël Mejía > Priority: Major > > In some distributed filesystems, such as HDFS, we should be able to hint to > Spark the preferred locations of splits. > Here is an example of how Spark does that for Hadoop RDDs: > https://github.com/apache/spark/blob/branch-1.6/core/src/main/scala/org/apache/spark/rdd/NewHadoopRDD.scala#L249 -- This message was sent by Atlassian Jira (v8.3.4#803005)