[ https://issues.apache.org/jira/browse/SPARK-5137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14269263#comment-14269263 ]
Al M edited comment on SPARK-5137 at 1/8/15 12:30 PM: ------------------------------------------------------ That's right. "a" has 11 partitions and "b" has a lot more. I can see why you wouldn't want to force a shuffle on "a" since that's unnecessary processing. Thanks for your detailed explanation and quick response. I'll close this since I agree that it behaves correctly. was (Author: alrocks47): That's right. _emphasis_a_emphasis_ has 11 partitions and _emphasis_b_emphasis_ has a lot more. I can see why you wouldn't want to force a shuffle on _emphasis_a_emphasis_ since that's unnecessary processing. Thanks for your detailed explanation and quick response. I'll close this since I agree that it behaves correctly. > subtract does not take the spark.default.parallelism into account > ----------------------------------------------------------------- > > Key: SPARK-5137 > URL: https://issues.apache.org/jira/browse/SPARK-5137 > Project: Spark > Issue Type: Bug > Affects Versions: 1.2.0 > Environment: CENTOS 6; scala > Reporter: Al M > Priority: Trivial > > The 'subtract' function (PairRDDFunctions.scala) in scala does not use the > default parallelism value set in the config (spark.default.parallelism). > This is easy enough to work around. I can just load the property and pass it > in as an argument. > It would be great if subtract used the default value, just like all the other > PairRDDFunctions. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org