[ https://issues.apache.org/jira/browse/SPARK-5137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14269263#comment-14269263 ]
Al M commented on SPARK-5137: ----------------------------- That's right. {code}a{code} has 11 partitions and {code}b{code} has a lot more. I can see why you wouldn't want to force a shuffle on {code}a{code} since that's unnecessary processing. Thanks for your detailed explanation and quick response. I'll close this since I agree that it behaves correctly. > subtract does not take the spark.default.parallelism into account > ----------------------------------------------------------------- > > Key: SPARK-5137 > URL: https://issues.apache.org/jira/browse/SPARK-5137 > Project: Spark > Issue Type: Bug > Affects Versions: 1.2.0 > Environment: CENTOS 6; scala > Reporter: Al M > Priority: Trivial > > The 'subtract' function (PairRDDFunctions.scala) in scala does not use the > default parallelism value set in the config (spark.default.parallelism). > This is easy enough to work around. I can just load the property and pass it > in as an argument. > It would be great if subtract used the default value, just like all the other > PairRDDFunctions. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org