David Ross created SPARK-6757: --------------------------------- Summary: spark.sql.shuffle.partitions is global, not per connection Key: SPARK-6757 URL: https://issues.apache.org/jira/browse/SPARK-6757 Project: Spark Issue Type: Bug Components: SQL Affects Versions: 1.3.0 Reporter: David Ross
We are trying to use the {{spark.sql.shuffle.partitions}} parameter to handle large queries differently from smaller queries. We expected that this parameter would be respected per connection, but it seems to be global. For example, in try this in two separate JDBC connections: Connection 1: {code} SET spark.sql.shuffle.partitions=10; SELECT * FROM some_table; {code} The correct number {{10}} was used. Connection 2: {code} SET spark.sql.shuffle.partitions=100; SELECT * FROM some_table; {code} The correct number {{100}} was used. Back to connection 1: {code} SELECT * FROM some_table; {code} We expected the number {{10}} to be used but {{100}} is used. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org