[ https://issues.apache.org/jira/browse/SPARK-23304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16349220#comment-16349220 ]
Thomas Graves commented on SPARK-23304: --------------------------------------- If it helps , spark 2.3 # partitions is 317531 and spark 2.2 is 166290 > Spark SQL coalesce() against hive not working > --------------------------------------------- > > Key: SPARK-23304 > URL: https://issues.apache.org/jira/browse/SPARK-23304 > Project: Spark > Issue Type: Bug > Components: SQL > Affects Versions: 2.3.0 > Reporter: Thomas Graves > Assignee: Xiao Li > Priority: Blocker > > Testing with spark 2.3 and I see a difference in the sql coalesce talking to > hive vs spark 2.2. It seems spark 2.3 ignores the coalesce. > > Query: > spark.sql("SELECT COUNT(DISTINCT(something)) FROM sometable WHERE dt >= > '20170301' AND dt <= '20170331' AND something IS NOT > NULL").coalesce(160000).show() > > in spark 2.2 the coalesce works here, but in spark 2.3, it doesn't. -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org