[ https://issues.apache.org/jira/browse/SPARK-23304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16353325#comment-16353325 ]
Wenchen Fan commented on SPARK-23304: ------------------------------------- I didn't follow the discussion here closely, but FYI, the document of `Dataset.coalesce` says: {code:java} * Returns a new Dataset that has exactly `numPartitions` partitions, when the fewer partitions * are requested. If a larger number of partitions is requested, it will stay at the current * number of partitions{code} > Spark SQL coalesce() against hive not working > --------------------------------------------- > > Key: SPARK-23304 > URL: https://issues.apache.org/jira/browse/SPARK-23304 > Project: Spark > Issue Type: Bug > Components: SQL > Affects Versions: 2.2.1, 2.3.0 > Reporter: Thomas Graves > Assignee: Xiao Li > Priority: Major > Attachments: spark22_oldorc_explain.txt, spark23_oldorc_explain.txt, > spark23_oldorc_explain_convermetastoreorcfalse.txt > > > The query below seems to ignore the coalesce. This is running spark 2.2 or > spark 2.3 against hive, which is reading orc: > > Query: > spark.sql("SELECT COUNT(DISTINCT(something)) FROM sometable WHERE dt >= > '20170301' AND dt <= '20170331' AND something IS NOT > NULL").coalesce(160000).show() > -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org