[ https://issues.apache.org/jira/browse/SPARK-37185?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17437009#comment-17437009 ]
mathieu longtin commented on SPARK-37185: ----------------------------------------- Additional note: if there's a "group by" in the query, this is not an issue. > DataFrame.take() only uses one worker > ------------------------------------- > > Key: SPARK-37185 > URL: https://issues.apache.org/jira/browse/SPARK-37185 > Project: Spark > Issue Type: Bug > Components: SQL > Affects Versions: 3.1.1, 3.2.0 > Environment: CentOS 7 > Reporter: mathieu longtin > Priority: Major > > Say you have query: > {code:java} > >>> df = spark.sql("select * from mytable where x = 99"){code} > Now, out of billions of row, there's only ten rows where x is 99. > If I do: > {code:java} > >>> df.limit(10).collect() > [Stage 1:> (0 + 1) / 1]{code} > It only uses one worker. This takes a really long time since one CPU is > reading the billions of row. > However, if I do this: > {code:java} > >>> df.limit(10).rdd.collect() > [Stage 1:> (0 + 10) / 22]{code} > All the workers are running. > I think there's some optimization issue DataFrame.take(...). > This did not use to be an issue, but I'm not sure if it was working with 3.0 > or 2.4. -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org