[ https://issues.apache.org/jira/browse/SPARK-23945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16468045#comment-16468045 ]
Nicholas Chammas commented on SPARK-23945: ------------------------------------------ > So in the grand scheme of things I'd expect DataFrames to be able to do > everything that SQL can and vice versa Since writing this, I realized that the DataFrame API is able to express `IN` and `NOT IN` via an inner join and left anti join respectively. And I suspect most other cases where I may have thought the DataFrame API is less powerful than SQL are incorrect. The various DataFrame join types basically cover a lot of the stuff you'd want to do with subqueries. So I'd actually be fine with closing this out as "Won't Fix" and instructing users, in the particular example I provided above, to express their query as follows: {code:java} (table1 .join( table2, on='name', how='left_anti', ) ){code} This is equivalent to the SQL query I posted, and does not require that anything be collected locally, so it scales just as well. [~hvanhovell] - Does this make sense? > Column.isin() should accept a single-column DataFrame as input > -------------------------------------------------------------- > > Key: SPARK-23945 > URL: https://issues.apache.org/jira/browse/SPARK-23945 > Project: Spark > Issue Type: Improvement > Components: SQL > Affects Versions: 2.3.0 > Reporter: Nicholas Chammas > Priority: Minor > > In SQL you can filter rows based on the result of a subquery: > {code:java} > SELECT * > FROM table1 > WHERE name NOT IN ( > SELECT name > FROM table2 > );{code} > In the Spark DataFrame API, the equivalent would probably look like this: > {code:java} > (table1 > .where( > ~col('name').isin( > table2.select('name') > ) > ) > ){code} > However, .isin() currently [only accepts a local list of > values|http://spark.apache.org/docs/latest/api/python/pyspark.sql.html#pyspark.sql.Column.isin]. > I imagine making this enhancement would happen as part of a larger effort to > support correlated subqueries in the DataFrame API. > Or perhaps there is no plan to support this style of query in the DataFrame > API, and queries like this should instead be written in a different way? How > would we write a query like the one I have above in the DataFrame API, > without needing to collect values locally for the NOT IN filter? > -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org