[ https://issues.apache.org/jira/browse/SPARK-21853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16144697#comment-16144697 ]
Shailesh Kini commented on SPARK-21853: --------------------------------------- As a work around, I saved the dataset DS3 into parquet format and read it again after which I was able to successfully call the except. > Getting an exception while calling the except method on the dataframe > --------------------------------------------------------------------- > > Key: SPARK-21853 > URL: https://issues.apache.org/jira/browse/SPARK-21853 > Project: Spark > Issue Type: Question > Components: Spark Shell > Affects Versions: 2.1.1 > Reporter: Shailesh Kini > Attachments: SparkException.txt > > > I am getting an exception while calling except on the Dataset. > org.apache.spark.sql.AnalysisException: resolved attribute(s) > SVC_BILLING_PERIOD#37723 missing from > I read 2 csv files into datasets DS1 and DS2, which I join (full outer) to > create DS3. DS3 has some rows which are similar with the exception of one > column. I need to isolate those rows and remove the similar rows. I use > groupBy with the count > 1 on a few columns in DS3 to get those similar rows > - dataset DS4. DS4 has only a few columns and not all so I join it back with > DS3 on the aggregate columns to get a new dataset DS5 which has the same > columns as DS3. To get a clean dataset without any of those similar rows, I > am calling DS3.except(DS5) which throws the exception. The attribute is one > of the filtering criteria I use which creating DS1. > Attaching the exception to this ticket. -- This message was sent by Atlassian JIRA (v6.4.14#64029) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org