[
https://issues.apache.org/jira/browse/SPARK-21853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shailesh Kini updated SPARK-21853:
----------------------------------
Attachment: SparkException.txt
> Getting an exception while calling the except method on the dataframe
> ---------------------------------------------------------------------
>
> Key: SPARK-21853
> URL: https://issues.apache.org/jira/browse/SPARK-21853
> Project: Spark
> Issue Type: Question
> Components: Spark Shell
> Affects Versions: 2.1.1
> Reporter: Shailesh Kini
> Attachments: SparkException.txt
>
>
> I am getting an exception while calling except on the Dataset.
> org.apache.spark.sql.AnalysisException: resolved attribute(s)
> SVC_BILLING_PERIOD#37723 missing from
> I have 2 csv. I create two dataset DS1 and DS2, which I join to create DS3. I
> need to filter out duplicates for further processing. I aggregate on DS3
> dataset by on some columns and filter when the count > 1. This is DS4. I now
> join DS3 with DS4 on those columns and get DS5. DS5 has the same structure as
> DS3 as I drop the columns from the join. DS5 now has all the rows which are
> duplicate. I then call the except on DS3 to get me a dataset DS5 which all
> the rows not in DS5. I am planning to filter and remove of of the duplicates
> (all the columns are not duplicates so I need to use filter) and union it
> with DS6 to get the dataset free of duplicates.
> Attaching the exception to this ticket.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
