Re: mysterious spark.sql.utils.AnalysisException Union in spark 3.3.2, but not seen in 3.4.0+

with allowMissingColumns set to > True. I suspect the reported union in the exception is probably inserted in > the plan by spark optimizer? > > spark.sql.utils.AnalysisException: Union can only be performed on tables > with the same number of columns, but the first table has 7 c

mysterious spark.sql.utils.AnalysisException Union in spark 3.3.2, but not seen in 3.4.0+

to True. I suspect the reported union in the exception is probably inserted in the plan by spark optimizer? spark.sql.utils.AnalysisException: Union can only be performed on tables with the same number of columns, but the first table has 7 columns and the second table has 6 columns The issue seems

Re:unresolved operator union in Spark 1.6

You can upgrade your Spark to version 2.x. At 2019-08-20 19:16:50, "马阳阳" wrote: Hi all, We encountered the following problem when using Spark 1.6. The following SQL: ``` SELECT a,b,c FROM A UNION ALL SELECT a,b,c FROM B ``` If A and B are two views created with

unresolved operator union in Spark 1.6

Hi all, We encountered the following problem when using Spark 1.6. The following SQL: ``` SELECT a,b,c FROM A UNION ALL SELECT a,b,c FROM B ``` If A and B are two views created with SQLContext.registerDataFrameAsTable(), Spark will fail with “unresolved operator Union”. But if A and B are

Unresolved operator Union in Spark 1.6

Hi all, We encountered the following problem when using Spark 1.6. The following SQL: ``` SELECT a,b,c FROM A UNION ALL SELECT a,b,c FROM B ``` If A and B are two views created with SQLContext.registerDataFrameAsTable(), Spark will fail with “unresolved operator Union”. But if A and

Re: Union in Spark

Hi Deep, What is your configuration and what is the size of the 2 data sets? Thanks Arush On Mon, Feb 2, 2015 at 11:56 AM, Deep Pradhan pradhandeep1...@gmail.com wrote: I did not check the console because once the job starts I cannot run anything else and have to force shutdown the system. I

Re: Union in Spark

The configuration is 16GB ram and 1TB HD. have a single node Spark cluster. Even after setting driver memory to 5g and executor memory to 3g, I get this error. The size of the data set is 350 KB and the set that it works well is hardly few KBs. On Mon, Feb 2, 2015 at 1:18 PM, Arush Kharbanda

Union in Spark

Hi, Is there any better operation than Union. I am using union and the cluster is getting stuck with a large data set. Thank you

Re: Union in Spark

Hi Deep, what do you mean by stuck? Jerry On Mon, Feb 2, 2015 at 12:44 AM, Deep Pradhan pradhandeep1...@gmail.com wrote: Hi, Is there any better operation than Union. I am using union and the cluster is getting stuck with a large data set. Thank you

Re: Union in Spark

Hi Deep, How do you know the cluster is not responsive because of Union? Did you check the spark web console? Best Regards, Jerry On Mon, Feb 2, 2015 at 1:21 AM, Deep Pradhan pradhandeep1...@gmail.com wrote: The cluster hangs. On Mon, Feb 2, 2015 at 11:25 AM, Jerry Lam

Re: Union in Spark

The cluster hangs. On Mon, Feb 2, 2015 at 11:25 AM, Jerry Lam chiling...@gmail.com wrote: Hi Deep, what do you mean by stuck? Jerry On Mon, Feb 2, 2015 at 12:44 AM, Deep Pradhan pradhandeep1...@gmail.com wrote: Hi, Is there any better operation than Union. I am using union and the

Re: Union in Spark

I did not check the console because once the job starts I cannot run anything else and have to force shutdown the system. I commented parts of codes and I tested. I doubt it is because of union. So, I want to change it to something else and see if the problem persists. Thank you On Mon, Feb 2,