Arthur created SPARK-35008:
------------------------------
Summary: Spark hangs after on doing multiple unions
Key: SPARK-35008
URL: https://issues.apache.org/jira/browse/SPARK-35008
Project: Spark
Issue Type: Question
Components: SQL
Affects Versions: 3.1.1
Reporter: Arthur
I'm trying to union a list of {{Dataset<Rows>}} like so:
{{}}
{code:java}
totalSet = sparkSession.emptyDataset(RowEncoder.apply(schema));
for(var currentDataset : datasetList)
totalSet = totalSet.union(currentDataset);
totalSet.show(10000,false);
{code}
{{}}
There's about 160 datasets in datasetList. totalSet should only have 1k rows at
the end of the loop. However, spark gets stuck here in this loop. If I union
just 1 dataset from datasetList it works just fine. I thought doing unions
ought to be a O(1) operation, but that doesn't seem to be the case. Does anyone
know why this code hangs?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
