[
https://issues.apache.org/jira/browse/SPARK-31698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Viacheslav Tradunsky updated SPARK-31698:
-----------------------------------------
Environment: AWS EMR: 30 machines, 7TB RAM total. (was: AWS EMR: 30
machine, 7TB RAM total.)
> NPE on big dataset plans
> ------------------------
>
> Key: SPARK-31698
> URL: https://issues.apache.org/jira/browse/SPARK-31698
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 2.4.4
> Environment: AWS EMR: 30 machines, 7TB RAM total.
> Reporter: Viacheslav Tradunsky
> Priority: Major
> Attachments: Spark_NPE_big_dataset.log
>
>
> We have big dataset containing 275 SQL operations more than 275 joins.
> On the terminal operation to write data, it fails with NullPointerException.
>
> I understand that such big number of operations might not be what spark is
> designed for, but NullPointerException is not an ideal way to fail in this
> case.
>
> For more details, please see the stacktrace.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
