[jira] [Commented] (SPARK-40233) Unable to load large pandas dataframe to pyspark

2022-08-31 Thread Niranda Perera (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17598566#comment-17598566 ] Niranda Perera commented on SPARK-40233: Well, the driver actually hangs after throwing that

[jira] [Commented] (SPARK-40233) Unable to load large pandas dataframe to pyspark

2022-08-31 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17598550#comment-17598550 ] Sean R. Owen commented on SPARK-40233: -- That's what happens, right? Spark is of course meant to

[jira] [Commented] (SPARK-40233) Unable to load large pandas dataframe to pyspark

2022-08-31 Thread Niranda Perera (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17598546#comment-17598546 ] Niranda Perera commented on SPARK-40233: [~srowen] shouldn't spark driver program terminate/

[jira] [Commented] (SPARK-40233) Unable to load large pandas dataframe to pyspark

2022-08-31 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17598524#comment-17598524 ] Sean R. Owen commented on SPARK-40233: -- This is more a problem with trying send a huge amount of

[jira] [Commented] (SPARK-40233) Unable to load large pandas dataframe to pyspark

2022-08-30 Thread Niranda Perera (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17598021#comment-17598021 ] Niranda Perera commented on SPARK-40233: I believe the issue is related to executors not being