[
https://issues.apache.org/jira/browse/SPARK-38677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thomas Graves updated SPARK-38677:
----------------------------------
Description:
In spark 3.2.1 (spark 3.2.0 doesn't show this issue), pyspark will hang when
running and RDD map operations and converting to a dataframe. Code is below to
reproduce.
Env:
spark 3.2.1 local mode, just run {{./bin/pyspark --driver-memory XXXXG
--driver-cores XXXX}}
{{download dataset from here
[https://rapidsai-data.s3.us-east-2.amazonaws.com/spark/mortgage.zip]}}
{{just 200000 rows could reproduce the issue }}{{head -n 200000
mortgage_eval_merged.csv > mortgage_eval_merged-small.csv}}{{{} but if the
input dataset is small, such 50000 rows, it works well.{}}}{{{{}}{}}}run codes
below:
{code:java}
path = "/XXXX/mortgage_eval_merged-small.csv"
src_data = sc.textFile(path).map(lambda x:x.split(","))
column_list
=['c1','c2','c3','c4','c5','c6','c7','c8','c9','c10','c11','c12','c13','c14','c15','c16','c17','c18','c19','c20','c21','c22','c23','c24','c25','c26','c27','c28']
df = spark.createDataFrame(src_data,column_list)
print(df.show(1)){code}
was:
In spark 3.2.1 (spark 3.2.0 doesn't show this issue), pyspark will hang when
running and RDD map operations and converting to a dataframe. Code is below to
reproduce.
Env:
spark 3.2.1 local mode, just run {{./bin/pyspark --driver-memory XXXXG
--driver-cores XXXX}}
{{download dataset from here
[https://rapidsai-data.s3.us-east-2.amazonaws.com/spark/mortgage.zip]}}
{{just 200000 rows could reproduce the issue }}{{head -n 200000
mortgage_eval_merged.csv > mortgage_eval_merged-small.csv}}{{{} but if the
input dataset is small, such 50000 rows, it works well.{}}}{{{}{}}}run codes
below:
{code:java}
path = "/XXXX/mortgage_eval_merged-small.csv" src_data =
sc.textFile(path).map(lambda x:x.split(",")) column_list =
['c1','c2','c3','c4','c5','c6','c7','c8','c9','c10','c11','c12','c13','c14','c15','c16','c17','c18','c19','c20','c21','c22','c23','c24','c25','c26','c27','c28']
df = spark.createDataFrame(src_data,column_list) print(df.show(1)){code}
> pyspark hangs in local mode running rdd map operation
> -----------------------------------------------------
>
> Key: SPARK-38677
> URL: https://issues.apache.org/jira/browse/SPARK-38677
> Project: Spark
> Issue Type: Bug
> Components: PySpark
> Affects Versions: 3.2.1, 3.3.0
> Reporter: Thomas Graves
> Priority: Blocker
>
> In spark 3.2.1 (spark 3.2.0 doesn't show this issue), pyspark will hang when
> running and RDD map operations and converting to a dataframe. Code is below
> to reproduce.
> Env:
> spark 3.2.1 local mode, just run {{./bin/pyspark --driver-memory XXXXG
> --driver-cores XXXX}}
> {{download dataset from here
> [https://rapidsai-data.s3.us-east-2.amazonaws.com/spark/mortgage.zip]}}
> {{just 200000 rows could reproduce the issue }}{{head -n 200000
> mortgage_eval_merged.csv > mortgage_eval_merged-small.csv}}{{{} but if the
> input dataset is small, such 50000 rows, it works well.{}}}{{{{}}{}}}run
> codes below:
> {code:java}
> path = "/XXXX/mortgage_eval_merged-small.csv"
> src_data = sc.textFile(path).map(lambda x:x.split(","))
> column_list
> =['c1','c2','c3','c4','c5','c6','c7','c8','c9','c10','c11','c12','c13','c14','c15','c16','c17','c18','c19','c20','c21','c22','c23','c24','c25','c26','c27','c28']
> df = spark.createDataFrame(src_data,column_list)
> print(df.show(1)){code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
