youngxinler created SPARK-39670: ----------------------------------- Summary: pyspark set "spark.hadoop.validateOutputSpecs" to false can't take effect Key: SPARK-39670 URL: https://issues.apache.org/jira/browse/SPARK-39670 Project: Spark Issue Type: Bug Components: PySpark Affects Versions: 3.1.2 Reporter: youngxinler
in pyspark, set "spark.hadoop.validateOutputSpecs" to false can't take effect example: spark = SparkSession.builder.config("spark.hadoop.validateOutputSpecs", "false").appName("PythonPi").getOrCreate() rdd.saveAsTextFile(file_exist_path) pyspark run will report error: |pyspark.sql.utils.AnalysisException: path hdfs://zjyprc-hadoop/user/s_ai_service/xxx already exists.| -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org