Paul Praet created SPARK-40388: ---------------------------------- Summary: SQL configuration spark.sql.mapKeyDedupPolicy not always applied Key: SPARK-40388 URL: https://issues.apache.org/jira/browse/SPARK-40388 Project: Spark Issue Type: Bug Components: SQL Affects Versions: 3.1.2 Reporter: Paul Praet
I have set spark.sql.mapKeyDedupPolicy to LAST_WIN. However, I had still one failure where I got {quote}Caused by: org.apache.spark.SparkException: Job aborted due to stage failure: Task 7 in stage 1201.0 failed 4 times, most recent failure: Lost task 7.3 in stage 1201.0 (TID 1011313) (ip-10-1-34-47.eu-west-1.compute.internal executor 228): java.lang.RuntimeException: Duplicate map key domain was found, please check the input data. If you want to remove the duplicated keys, you can set spark.sql.mapKeyDedupPolicy to LAST_WIN so that the key inserted at last takes precedence. {quote} We are confident we set the right configuration in SparkConf (we can find it on the Spark UI -> Environment). It is our impression this configuration is not propagated reliably to the executors. -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org