Liu created SPARK-40284:
---------------------------
Summary: spark concurrent overwrite mode writes data to files in
HDFS format, all request data write success
Key: SPARK-40284
URL: https://issues.apache.org/jira/browse/SPARK-40284
Project: Spark
Issue Type: Improvement
Components: Input/Output
Affects Versions: 3.0.1
Reporter: Liu
We use Spark as a service. The same Spark service needs to handle multiple
requests, but I have a problem with this
When multiple requests are overwritten to a directory at the same time, the
results of two overwrite requests may be written successfully. I think this
does not meet the definition of overwrite write
First I ran Write SQL1, then I ran Write SQL2, and I found that both data were
written in the end, which I thought was unreasonable
{code:java}
sparkSession.udf.register("sleep", (time: Long) => Thread.sleep(time))
-- write sql1
sparkSession.sql("select 1 as id, sleep(40000) as
time").write.mode(SaveMode.Overwrite).parquet("path")
-- write sql2
sparkSession.sql("select 2 as id, 1 as
time").write.mode(SaveMode.Overwrite).parquet("path") {code}
When the spark source, and I saw that all these logic in
InsertIntoHadoopFsRelationCommand this class.
When the target directory already exists, Spark directly deletes the target
directory and writes to the _temporary directory that it requests. However,
when multiple requests are written, the data will all append in; For example,
in Write SQL above, this procedure occurs
1. Run write SQL 1, SQL 1 to create the _TEMPORARY directory, and continue
2. Run write SQL 2 to delete the entire target directory and create its own
_TEMPORARY
3. Sql2 writes data
4. SQL 1 completion. The corresponding _Temporary /0/attemp_id directory does
not exist and fails. However, the task is retried, but the directory is not
deleted when the task is retried. Therefore, the execution result of SQL1 is
sent to the target directory by append
Based on the above process, the write process, can you do a directory check
before the write task or some other way to avoid this kind of problem
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
