[
https://issues.apache.org/jira/browse/SPARK-40284?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Liu updated SPARK-40284:
------------------------
Description:
We use Spark as a service. The same Spark service needs to handle multiple
requests, but I have a problem with this
When multiple requests are overwritten to a directory at the same time, the
results of two overwrite requests may be written successfully. I think this
does not meet the definition of overwrite write
First I ran Write SQL1, then I ran Write SQL2, and I found that both data were
written in the end, which I thought was unreasonable
{code:java}
sparkSession.udf.register("sleep", (time: Long) => Thread.sleep(time))
-- write sql1
sparkSession.sql("select 1 as id, sleep(40000) as
time").write.mode(SaveMode.Overwrite).parquet("path")
-- write sql2
sparkSession.sql("select 2 as id, 1 as
time").write.mode(SaveMode.Overwrite).parquet("path") {code}
When the spark source, and I saw that all these logic in
InsertIntoHadoopFsRelationCommand this class.
When the target directory already exists, Spark directly deletes the target
directory and writes to the _temporary directory that it requests. However,
when multiple requests are written, the data will all append in; For example,
in Write SQL above, this procedure occurs
1. Run write SQL 1, SQL 1 to create the _TEMPORARY directory, and continue
2. Run write SQL 2 to delete the entire target directory and create its own
_TEMPORARY
3. Sql2 writes data
4. SQL 1 completion. The corresponding _temporary /0/attemp_id directory does
not exist and fails. However, the task is retried, but the _temporary
directory is not deleted when the task is retried. Therefore, the execution
result of SQL1 is sent to the target directory by append
Based on the above process, the write process, can you do a directory check
before the write task or some other way to avoid this kind of problem
was:
We use Spark as a service. The same Spark service needs to handle multiple
requests, but I have a problem with this
When multiple requests are overwritten to a directory at the same time, the
results of two overwrite requests may be written successfully. I think this
does not meet the definition of overwrite write
First I ran Write SQL1, then I ran Write SQL2, and I found that both data were
written in the end, which I thought was unreasonable
{code:java}
sparkSession.udf.register("sleep", (time: Long) => Thread.sleep(time))
-- write sql1
sparkSession.sql("select 1 as id, sleep(40000) as
time").write.mode(SaveMode.Overwrite).parquet("path")
-- write sql2
sparkSession.sql("select 2 as id, 1 as
time").write.mode(SaveMode.Overwrite).parquet("path") {code}
When the spark source, and I saw that all these logic in
InsertIntoHadoopFsRelationCommand this class.
When the target directory already exists, Spark directly deletes the target
directory and writes to the _temporary directory that it requests. However,
when multiple requests are written, the data will all append in; For example,
in Write SQL above, this procedure occurs
1. Run write SQL 1, SQL 1 to create the _TEMPORARY directory, and continue
2. Run write SQL 2 to delete the entire target directory and create its own
_TEMPORARY
3. Sql2 writes data
4. SQL 1 completion. The corresponding _Temporary /0/attemp_id directory does
not exist and fails. However, the task is retried, but the directory is not
deleted when the task is retried. Therefore, the execution result of SQL1 is
sent to the target directory by append
Based on the above process, the write process, can you do a directory check
before the write task or some other way to avoid this kind of problem
> spark concurrent overwrite mode writes data to files in HDFS format, all
> request data write success
> ----------------------------------------------------------------------------------------------------
>
> Key: SPARK-40284
> URL: https://issues.apache.org/jira/browse/SPARK-40284
> Project: Spark
> Issue Type: Improvement
> Components: Input/Output
> Affects Versions: 3.0.1
> Reporter: Liu
> Priority: Major
>
> We use Spark as a service. The same Spark service needs to handle multiple
> requests, but I have a problem with this
> When multiple requests are overwritten to a directory at the same time, the
> results of two overwrite requests may be written successfully. I think this
> does not meet the definition of overwrite write
> First I ran Write SQL1, then I ran Write SQL2, and I found that both data
> were written in the end, which I thought was unreasonable
> {code:java}
> sparkSession.udf.register("sleep", (time: Long) => Thread.sleep(time))
> -- write sql1
> sparkSession.sql("select 1 as id, sleep(40000) as
> time").write.mode(SaveMode.Overwrite).parquet("path")
> -- write sql2
> sparkSession.sql("select 2 as id, 1 as
> time").write.mode(SaveMode.Overwrite).parquet("path") {code}
> When the spark source, and I saw that all these logic in
> InsertIntoHadoopFsRelationCommand this class.
>
> When the target directory already exists, Spark directly deletes the target
> directory and writes to the _temporary directory that it requests. However,
> when multiple requests are written, the data will all append in; For example,
> in Write SQL above, this procedure occurs
> 1. Run write SQL 1, SQL 1 to create the _TEMPORARY directory, and continue
> 2. Run write SQL 2 to delete the entire target directory and create its own
> _TEMPORARY
> 3. Sql2 writes data
> 4. SQL 1 completion. The corresponding _temporary /0/attemp_id directory does
> not exist and fails. However, the task is retried, but the _temporary
> directory is not deleted when the task is retried. Therefore, the execution
> result of SQL1 is sent to the target directory by append
>
> Based on the above process, the write process, can you do a directory check
> before the write task or some other way to avoid this kind of problem
>
>
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
