[jira] [Updated] (SPARK-28302) SparkLauncher: The process cannot access the file because it is being used by another process
[ https://issues.apache.org/jira/browse/SPARK-28302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wuyi updated SPARK-28302: - Description: When using SparkLauncher to submit applications concurrently with a thread pool under *Windows*, some apps would show that "The process cannot access the file because it is being used by another process" and remains in LOST state at the end. (Issue can be reproduced with attach file.) After digging into the code, I find that, Windows cmd %RANDOM% would return the same number if we call it instantly( < 500ms) after last call. As a result, SparkLauncher would get same output file(spark-class-launcher-output-%RANDOM%.txt) for apps. Then, the following app would hit the issue when it try to write the same file which has already been occupied by another app. was: When using SparkLauncher to submit applications concurrently with a thread pool under *Windows*, some apps would show that "The process cannot access the file because it is being used by another process" and remains in LOST state at the end. Issue can be reproduced with attach file. After digging into the code, I find that, Windows cmd %RANDOM% would return the same number if we call it instantly( < 500ms) after last call. As a result, SparkLauncher would get same output file(spark-class-launcher-output-%RANDOM%.txt) for apps. Then, the following app would hit the issue when it try to write the same file which has already been occupied by another app. > SparkLauncher: The process cannot access the file because it is being used by > another process > - > > Key: SPARK-28302 > URL: https://issues.apache.org/jira/browse/SPARK-28302 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 2.4.0 > Environment: Windows 7 >Reporter: wuyi >Priority: Major > Attachments: Main.scala > > > When using SparkLauncher to submit applications concurrently with a thread > pool under *Windows*, some apps would show that "The process cannot access > the file because it is being used by another process" and remains in LOST > state at the end. (Issue can be reproduced with attach file.) > > After digging into the code, I find that, Windows cmd %RANDOM% would return > the same number if we call it instantly( < 500ms) after last call. As a > result, SparkLauncher would get same output > file(spark-class-launcher-output-%RANDOM%.txt) for apps. Then, the following > app would hit the issue when it try to write the same file which has already > been occupied by another app. > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-28302) SparkLauncher: The process cannot access the file because it is being used by another process
[ https://issues.apache.org/jira/browse/SPARK-28302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wuyi updated SPARK-28302: - Attachment: Main.scala > SparkLauncher: The process cannot access the file because it is being used by > another process > - > > Key: SPARK-28302 > URL: https://issues.apache.org/jira/browse/SPARK-28302 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 2.4.0 > Environment: Windows 7 >Reporter: wuyi >Priority: Major > Attachments: Main.scala > > > When using SparkLauncher to submit applications concurrently with a thread > pool under *Windows*, some apps would show that "The process cannot access > the file because it is being used by another process" and remains in LOST > state at the end. > > After digging into the code, I find that, Windows cmd %RANDOM% would return > the same number if we call it instantly( < 500ms) after last call. As a > result, SparkLauncher would get same output > file(spark-class-launcher-output-%RANDOM%.txt) for apps. Then, the following > app would hit the issue when it try to write the same file which has already > been occupied by another app. > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-28302) SparkLauncher: The process cannot access the file because it is being used by another process
[ https://issues.apache.org/jira/browse/SPARK-28302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wuyi updated SPARK-28302: - Description: When using SparkLauncher to submit applications concurrently with a thread pool under *Windows*, some apps would show that "The process cannot access the file because it is being used by another process" and remains in LOST state at the end. Issue can be reproduced with attach file. After digging into the code, I find that, Windows cmd %RANDOM% would return the same number if we call it instantly( < 500ms) after last call. As a result, SparkLauncher would get same output file(spark-class-launcher-output-%RANDOM%.txt) for apps. Then, the following app would hit the issue when it try to write the same file which has already been occupied by another app. was: When using SparkLauncher to submit applications concurrently with a thread pool under *Windows*, some apps would show that "The process cannot access the file because it is being used by another process" and remains in LOST state at the end. After digging into the code, I find that, Windows cmd %RANDOM% would return the same number if we call it instantly( < 500ms) after last call. As a result, SparkLauncher would get same output file(spark-class-launcher-output-%RANDOM%.txt) for apps. Then, the following app would hit the issue when it try to write the same file which has already been occupied by another app. > SparkLauncher: The process cannot access the file because it is being used by > another process > - > > Key: SPARK-28302 > URL: https://issues.apache.org/jira/browse/SPARK-28302 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 2.4.0 > Environment: Windows 7 >Reporter: wuyi >Priority: Major > Attachments: Main.scala > > > When using SparkLauncher to submit applications concurrently with a thread > pool under *Windows*, some apps would show that "The process cannot access > the file because it is being used by another process" and remains in LOST > state at the end. Issue can be reproduced with attach file. > > After digging into the code, I find that, Windows cmd %RANDOM% would return > the same number if we call it instantly( < 500ms) after last call. As a > result, SparkLauncher would get same output > file(spark-class-launcher-output-%RANDOM%.txt) for apps. Then, the following > app would hit the issue when it try to write the same file which has already > been occupied by another app. > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
