[
https://issues.apache.org/jira/browse/SPARK-44152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17746436#comment-17746436
]
Stijn De Haes commented on SPARK-44152:
---------------------------------------
We are seeing this issue too, the problem seems to be this PR:
[https://github.com/apache/spark/pull/37417]
When building an image we copy the jar's into the workdir location, however now
when the job is running spark will remove everything in that workdir location.
Resulting in this error. I am not sure on how to continue, what would be the
best location to copy the assembly jar?
> Upgrade to spark 3.4.0 from 3.3.2 gives Exception in thread "main"
> java.nio.file.NoSuchFileException: , although jar is present in the location
> -----------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-44152
> URL: https://issues.apache.org/jira/browse/SPARK-44152
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 3.4.0
> Reporter: Hyukjin Kwon
> Priority: Major
>
>
> I have a spark application that is deployed using k8s and it is of version
> 3.3.2 Recently there were some vulneabilities in spark 3.3.2
> I changed my dockerfile to download 3.4.0 instead of 3.3.2 and also my
> application jar is built on spark 3.4.0
> However while deploying, I get this error
>
> *{{Exception in thread "main" java.nio.file.NoSuchFileException:
> <path>/spark-assembly-1.0.jar}}*
>
> I have this in deployment.yaml of the app
>
> *mainApplicationFile: "local:///<path>/spark-assembly-1.0.jar"*
>
>
>
>
> and I have not changed anything related to that. I see that some code has
> changed in spark 3.4.0 core's source code regarding jar location.
> Has it really changed the functionality ? Is there anyone who is facing same
> issue as me ? Should the path be specified in a different way?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
