[ https://issues.apache.org/jira/browse/SPARK-41377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Apache Spark reassigned SPARK-41377: ------------------------------------ Assignee: Apache Spark > Fix spark-version-info.properties not found on Windows > ------------------------------------------------------ > > Key: SPARK-41377 > URL: https://issues.apache.org/jira/browse/SPARK-41377 > Project: Spark > Issue Type: Bug > Components: Spark Core > Affects Versions: 3.4.0 > Environment: Windows 10 > Reporter: Gautham Banasandra > Assignee: Apache Spark > Priority: Major > > h1. *What is the problem?* > The spark-version-info.properties files is read during initialization of > SparkContext. The absence of the same causes the Spark AM's launch to fail > with the following error - > {code} > 22/12/04 17:18:24 ERROR ApplicationMaster: User class threw exception: > java.lang.ExceptionInInitializerError > at org.apache.spark.package$.<init>(package.scala:93) > at org.apache.spark.package$.<clinit>(package.scala) > at org.apache.spark.SparkContext.$anonfun$new$1(SparkContext.scala:195) > at org.apache.spark.internal.Logging.logInfo(Logging.scala:60) > at org.apache.spark.internal.Logging.logInfo$(Logging.scala:59) > at org.apache.spark.SparkContext.logInfo(SparkContext.scala:84) > at org.apache.spark.SparkContext.<init>(SparkContext.scala:195) > at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2739) > at > org.apache.spark.sql.SparkSession$Builder.$anonfun$getOrCreate$2(SparkSession.scala:978) > at scala.Option.getOrElse(Option.scala:189) > at > org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:972) > at org.apache.spark.examples.SparkPi$.main(SparkPi.scala:30) > at org.apache.spark.examples.SparkPi.main(SparkPi.scala) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:747) > Caused by: org.apache.spark.SparkException: Could not find > spark-version-info.properties > at org.apache.spark.package$SparkBuildInfo$.<init>(package.scala:62) > at org.apache.spark.package$SparkBuildInfo$.<clinit>(package.scala) > ... 18 more > {code} > h1. *Why did the problem happen?* > While building Spark, the spark-build-info.properties file [is generated > using > bash|https://github.com/apache/spark/blob/d62c18b7497997188ec587e1eb62e75c979c1c93/core/pom.xml#L560-L564]. > In Windows environment, if Windows Subsystem for Linux (WSL) is installed, > it somehow overrides the other bash executables in the PATH, as noted in > SPARK-40739. The bash in WSL has a different mounting configuration and thus, > the target location specified for spark-build-info.properties won't be the > expected location. Ultimately, this leads to spark-build-info.properties to > get excluded from the spark-core jar, thus causing the SparkContext > initialization to fail with the above depicted error message. > h1. *How can the problem be solved?* > I propose to fix this issue by enhancing the build system to automatically > detect and switch between using Powershell for Windows and Bash for > non-Windows OS. This yields a seamless build experience for all the platform > users. Particularly for Windows users, to not worry about the conflicts with > different bash installations. -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org