pan3793 commented on code in PR #6413:
URL: https://github.com/apache/kyuubi/pull/6413#discussion_r1610972251
##########
kyuubi-server/src/main/scala/org/apache/kyuubi/engine/spark/SparkProcessBuilder.scala:
##########
@@ -364,11 +364,14 @@ object SparkProcessBuilder {
final private val SPARK_CONF_FILE_NAME = "spark-defaults.conf"
final private[kyuubi] val SPARK_CORE_SCALA_VERSION_REGEX =
- """^spark-core_(\d\.\d+).*.jar$""".r
+ """^spark-core_(\d\.\d+)-.*\.jar$""".r
- final private[kyuubi] val SPARK_HOME_REGEX_SCALA_212 =
- """^spark-\d+\.\d+\.\d+-bin-hadoop\d+(\.\d+)?$""".r
+ final private[kyuubi] val SPARK3_HOME_REGEX_SCALA_212 =
+ """^spark-3\.\d+\.\d+-bin-hadoop\d+(\.\d+)?$""".r
- final private[kyuubi] val SPARK_HOME_REGEX_SCALA_213 =
- """^spark-\d+\.\d+\.\d+-bin-hadoop\d(\.\d+)?+-scala\d+(\.\d+)?$""".r
+ final private[kyuubi] val SPARK3_HOME_REGEX_SCALA_213 =
+ """^spark-3\.\d+\.\d+-bin-hadoop\d(\.\d+)?+-scala2\.13$""".r
+
+ final private[kyuubi] val SPARK4_HOME_REGEX_SCALA_213 =
+ """^spark-4\.\d+\.\d+(-\w*)?-bin-hadoop\d(\.\d+)?+$""".r
Review Comment:
SPARK_HOME detection should be conservative since it is only used for
testing purposes, and users/developers can always set SPARK_HOME explicitly to
override it. While `SPARK_CORE_SCALA_VERSION_REGEX` is used to detect the Scala
version, it should consider vender-provided Spark distributions too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
