Erik Krogen created SPARK-37121:
-----------------------------------
Summary: TestUtils.isPythonVersionAtLeast38 returns incorrect
results
Key: SPARK-37121
URL: https://issues.apache.org/jira/browse/SPARK-37121
Project: Spark
Issue Type: Bug
Components: Tests
Affects Versions: 3.2.0
Reporter: Erik Krogen
I was working on {{HiveExternalCatalogVersionsSuite}} recently and noticed that
it was never running against the Spark 2.x release lines, only the 3.x ones.
The problem was coming from here, specifically the Python 3.8+ version check:
{code}
versions
.filter(v => v.startsWith("3") || !TestUtils.isPythonVersionAtLeast38())
.filter(v => v.startsWith("3") ||
!SystemUtils.isJavaVersionAtLeast(JavaVersion.JAVA_9))
{code}
I found that {{TestUtils.isPythonVersionAtLeast38()}} was always returning
true, even when my system installation of Python3 was 3.7. Thinking it was an
environment issue, I pulled up a debugger to check which version of Python the
test JVM was seeing, and it was in fact Python 3.7.
Turns out the issue is with the {{isPythonVersionAtLeast38}} method:
{code}
def isPythonVersionAtLeast38(): Boolean = {
val attempt = if (Utils.isWindows) {
Try(Process(Seq("cmd.exe", "/C", "python3 --version"))
.run(ProcessLogger(s => s.startsWith("Python 3.8") ||
s.startsWith("Python 3.9")))
.exitValue())
} else {
Try(Process(Seq("sh", "-c", "python3 --version"))
.run(ProcessLogger(s => s.startsWith("Python 3.8") ||
s.startsWith("Python 3.9")))
.exitValue())
}
attempt.isSuccess && attempt.get == 0
}
{code}
It's trying to evaluate the version of Python using a {{ProcessLogger}}, but
the logger accepts a {{String => Unit}} function, i.e., it does not make use of
the return value in any way (since it's meant for logging). So the result of
the {{startsWith}} checks are thrown away, and {{attempt.isSuccess &&
attempt.get == 0}} will always be true as long as your system has a {{python3}}
binary of any version.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
