[ https://issues.apache.org/jira/browse/SPARK-37121?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hyukjin Kwon reassigned SPARK-37121: ------------------------------------ Assignee: Erik Krogen > TestUtils.isPythonVersionAtLeast38 returns incorrect results > ------------------------------------------------------------ > > Key: SPARK-37121 > URL: https://issues.apache.org/jira/browse/SPARK-37121 > Project: Spark > Issue Type: Bug > Components: Tests > Affects Versions: 3.2.0 > Reporter: Erik Krogen > Assignee: Erik Krogen > Priority: Major > > I was working on {{HiveExternalCatalogVersionsSuite}} recently and noticed > that it was never running against the Spark 2.x release lines, only the 3.x > ones. The problem was coming from here, specifically the Python 3.8+ version > check: > {code} > versions > .filter(v => v.startsWith("3") || !TestUtils.isPythonVersionAtLeast38()) > .filter(v => v.startsWith("3") || > !SystemUtils.isJavaVersionAtLeast(JavaVersion.JAVA_9)) > {code} > I found that {{TestUtils.isPythonVersionAtLeast38()}} was always returning > true, even when my system installation of Python3 was 3.7. Thinking it was an > environment issue, I pulled up a debugger to check which version of Python > the test JVM was seeing, and it was in fact Python 3.7. > Turns out the issue is with the {{isPythonVersionAtLeast38}} method: > {code} > def isPythonVersionAtLeast38(): Boolean = { > val attempt = if (Utils.isWindows) { > Try(Process(Seq("cmd.exe", "/C", "python3 --version")) > .run(ProcessLogger(s => s.startsWith("Python 3.8") || > s.startsWith("Python 3.9"))) > .exitValue()) > } else { > Try(Process(Seq("sh", "-c", "python3 --version")) > .run(ProcessLogger(s => s.startsWith("Python 3.8") || > s.startsWith("Python 3.9"))) > .exitValue()) > } > attempt.isSuccess && attempt.get == 0 > } > {code} > It's trying to evaluate the version of Python using a {{ProcessLogger}}, but > the logger accepts a {{String => Unit}} function, i.e., it does not make use > of the return value in any way (since it's meant for logging). So the result > of the {{startsWith}} checks are thrown away, and {{attempt.isSuccess && > attempt.get == 0}} will always be true as long as your system has a > {{python3}} binary of any version. -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org