This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a commit to branch branch-3.0 in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.0 by this push: new 4e13ba9 [SPARK-31238][SPARK-31284][TEST][FOLLOWUP] Fix readResourceOrcFile to create a local file from resource 4e13ba9 is described below commit 4e13ba90446745fc5a9f46ed1f80c6eefb738795 Author: Dongjoon Hyun <dongj...@apache.org> AuthorDate: Fri Mar 27 18:44:53 2020 -0700 [SPARK-31238][SPARK-31284][TEST][FOLLOWUP] Fix readResourceOrcFile to create a local file from resource ### What changes were proposed in this pull request? This PR aims to copy a test resource file to a local file in `OrcTest` suite before reading it. ### Why are the changes needed? SPARK-31238 and SPARK-31284 added test cases to access the resouce file in `sql/core` module from `sql/hive` module. In **Maven** test environment, this causes a failure. ``` - SPARK-31238: compatibility with Spark 2.4 in reading dates *** FAILED *** java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative path in absolute URI: jar:file:/home/jenkins/workspace/spark-master-test-maven-hadoop-3.2-hive-2.3-jdk-11/sql/core/target/spark-sql_2.12-3.1.0-SNAPSHOT-tests.jar!/test-data/before_1582_date_v2_4.snappy.orc ``` ``` - SPARK-31284: compatibility with Spark 2.4 in reading timestamps *** FAILED *** java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative path in absolute URI: jar:file:/home/jenkins/workspace/spark-master-test-maven-hadoop-3.2-hive-2.3/sql/core/target/spark-sql_2.12-3.1.0-SNAPSHOT-tests.jar!/test-data/before_1582_ts_v2_4.snappy.orc ``` ### Does this PR introduce any user-facing change? No ### How was this patch tested? Pass the Jenkins with Maven. Closes #28059 from dongjoon-hyun/SPARK-31238. Authored-by: Dongjoon Hyun <dongj...@apache.org> Signed-off-by: Dongjoon Hyun <dongj...@apache.org> (cherry picked from commit d025ddbaa7e7b9746d8e47aeed61ed39d2f09f0e) Signed-off-by: Dongjoon Hyun <dongj...@apache.org> --- .../org/apache/spark/sql/execution/datasources/orc/OrcTest.scala | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/orc/OrcTest.scala b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/orc/OrcTest.scala index 16772fe..e929f90 100644 --- a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/orc/OrcTest.scala +++ b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/orc/OrcTest.scala @@ -22,6 +22,7 @@ import java.io.File import scala.reflect.ClassTag import scala.reflect.runtime.universe.TypeTag +import org.apache.commons.io.FileUtils import org.scalatest.BeforeAndAfterAll import org.apache.spark.sql._ @@ -136,6 +137,10 @@ abstract class OrcTest extends QueryTest with FileBasedDataSourceTest with Befor protected def readResourceOrcFile(name: String): DataFrame = { val url = Thread.currentThread().getContextClassLoader.getResource(name) - spark.read.orc(url.toString) + // Copy to avoid URISyntaxException when `sql/hive` accesses the resources in `sql/core` + val file = File.createTempFile("orc-test", ".orc") + file.deleteOnExit(); + FileUtils.copyURLToFile(url, file) + spark.read.orc(file.getAbsolutePath) } } --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org