Repository: spark Updated Branches: refs/heads/master ad0dadaa2 -> b6a7aa4f7
[SPARK-19221][PROJECT INFRA][R] Add winutils binaries to the path in AppVeyor tests for Hadoop libraries to call native codes properly ## What changes were proposed in this pull request? It seems Hadoop libraries need winutils binaries for native libraries in the path. It is not a problem in tests for now because we are only testing SparkR on Windows via AppVeyor but it can be a problem if we run Scala tests via AppVeyor as below: ``` - SPARK-18220: read Hive orc table with varchar column *** FAILED *** (3 seconds, 937 milliseconds) org.apache.spark.sql.execution.QueryExecutionException: FAILED: Execution Error, return code -101 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask. org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(Ljava/lang/String;I)Z at org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$runHive$1.apply(HiveClientImpl.scala:625) at org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$runHive$1.apply(HiveClientImpl.scala:609) at org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$withHiveState$1.apply(HiveClientImpl.scala:283) ... ``` This PR proposes to add it to the `Path` for AppVeyor tests. ## How was this patch tested? Manually via AppVeyor. **Before** https://ci.appveyor.com/project/spark-test/spark/build/549-windows-complete/job/gc8a1pjua2bc4i8m **After** https://ci.appveyor.com/project/spark-test/spark/build/572-windows-complete/job/c4vrysr5uvj2hgu7 Author: hyukjinkwon <gurwls...@gmail.com> Closes #16584 from HyukjinKwon/set-path-appveyor. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/b6a7aa4f Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/b6a7aa4f Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/b6a7aa4f Branch: refs/heads/master Commit: b6a7aa4f770634e6db7244e88f8b6273fb9b6d1e Parents: ad0dada Author: hyukjinkwon <gurwls...@gmail.com> Authored: Sat Jan 14 08:31:07 2017 -0800 Committer: Shivaram Venkataraman <shiva...@cs.berkeley.edu> Committed: Sat Jan 14 08:31:07 2017 -0800 ---------------------------------------------------------------------- dev/appveyor-install-dependencies.ps1 | 1 + 1 file changed, 1 insertion(+) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/spark/blob/b6a7aa4f/dev/appveyor-install-dependencies.ps1 ---------------------------------------------------------------------- diff --git a/dev/appveyor-install-dependencies.ps1 b/dev/appveyor-install-dependencies.ps1 index 087b866..b72d6b5 100644 --- a/dev/appveyor-install-dependencies.ps1 +++ b/dev/appveyor-install-dependencies.ps1 @@ -109,6 +109,7 @@ Invoke-Expression "7z.exe x winutils-master.zip" # add hadoop bin to environment variables $env:HADOOP_HOME = "$hadoopPath/winutils-master/hadoop-$hadoopVer" +$env:Path += ";$env:HADOOP_HOME\bin" Pop-Location --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
