Hi Konstatin,
We use hadoop as a library in a few places in Spark. I wonder why the path
includes "null" though.
Could you provide the full stack trace?
Andrew
2014-07-02 9:38 GMT-07:00 Konstantin Kudryavtsev <
kudryavtsev.konstan...@gmail.com>:
> Hi all,
>
> I'm trying to run some transformation on *Spark*, it works fine on
> cluster (YARN, linux machines). However, when I'm trying to run it on local
> machine (*Windows 7*) under unit test, I got errors:
>
> java.io.IOException: Could not locate executable null\bin\winutils.exe in the
> Hadoop binaries.
> at org.apache.hadoop.util.Shell.getQualifiedBinPath(Shell.java:318)
> at org.apache.hadoop.util.Shell.getWinUtilsPath(Shell.java:333)
> at org.apache.hadoop.util.Shell.<clinit>(Shell.java:326)
> at org.apache.hadoop.util.StringUtils.<clinit>(StringUtils.java:76)
> at org.apache.hadoop.security.Groups.parseStaticMapping(Groups.java:93)
>
>
> My code is following:
>
> @Test
> def testETL() = {
> val conf = new SparkConf()
> val sc = new SparkContext("local", "test", conf)
> try {
> val etl = new IxtoolsDailyAgg() // empty constructor
>
> val data = sc.parallelize(List("in1", "in2", "in3"))
>
> etl.etl(data) // rdd transformation, no access to SparkContext or
> Hadoop
> Assert.assertTrue(true)
> } finally {
> if(sc != null)
> sc.stop()
> }
> }
>
>
> Why is it trying to access hadoop at all? and how can I fix it? Thank you
> in advance
>
> Thank you,
> Konstantin Kudryavtsev
>
