I guess that you are using an old version of Spark, 1.4. please try Spark version 1.5+
> On May 17, 2016, at 18:42, Mike Lewis <mle...@nephilaadvisors.co.uk> wrote: > > Thanks, I’m just using RStudio. Running locally is fine, just issue with > having cluster in Linux and workers looking for Windows path, > Which must be being passed through by the driver I guess. I checked the > spark-env.sh on each node and the appropriate SPARK_HOME is set > correctly…. > > > From: Sun Rui [mailto:sunrise_...@163.com] > Sent: 17 May 2016 11:32 > To: Mike Lewis > Cc: user@spark.apache.org > Subject: Re: SparkR query > > Lewis, > 1. Could you check the values of “SPARK_HOME” environment on all of your > worker nodes? > 2. How did you start your SparkR shell? > > On May 17, 2016, at 18:07, Mike Lewis <mle...@nephilaadvisors.co.uk > <mailto:mle...@nephilaadvisors.co.uk>> wrote: > > Hi, > > I have a SparkR driver process that connects to a master running on Linux, > I’ve tried to do a simple test, e.g. > > sc <- sparkR.init(master="spark://my-linux-host.dev.local:7077 > <spark://my-linux-host.dev.local:7077>", > sparkEnvir=list(spark.cores.max="4")) > x <- SparkR:::parallelize(sc,1:100,2) > y <- count(x) > > But I can see that the worker nodes are failing, they are looking for the > Windows (rather than linux path) to > Daemon.R > > 16/05/17 06:52:33 INFO BufferedStreamThread: Fatal error: cannot open file > 'E:/Dev/spark/spark-1.4.0-bin-hadoop2.6/R/lib/SparkR/worker/daemon.R': No > such file or directory > > Is this a configuration setting that I’m missing, the worker nodes (linux) > shouldn’t be looking in the spark home of the driver (windows) ? > If so, I’d appreciate someone letting me know what I need to change/set. > > Thanks, > Mike Lewis > > > -------------------------------------------------------------------------------------------------------------------------- > > This email has been sent to you on behalf of Nephila Advisors LLC > (“Advisors”). Advisors provides consultancy services to Nephila Capital Ltd. > (“Capital”), an investment advisor managed and carrying on business in > Bermuda. Advisors and its employees do not act as agents for Capital or the > funds it advises and do not have the authority to bind Capital or such funds > to any transaction or agreement. > > The information in this e-mail, and any attachment therein, is confidential > and for use by the addressee only. Any use, disclosure, reproduction, > modification or distribution of the contents of this e-mail, or any part > thereof, other than by the intended recipient, is strictly prohibited. If you > are not the intended recipient, please return the e-mail to the sender and > delete it from your computer. This email is for information purposes only, > nothing contained herein constitutes an offer to sell or buy securities, as > such an offer may only be made from a properly authorized offering document. > Although Nephila attempts to sweep e-mail and attachments for viruses, it > does not guarantee that either are virus-free and accepts no liability for > any damage sustained as a result of viruses. > -------------------------------------------------------------------------------------------------------------------------- > > > -------------------------------------------------------------------------------------------------------------------------- > > This email has been sent to you on behalf of Nephila Advisors UK (“Advisors > UK”). Advisors UK provides consultancy services to Nephila Capital Ltd. > (“Capital”), an investment advisor managed and carrying on business in > Bermuda. Advisors UK and its employees do not act as agents for Capital or > the funds it advises and do not have the authority to bind Capital or such > funds to any transaction or agreement. > > The information in this e-mail, and any attachment therein, is confidential > and for use by the addressee only. Any use, disclosure, reproduction, > modification or distribution of the contents of this e-mail, or any part > thereof, other than by the intended recipient, is strictly prohibited. If you > are not the intended recipient, please return the e-mail to the sender and > delete it from your computer. This email is for information purposes only, > nothing contained herein constitutes an offer to sell or buy securities, as > such an offer may only be made from a properly authorized offering document. > Although Nephila attempts to sweep e-mail and attachments for viruses, it > does not guarantee that either are virus-free and accepts no liability for > any damage sustained as a result of viruses. > -------------------------------------------------------------------------------------------------------------------------- > > > -------------------------------------------------------------------------------------------------------------------------- > > This email has been sent to you on behalf of Nephila Advisors LLC > (“Advisors”). Advisors provides consultancy services to Nephila Capital Ltd. > (“Capital”), an investment advisor managed and carrying on business in > Bermuda. Advisors and its employees do not act as agents for Capital or the > funds it advises and do not have the authority to bind Capital or such funds > to any transaction or agreement. > > The information in this e-mail, and any attachment therein, is confidential > and for use by the addressee only. Any use, disclosure, reproduction, > modification or distribution of the contents of this e-mail, or any part > thereof, other than by the intended recipient, is strictly prohibited. If you > are not the intended recipient, please return the e-mail to the sender and > delete it from your computer. This email is for information purposes only, > nothing contained herein constitutes an offer to sell or buy securities, as > such an offer may only be made from a properly authorized offering document. > Although Nephila attempts to sweep e-mail and attachments for viruses, it > does not guarantee that either are virus-free and accepts no liability for > any damage sustained as a result of viruses. > -------------------------------------------------------------------------------------------------------------------------- > > > -------------------------------------------------------------------------------------------------------------------------- > > This email has been sent to you on behalf of Nephila Advisors UK (“Advisors > UK”). Advisors UK provides consultancy services to Nephila Capital Ltd. > (“Capital”), an investment advisor managed and carrying on business in > Bermuda. Advisors UK and its employees do not act as agents for Capital or > the funds it advises and do not have the authority to bind Capital or such > funds to any transaction or agreement. > > The information in this e-mail, and any attachment therein, is confidential > and for use by the addressee only. Any use, disclosure, reproduction, > modification or distribution of the contents of this e-mail, or any part > thereof, other than by the intended recipient, is strictly prohibited. If you > are not the intended recipient, please return the e-mail to the sender and > delete it from your computer. This email is for information purposes only, > nothing contained herein constitutes an offer to sell or buy securities, as > such an offer may only be made from a properly authorized offering document. > Although Nephila attempts to sweep e-mail and attachments for viruses, it > does not guarantee that either are virus-free and accepts no liability for > any damage sustained as a result of viruses. > --------------------------------------------------------------------------------------------------------------------------