Lewis, 1. Could you check the values of “SPARK_HOME” environment on all of your worker nodes? 2. How did you start your SparkR shell?
> On May 17, 2016, at 18:07, Mike Lewis <mle...@nephilaadvisors.co.uk> wrote: > > Hi, > > I have a SparkR driver process that connects to a master running on Linux, > I’ve tried to do a simple test, e.g. > > sc <- sparkR.init(master="spark://my-linux-host.dev.local:7077 > <spark://my-linux-host.dev.local:7077>", > sparkEnvir=list(spark.cores.max="4")) > x <- SparkR:::parallelize(sc,1:100,2) > y <- count(x) > > But I can see that the worker nodes are failing, they are looking for the > Windows (rather than linux path) to > Daemon.R > > 16/05/17 06:52:33 INFO BufferedStreamThread: Fatal error: cannot open file > 'E:/Dev/spark/spark-1.4.0-bin-hadoop2.6/R/lib/SparkR/worker/daemon.R': No > such file or directory > > Is this a configuration setting that I’m missing, the worker nodes (linux) > shouldn’t be looking in the spark home of the driver (windows) ? > If so, I’d appreciate someone letting me know what I need to change/set. > > Thanks, > Mike Lewis > > > -------------------------------------------------------------------------------------------------------------------------- > > This email has been sent to you on behalf of Nephila Advisors LLC > (“Advisors”). Advisors provides consultancy services to Nephila Capital Ltd. > (“Capital”), an investment advisor managed and carrying on business in > Bermuda. Advisors and its employees do not act as agents for Capital or the > funds it advises and do not have the authority to bind Capital or such funds > to any transaction or agreement. > > The information in this e-mail, and any attachment therein, is confidential > and for use by the addressee only. Any use, disclosure, reproduction, > modification or distribution of the contents of this e-mail, or any part > thereof, other than by the intended recipient, is strictly prohibited. If you > are not the intended recipient, please return the e-mail to the sender and > delete it from your computer. This email is for information purposes only, > nothing contained herein constitutes an offer to sell or buy securities, as > such an offer may only be made from a properly authorized offering document. > Although Nephila attempts to sweep e-mail and attachments for viruses, it > does not guarantee that either are virus-free and accepts no liability for > any damage sustained as a result of viruses. > -------------------------------------------------------------------------------------------------------------------------- > > > -------------------------------------------------------------------------------------------------------------------------- > > This email has been sent to you on behalf of Nephila Advisors UK (“Advisors > UK”). Advisors UK provides consultancy services to Nephila Capital Ltd. > (“Capital”), an investment advisor managed and carrying on business in > Bermuda. Advisors UK and its employees do not act as agents for Capital or > the funds it advises and do not have the authority to bind Capital or such > funds to any transaction or agreement. > > The information in this e-mail, and any attachment therein, is confidential > and for use by the addressee only. Any use, disclosure, reproduction, > modification or distribution of the contents of this e-mail, or any part > thereof, other than by the intended recipient, is strictly prohibited. If you > are not the intended recipient, please return the e-mail to the sender and > delete it from your computer. This email is for information purposes only, > nothing contained herein constitutes an offer to sell or buy securities, as > such an offer may only be made from a properly authorized offering document. > Although Nephila attempts to sweep e-mail and attachments for viruses, it > does not guarantee that either are virus-free and accepts no liability for > any damage sustained as a result of viruses. > --------------------------------------------------------------------------------------------------------------------------
