HI All, Right now I'm trying to execute a script using this command:
nohup $SPARK_HOME/bin/spark-shell < $HOME/my-script.scala > $HOME/my-script.log 2>&1 & my-script.scala just have 1 line of code: println("hallo world") But after waiting for a minute, I still don't receive the result from spark-shell. And the output log only contains: Spark assembly has been built with Hive, including Datanucleus jars on classpath SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/data1/hadoop/spark-1.0.0-bin-hadoop2/lib/spark-assembly-1.0.0-hadoop2.2.0.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/data1/hadoop/hbase-0.96.0-hadoop2/lib/slf4j-log4j12-1.6.4.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] 14/08/28 09:38:27 INFO spark.SecurityManager: Changing view acls to: hadoop 14/08/28 09:38:27 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(hadoop) 14/08/28 09:38:27 INFO spark.HttpServer: Starting HTTP Server 14/08/28 09:38:27 INFO server.Server: jetty-8.y.z-SNAPSHOT 14/08/28 09:38:27 INFO server.AbstractConnector: Started SocketConnector@0.0.0.0:49382<mailto:SocketConnector@0.0.0.0:49382> My question is: What is the right way to execute spark script? I really want to use spark-shell as a way to run cron job in the future, just like when I run R script. Best regards, Henry ________________________________ The privileged confidential information contained in this email is intended for use only by the addressees as indicated by the original sender of this email. If you are not the addressee indicated in this email or are not responsible for delivery of the email to such a person, please kindly reply to the sender indicating this fact and delete all copies of it from your computer and network server immediately. Your cooperation is highly appreciated. It is advised that any unauthorized use of confidential information of Winbond is strictly prohibited; and any information in this email irrelevant to the official business of Winbond shall be deemed as neither given nor endorsed by Winbond.