Thanks Akhil, your code is a big help to me,'cause perl script is the exactly thing i wanna try to run in spark. I will have a try.
-------------------------------- Thanks&Best regards! San.Luo ----- 原始邮件 ----- 发件人:Akhil Das <ak...@sigmoidanalytics.com> 收件人:罗辉 <luohui20...@sina.com> 抄送人:user <user@spark.apache.org> 主题:Re: how to distributed run a bash shell in spark 日期:2015年05月25日 00点53分 You mean you want to execute some shell commands from spark? Here's something i tried a while back. https://github.com/akhld/spark-exploitThanksBest Regards On Sun, May 24, 2015 at 4:53 PM, <luohui20...@sina.com> wrote: hello there I am trying to run a app in which part of it needs to run a shell.how to run a shell distributed in spark cluster.thanks. here's my code:import java.io.IOException; import java.util.ArrayList; import java.util.List; import org.apache.spark.SparkConf; import org.apache.spark.api.java.JavaRDD; import org.apache.spark.api.java.JavaSparkContext; import org.apache.spark.sql.DataFrame; import org.apache.spark.sql.Row; import org.apache.spark.sql.SQLContext; import org.apache.spark.sql.types.DataType; import org.apache.spark.sql.types.StructField; import org.apache.spark.sql.types.StructType; public class ShellCompare { public static void main(String[] args) { // TODO Auto-generated method stub SparkConf conf = new SparkConf().setAppName("ShellCompare").setMaster("spark://master:7077").set("spark.executor.memory", "6g"); JavaSparkContext sc = new JavaSparkContext(conf); for(int i=1;i<=21;i++){ execShell(i); } // execShell(1); sc.stop(); } private static void execShell(int i) { String shpath="/opt/sh/bin/sort.sh"; Process process =null; String var="/opt/data/shellcompare/chr" + i +".txt /opt/data/shellcompare/samplechr" + i +".txt /opt/data/shellcompare/result.txt 600"; // String var="/opt/data/chr1.txt /opt/data/chr1sample.txt /opt/sh/bin/result.txt 600"; String command2 = "sh " + shpath + " " + var; try { process = Runtime.getRuntime().exec(command2); process.waitFor(); } catch (InterruptedException e1) { // TODO Auto-generated catch block e1.printStackTrace(); } catch (IOException e) { // TODO Auto-generated catch block e.printStackTrace(); } } } -------------------------------- Thanks&Best regards! San.Luo