Hi, You can use pipe operator, if you are running shell script/perl script on some data. More information on my blog <http://blog.madhukaraphatak.com/pipe-in-spark/>.
Regards, Madhukara Phatak http://datamantra.io/ On Mon, May 25, 2015 at 8:02 AM, <luohui20...@sina.com> wrote: > Thanks Akhil, > > your code is a big help to me,'cause perl script is the exactly > thing i wanna try to run in spark. I will have a try. > > > -------------------------------- > > Thanks&Best regards! > San.Luo > > ----- 原始邮件 ----- > 发件人：Akhil Das <ak...@sigmoidanalytics.com> > 收件人：罗辉 <luohui20...@sina.com> > 抄送人：user <user@spark.apache.org> > 主题：Re: how to distributed run a bash shell in spark > 日期：2015年05月25日 00点53分 > > You mean you want to execute some shell commands from spark? Here's > something i tried a while back. https://github.com/akhld/spark-exploit > > Thanks > Best Regards > > On Sun, May 24, 2015 at 4:53 PM, <luohui20...@sina.com> wrote: > > hello there > > I am trying to run a app in which part of it needs to run a > shell.how to run a shell distributed in spark cluster.thanks. > > > here's my code: > > import java.io.IOException; > import java.util.ArrayList; > import java.util.List; > > import org.apache.spark.SparkConf; > import org.apache.spark.api.java.JavaRDD; > import org.apache.spark.api.java.JavaSparkContext; > import org.apache.spark.sql.DataFrame; > import org.apache.spark.sql.Row; > import org.apache.spark.sql.SQLContext; > import org.apache.spark.sql.types.DataType; > import org.apache.spark.sql.types.StructField; > import org.apache.spark.sql.types.StructType; > > public class ShellCompare { > public static void main(String[] args) { > // TODO Auto-generated method stub > SparkConf conf = new > SparkConf().setAppName("ShellCompare").setMaster("spark://master:7077").set("spark.executor.memory", > "6g"); > JavaSparkContext sc = new JavaSparkContext(conf); > > for(int i=1;i<=21;i++){ > execShell(i); > } > // execShell(1); > sc.stop(); > } > > private static void execShell(int i) { > String shpath="/opt/sh/bin/sort.sh"; > Process process =null; > > String var="/opt/data/shellcompare/chr" + i +".txt > /opt/data/shellcompare/samplechr" + i +".txt > /opt/data/shellcompare/result.txt 600"; > // String var="/opt/data/chr1.txt /opt/data/chr1sample.txt > /opt/sh/bin/result.txt 600"; > String command2 = "sh " + shpath + " " + var; > try { > process = Runtime.getRuntime().exec(command2); > process.waitFor(); > } catch (InterruptedException e1) { > // TODO Auto-generated catch block > e1.printStackTrace(); > } catch (IOException e) { > // TODO Auto-generated catch block > e.printStackTrace(); > } > } > > } > > > > -------------------------------- > > Thanks&Best regards! > San.Luo > > >
