[ https://issues.apache.org/jira/browse/PIG-4797?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
liyunzhang_intel updated PIG-4797: ---------------------------------- Description: There are a big performance difference in join between spark and mr mode. {code} daily = load './NYSE_daily' as (exchange:chararray, symbol:chararray, date:chararray, open:float, high:float, low:float, close:float, volume:int, adj_close:float); divs = load './NYSE_dividends' as (exchange:chararray, symbol:chararray, date:chararray, dividends:float); jnd = join daily by (exchange, symbol), divs by (exchange, symbol); store jnd into './join.out'; {code} join.sh {code} mode=$1 start=$(date +%s) ./pig -x $mode $PIG_HOME/bin/join.pig end=$(date +%s) execution_time=$(( $end - $start )) echo "execution_time:"$excution_time {code} The execution time: || |||mr||spark|| |join|20 sec|79 sec| You can download the test data NYSE_daily and NYSE_dividends in https://github.com/alanfgates/programmingpig/blob/master/data/ was: There are a big performance difference in join between spark and mr mode. {code} daily = load './NYSE_daily' as (exchange:chararray, symbol:chararray, date:chararray, open:float, high:float, low:float, close:float, volume:int, adj_close:float); divs = load './NYSE_dividends' as (exchange:chararray, symbol:chararray, date:chararray, dividends:float); jnd = join daily by (exchange, symbol), divs by (exchange, symbol); store jnd into './join.out'; {code} join.sh {code} mode=$1 start=$(date +%s) ./pig -x $mode $PIG_HOME/bin/join.pig end=$(date +%s) execution_time=$(( $end - $start )) echo "execution_time:"$excution_time {code} The execution time: ||||||mr||spark|| |join|20 sec|79 sec| You can download the test data NYSE_daily and NYSE_dividends in https://github.com/alanfgates/programmingpig/blob/master/data/ > Analyze JOIN performance and improve the same. > ---------------------------------------------- > > Key: PIG-4797 > URL: https://issues.apache.org/jira/browse/PIG-4797 > Project: Pig > Issue Type: Improvement > Components: spark > Reporter: Pallavi Rao > Assignee: Pallavi Rao > Labels: spork > > There are a big performance difference in join between spark and mr mode. > {code} > daily = load './NYSE_daily' as (exchange:chararray, symbol:chararray, > date:chararray, open:float, high:float, low:float, > close:float, volume:int, adj_close:float); > divs = load './NYSE_dividends' as (exchange:chararray, symbol:chararray, > date:chararray, dividends:float); > jnd = join daily by (exchange, symbol), divs by (exchange, symbol); > store jnd into './join.out'; > {code} > join.sh > {code} > mode=$1 > start=$(date +%s) > ./pig -x $mode $PIG_HOME/bin/join.pig > end=$(date +%s) > execution_time=$(( $end - $start )) > echo "execution_time:"$excution_time > {code} > The execution time: > || |||mr||spark|| > |join|20 sec|79 sec| > You can download the test data NYSE_daily and NYSE_dividends in > https://github.com/alanfgates/programmingpig/blob/master/data/ -- This message was sent by Atlassian JIRA (v6.3.4#6332)