[ https://issues.apache.org/jira/browse/PIG-4890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15347615#comment-15347615 ]
liyunzhang_intel commented on PIG-4890: --------------------------------------- Following is the result of pigmix between spark and mr mode. I use patch of PIG-4797, PIG-4903, PIG-4893. The test env: 3 nodes cluster( i assign 28 cpu core and 56g memery on each node) yarn configuration {code} yarn.nodemanager.resource.memory-mb=56G yarn.nodemanger.resource.cpu-vcores=28 {code} Spark configurations in pig.properties {code} spark.executor.cores=4 #6553m=6.4g spark.executor.memory=6553m #1638=1.6g spark.yarn.executor.memoryOverhead=1638 #2048m=2g spark.driver.memory=2048m spark.executor.instances=7 {code} ||Script||mr||spark|| |L_1|41|33| |L_2|37|28| |L_3|67|30| |L_4|36|29| |L_5|36|25| |L_6|36|27| |L_7|36|24| |L_8|25|24| |L_9|260|148| |L_10|298|148| |L_11|154|36| |L_12|41|32| |L_13|36|24| |L_14|53|22| |L_15|35|24| |L_16|35|25| |L_17|76|33| It shows that pig on spark runs faster than pig on mr. > Run pigmix on spark on yarn with multiple nodes > ----------------------------------------------- > > Key: PIG-4890 > URL: https://issues.apache.org/jira/browse/PIG-4890 > Project: Pig > Issue Type: Sub-task > Components: spark > Reporter: liyunzhang_intel > > Test pigmix in three nodes with 40 core and 60g memory. > The result of spark in multiple nodes and single node are > ||Script||multiple nodes||single node|| > |L_1|75|53| > |L_2|63|45| > |L_3|65|49| > |L_4|68|48| > |L_5|64|47| > |L_6|70|50| > |L_7|65|46| > |L_8|65|46| > |L_9|157|50| > |L_10|160|51| > |L_11|104|64| > |L_12|74|54| > |L_13|66|46| > |L_14|66|47| > |L_15|67|47| > |L_16|67|50| > |L_17|88|54| -- This message was sent by Atlassian JIRA (v6.3.4#6332)