[
https://issues.apache.org/jira/browse/PIG-4890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15347615#comment-15347615
]
liyunzhang_intel commented on PIG-4890:
---------------------------------------
Following is the result of pigmix between spark and mr mode. I use patch of
PIG-4797, PIG-4903, PIG-4893.
The test env: 3 nodes cluster( i assign 28 cpu core and 56g memery on each node)
yarn configuration
{code}
yarn.nodemanager.resource.memory-mb=56G
yarn.nodemanger.resource.cpu-vcores=28
{code}
Spark configurations in pig.properties
{code}
spark.executor.cores=4
#6553m=6.4g
spark.executor.memory=6553m
#1638=1.6g
spark.yarn.executor.memoryOverhead=1638
#2048m=2g
spark.driver.memory=2048m
spark.executor.instances=7
{code}
||Script||mr||spark||
|L_1|41|33|
|L_2|37|28|
|L_3|67|30|
|L_4|36|29|
|L_5|36|25|
|L_6|36|27|
|L_7|36|24|
|L_8|25|24|
|L_9|260|148|
|L_10|298|148|
|L_11|154|36|
|L_12|41|32|
|L_13|36|24|
|L_14|53|22|
|L_15|35|24|
|L_16|35|25|
|L_17|76|33|
It shows that pig on spark runs faster than pig on mr.
> Run pigmix on spark on yarn with multiple nodes
> -----------------------------------------------
>
> Key: PIG-4890
> URL: https://issues.apache.org/jira/browse/PIG-4890
> Project: Pig
> Issue Type: Sub-task
> Components: spark
> Reporter: liyunzhang_intel
>
> Test pigmix in three nodes with 40 core and 60g memory.
> The result of spark in multiple nodes and single node are
> ||Script||multiple nodes||single node||
> |L_1|75|53|
> |L_2|63|45|
> |L_3|65|49|
> |L_4|68|48|
> |L_5|64|47|
> |L_6|70|50|
> |L_7|65|46|
> |L_8|65|46|
> |L_9|157|50|
> |L_10|160|51|
> |L_11|104|64|
> |L_12|74|54|
> |L_13|66|46|
> |L_14|66|47|
> |L_15|67|47|
> |L_16|67|50|
> |L_17|88|54|
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
