[jira] [Commented] (PIG-5305) Enable yarn-client mode execution of tests in Spark (1) mode

2017-10-06 Thread Adam Szita (JIRA) 

[ 
https://issues.apache.org/jira/browse/PIG-5305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16194512#comment-16194512
 ] 

Adam Szita commented on PIG-5305:
-

Thanks for the review [~kellyzly], latest patch is now committed to trunk.

> Enable yarn-client mode execution of tests in Spark (1) mode
> 
>
> Key: PIG-5305
> URL: https://issues.apache.org/jira/browse/PIG-5305
> Project: Pig
>  Issue Type: Sub-task
>  Components: spark
>Reporter: Adam Szita
>Assignee: Adam Szita
> Attachments: PIG-5305.0.patch, PIG-5305.1.patch, PIG-5305.2.patch
>
>
> See parent jira (PIG-5305) for problem description



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (PIG-5305) Enable yarn-client mode execution of tests in Spark (1) mode

2017-10-05 Thread liyunzhang_intel (JIRA) 

[ 
https://issues.apache.org/jira/browse/PIG-5305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16192995#comment-16192995
 ] 

liyunzhang_intel commented on PIG-5305:
---

[~szita]: sorry for reply late. Out Of Office this week.
for the patch: +1.

> Enable yarn-client mode execution of tests in Spark (1) mode
> 
>
> Key: PIG-5305
> URL: https://issues.apache.org/jira/browse/PIG-5305
> Project: Pig
>  Issue Type: Sub-task
>  Components: spark
>Reporter: Adam Szita
>Assignee: Adam Szita
> Attachments: PIG-5305.0.patch, PIG-5305.1.patch, PIG-5305.2.patch
>
>
> See parent jira (PIG-5305) for problem description



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (PIG-5305) Enable yarn-client mode execution of tests in Spark (1) mode

2017-10-03 Thread Adam Szita (JIRA) 

[ 
https://issues.apache.org/jira/browse/PIG-5305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16189506#comment-16189506
 ] 

Adam Szita commented on PIG-5305:
-

[~kellyzly]: you may have artifacts already built on your filesystem? Can you 
try adding {{clean}} target as well to your execution?
{{ant clean -Dtest.junit.output.format=xml -Dtestcase=TestEvalPipeline 
-Dexectype=tez -Dhadoopversion=2 test-tez}}

This still fails for me on the current HEAD of trunk.

> Enable yarn-client mode execution of tests in Spark (1) mode
> 
>
> Key: PIG-5305
> URL: https://issues.apache.org/jira/browse/PIG-5305
> Project: Pig
>  Issue Type: Sub-task
>  Components: spark
>Reporter: Adam Szita
>Assignee: Adam Szita
> Attachments: PIG-5305.0.patch, PIG-5305.1.patch, PIG-5305.2.patch
>
>
> See parent jira (PIG-5305) for problem description



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (PIG-5305) Enable yarn-client mode execution of tests in Spark (1) mode

2017-09-24 Thread liyunzhang_intel (JIRA) 

[ 
https://issues.apache.org/jira/browse/PIG-5305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16178450#comment-16178450
 ] 

liyunzhang_intel commented on PIG-5305:
---

[~szita]:
sorry for reply late.
1 thing i was confused is unit test TestEvalPipeLine passes in tez mode without 
this patch by command
{code}
 ant -v -Dtest.junit.output.format=xml -Dtestcase=TestEvalPipeline 
-Dexectype=tez -Dhadoopversion=2 test-tez
{code}
code base:7399a1c
Before you mentioned that some unit tests failed with command {{test-tez}}.So 
is there some wrong with my env?
Patch looks good but please confirm this, thanks!

> Enable yarn-client mode execution of tests in Spark (1) mode
> 
>
> Key: PIG-5305
> URL: https://issues.apache.org/jira/browse/PIG-5305
> Project: Pig
>  Issue Type: Sub-task
>  Components: spark
>Reporter: Adam Szita
>Assignee: Adam Szita
> Attachments: PIG-5305.0.patch, PIG-5305.1.patch, PIG-5305.2.patch
>
>
> See parent jira (PIG-5305) for problem description



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (PIG-5305) Enable yarn-client mode execution of tests in Spark (1) mode

2017-09-22 Thread Adam Szita (JIRA) 

[ 
https://issues.apache.org/jira/browse/PIG-5305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16176186#comment-16176186
 ] 

Adam Szita commented on PIG-5305:
-

[~kellyzly] do you think this is ready for commit now?

> Enable yarn-client mode execution of tests in Spark (1) mode
> 
>
> Key: PIG-5305
> URL: https://issues.apache.org/jira/browse/PIG-5305
> Project: Pig
>  Issue Type: Sub-task
>  Components: spark
>Reporter: Adam Szita
>Assignee: Adam Szita
> Attachments: PIG-5305.0.patch, PIG-5305.1.patch, PIG-5305.2.patch
>
>
> See parent jira (PIG-5305) for problem description



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (PIG-5305) Enable yarn-client mode execution of tests in Spark (1) mode

2017-09-21 Thread Adam Szita (JIRA) 

[ 
https://issues.apache.org/jira/browse/PIG-5305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16174837#comment-16174837
 ] 

Adam Szita commented on PIG-5305:
-

[~kellyzly] yes {{src.exclude.dir}} was probably just left there, and had no 
use since the removal of Hadoop 1 support. Then Spark 2 support came with 
PIG-5157, and as you correctly point it out, resetting src.exclude.dir does 
influence {{jar}} target.

The reason we didn't see this before is because nobody used the {{test-tez}} 
target, in the Apache Jenkins job we use {{test-core-mrtez}] which runs all MR 
and then all Tez unit tests.

> Enable yarn-client mode execution of tests in Spark (1) mode
> 
>
> Key: PIG-5305
> URL: https://issues.apache.org/jira/browse/PIG-5305
> Project: Pig
>  Issue Type: Sub-task
>  Components: spark
>Reporter: Adam Szita
>Assignee: Adam Szita
> Attachments: PIG-5305.0.patch, PIG-5305.1.patch, PIG-5305.2.patch
>
>
> See parent jira (PIG-5305) for problem description



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (PIG-5305) Enable yarn-client mode execution of tests in Spark (1) mode

2017-09-19 Thread liyunzhang_intel (JIRA) 

[ 
https://issues.apache.org/jira/browse/PIG-5305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16172741#comment-16172741
 ] 

liyunzhang_intel commented on PIG-5305:
---

{quote}
I also checked, test-tez was not running properly since the Spark 2 support 
commit, because setTezEnv was clearing the excluded sources property. I fixed 
this in my latest patch as well.
{quote}
what you mean is before we add {{jar-simple}} in  the dependency of 
{{test-tez}} in PIG-5157, But in {{setTezEnv}}, it will reset 
{{src.exclude.dir}} , this will influence {{jar}} which use the property 
{{src.exclude.dir}}?
{code}
 
Compiling against Spark 2






Compiling against Spark 1

















{code}


> Enable yarn-client mode execution of tests in Spark (1) mode
> 
>
> Key: PIG-5305
> URL: https://issues.apache.org/jira/browse/PIG-5305
> Project: Pig
>  Issue Type: Sub-task
>  Components: spark
>Reporter: Adam Szita
>Assignee: Adam Szita
> Attachments: PIG-5305.0.patch, PIG-5305.1.patch, PIG-5305.2.patch
>
>
> See parent jira (PIG-5305) for problem description



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (PIG-5305) Enable yarn-client mode execution of tests in Spark (1) mode

2017-09-19 Thread Adam Szita (JIRA) 

[ 
https://issues.apache.org/jira/browse/PIG-5305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16171478#comment-16171478
 ] 

Adam Szita commented on PIG-5305:
-

[~kellyzly]
1: removed the dependecy from test-tez. I also checked, test-tez was not 
running properly since the Spark 2 support commit, because {{setTezEnv}} was 
clearing the excluded sources property. I fixed this in my latest patch as well.

2: There were quite a couple of failures at first, that's why I had to add a 
reset feature of SparkContexts into SparkLauncher. With the latest patch it 
shouldn't have any failures.

> Enable yarn-client mode execution of tests in Spark (1) mode
> 
>
> Key: PIG-5305
> URL: https://issues.apache.org/jira/browse/PIG-5305
> Project: Pig
>  Issue Type: Sub-task
>  Components: spark
>Reporter: Adam Szita
>Assignee: Adam Szita
> Attachments: PIG-5305.0.patch, PIG-5305.1.patch, PIG-5305.2.patch
>
>
> See parent jira (PIG-5305) for problem description



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (PIG-5305) Enable yarn-client mode execution of tests in Spark (1) mode

2017-09-18 Thread liyunzhang_intel (JIRA) 

[ 
https://issues.apache.org/jira/browse/PIG-5305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16170981#comment-16170981
 ] 

liyunzhang_intel commented on PIG-5305:
---

[~szita]:
1. {code}

{code}

why need pigtest-jar in test-tez?
2. is there any unit test failures if convert SPARK_MASTER from "local" to 
"yarn-client"?

> Enable yarn-client mode execution of tests in Spark (1) mode
> 
>
> Key: PIG-5305
> URL: https://issues.apache.org/jira/browse/PIG-5305
> Project: Pig
>  Issue Type: Sub-task
>  Components: spark
>Reporter: Adam Szita
>Assignee: Adam Szita
> Attachments: PIG-5305.0.patch, PIG-5305.1.patch
>
>
> See parent jira (PIG-5305) for problem description



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (PIG-5305) Enable yarn-client mode execution of tests in Spark (1) mode

2017-09-18 Thread Adam Szita (JIRA) 

[ 
https://issues.apache.org/jira/browse/PIG-5305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16169815#comment-16169815
 ] 

Adam Szita commented on PIG-5305:
-

Thanks for the comments [~kellyzly].
Attached [^PIG-5305.1.patch].

1. Correct, test-core-mrtez indeed doesn't need jar-simple, I removed that. 
However I'd like to keep pigtest-jar target calls in test related targets. For 
example if someone launches {{ant clean test -Dtest.exec.type=spark}} we have 
too keep it on {{test-core}} target as well.
2. Added comment as requested.


> Enable yarn-client mode execution of tests in Spark (1) mode
> 
>
> Key: PIG-5305
> URL: https://issues.apache.org/jira/browse/PIG-5305
> Project: Pig
>  Issue Type: Sub-task
>  Components: spark
>Reporter: Adam Szita
>Assignee: Adam Szita
> Attachments: PIG-5305.0.patch, PIG-5305.1.patch
>
>
> See parent jira (PIG-5305) for problem description



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (PIG-5305) Enable yarn-client mode execution of tests in Spark (1) mode

2017-09-14 Thread liyunzhang_intel (JIRA) 

[ 
https://issues.apache.org/jira/browse/PIG-5305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16167284#comment-16167284
 ] 

liyunzhang_intel commented on PIG-5305:
---

[~szita]: several suggestions
1. can we only modify {{test-spark}} target in build.xml? I guess for 
{{test-tez}}, there is no need for {{pigtest-jar}}
meanwhile there is no need to add {{jar-simple}} in the dependency of 
{{test-tez}} as the dependency of {{compile-test}} includes {{jar-simple}}. If 
my understanding is not right, tell me.
2.please add comment {{added feature to re-initialize SparkContext when 
switching between cluster and local mode PigServers}} on related code. 

Besides, is there any unit test failures if convert  {{SPARK_MASTER}} from 
"local" to "yarn-client"?

> Enable yarn-client mode execution of tests in Spark (1) mode
> 
>
> Key: PIG-5305
> URL: https://issues.apache.org/jira/browse/PIG-5305
> Project: Pig
>  Issue Type: Sub-task
>  Components: spark
>Reporter: Adam Szita
>Assignee: Adam Szita
> Attachments: PIG-5305.0.patch
>
>
> See parent jira (PIG-5305) for problem description



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (PIG-5305) Enable yarn-client mode execution of tests in Spark (1) mode

2017-09-14 Thread Adam Szita (JIRA) 

[ 
https://issues.apache.org/jira/browse/PIG-5305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16166543#comment-16166543
 ] 

Adam Szita commented on PIG-5305:
-

Attached [^PIG-5305.0.patch] to enable running tests in yarn-client mode for 
Spark execution.

Main changes:
* build.xml: added target to build a jar with all test classes. This is 
required so that we can pass this test jar onto SparkContext which then 
distributes it among Spark executors + set SPARK_MASTER env var to "yarn-client"
* SparkLauncher: added feature to re-initialize SparkContext when switching 
between cluster and local mode PigServers + only setting 
ChildFirstURLClassLoader during cluster mode

[~kellyzly] can you please take a look?


> Enable yarn-client mode execution of tests in Spark (1) mode
> 
>
> Key: PIG-5305
> URL: https://issues.apache.org/jira/browse/PIG-5305
> Project: Pig
>  Issue Type: Sub-task
>  Components: spark
>Reporter: Adam Szita
>Assignee: Adam Szita
> Attachments: PIG-5305.0.patch
>
>
> See parent jira (PIG-5305) for problem description



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)