[jira] [Commented] (PIG-5305) Enable yarn-client mode execution of tests in Spark (1) mode

2017-10-05 Thread liyunzhang_intel (JIRA)
[ https://issues.apache.org/jira/browse/PIG-5305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16192995#comment-16192995 ] liyunzhang_intel commented on PIG-5305: --- [~szita]: sorry for reply late. Out Of Office this week. for

[jira] [Commented] (PIG-5305) Enable yarn-client mode execution of tests in Spark (1) mode

2017-09-24 Thread liyunzhang_intel (JIRA)
[ https://issues.apache.org/jira/browse/PIG-5305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16178450#comment-16178450 ] liyunzhang_intel commented on PIG-5305: --- [~szita]: sorry for reply late. 1 thing i was confused is

[jira] [Comment Edited] (PIG-5305) Enable yarn-client mode execution of tests in Spark (1) mode

2017-09-19 Thread liyunzhang_intel (JIRA)
[ https://issues.apache.org/jira/browse/PIG-5305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16172741#comment-16172741 ] liyunzhang_intel edited comment on PIG-5305 at 9/20/17 4:48 AM: {quote} I

[jira] [Comment Edited] (PIG-5305) Enable yarn-client mode execution of tests in Spark (1) mode

2017-09-19 Thread liyunzhang_intel (JIRA)
[ https://issues.apache.org/jira/browse/PIG-5305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16172741#comment-16172741 ] liyunzhang_intel edited comment on PIG-5305 at 9/20/17 4:47 AM: {quote} I

[jira] [Commented] (PIG-5305) Enable yarn-client mode execution of tests in Spark (1) mode

2017-09-19 Thread liyunzhang_intel (JIRA)
[ https://issues.apache.org/jira/browse/PIG-5305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16172741#comment-16172741 ] liyunzhang_intel commented on PIG-5305: --- {quote} I also checked, test-tez was not running properly

[jira] [Commented] (PIG-5305) Enable yarn-client mode execution of tests in Spark (1) mode

2017-09-18 Thread liyunzhang_intel (JIRA)
[ https://issues.apache.org/jira/browse/PIG-5305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16170981#comment-16170981 ] liyunzhang_intel commented on PIG-5305: --- [~szita]: 1. {code} {code} why need pigtest-jar in

[jira] [Comment Edited] (PIG-5305) Enable yarn-client mode execution of tests in Spark (1) mode

2017-09-14 Thread liyunzhang_intel (JIRA)
[ https://issues.apache.org/jira/browse/PIG-5305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16167284#comment-16167284 ] liyunzhang_intel edited comment on PIG-5305 at 9/15/17 3:49 AM: [~szita]:

[jira] [Commented] (PIG-5277) Spark mode is writing nulls among tuples to the output

2017-08-11 Thread liyunzhang_intel (JIRA)
[ https://issues.apache.org/jira/browse/PIG-5277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16123129#comment-16123129 ] liyunzhang_intel commented on PIG-5277: --- [~szita]: use #2. leave it to further investigation. first

[jira] [Commented] (PIG-5283) Configuration is not passed to SparkPigSplits on the backend

2017-08-09 Thread liyunzhang_intel (JIRA)
[ https://issues.apache.org/jira/browse/PIG-5283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16119488#comment-16119488 ] liyunzhang_intel commented on PIG-5283: --- [~szita]: thanks for your explaination. +1 > Configuration

[jira] [Commented] (PIG-5283) Configuration is not passed to SparkPigSplits on the backend

2017-08-08 Thread liyunzhang_intel (JIRA)
[ https://issues.apache.org/jira/browse/PIG-5283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16119287#comment-16119287 ] liyunzhang_intel commented on PIG-5283: --- [~szita]: I can understand the reason why need set

[jira] [Commented] (PIG-5283) Configuration is not passed to SparkPigSplits on the backend

2017-08-08 Thread liyunzhang_intel (JIRA)
[ https://issues.apache.org/jira/browse/PIG-5283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16118075#comment-16118075 ] liyunzhang_intel commented on PIG-5283: --- [~szita]: {quote} My only question is that if we should

[jira] [Commented] (PIG-5277) Spark mode is writing nulls among tuples to the output

2017-08-03 Thread liyunzhang_intel (JIRA)
[ https://issues.apache.org/jira/browse/PIG-5277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16113820#comment-16113820 ] liyunzhang_intel commented on PIG-5277: --- [~szita]: give me some time to investigate #4. If also can

[jira] [Commented] (PIG-5283) Configuration is not passed to SparkPigSplits on the backend

2017-08-03 Thread liyunzhang_intel (JIRA)
[ https://issues.apache.org/jira/browse/PIG-5283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16113817#comment-16113817 ] liyunzhang_intel commented on PIG-5283: --- [~nkollar] and [~szita]: Can we set the correct value of

[jira] [Commented] (PIG-5283) Configuration is not passed to SparkPigSplits on the backend

2017-08-03 Thread liyunzhang_intel (JIRA)
[ https://issues.apache.org/jira/browse/PIG-5283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16112449#comment-16112449 ] liyunzhang_intel commented on PIG-5283: --- [~szita]: In

[jira] [Commented] (PIG-5276) building "jar" should not call "clean"

2017-08-01 Thread liyunzhang_intel (JIRA)
[ https://issues.apache.org/jira/browse/PIG-5276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16109250#comment-16109250 ] liyunzhang_intel commented on PIG-5276: --- [~nkollar]:+1 > building "jar" should not call "clean" >

[jira] [Comment Edited] (PIG-5276) building "jar" should not call "clean"

2017-07-31 Thread liyunzhang_intel (JIRA)
[ https://issues.apache.org/jira/browse/PIG-5276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16106927#comment-16106927 ] liyunzhang_intel edited comment on PIG-5276 at 7/31/17 7:50 AM: [~nkollar]:

[jira] [Commented] (PIG-5276) building "jar" should not call "clean"

2017-07-31 Thread liyunzhang_intel (JIRA)
[ https://issues.apache.org/jira/browse/PIG-5276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16106927#comment-16106927 ] liyunzhang_intel commented on PIG-5276: --- [~nkollar]: when upgrading to spark2, we need to delete

[jira] [Commented] (PIG-5276) building "jar" should not call "clean"

2017-07-30 Thread liyunzhang_intel (JIRA)
[ https://issues.apache.org/jira/browse/PIG-5276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16106725#comment-16106725 ] liyunzhang_intel commented on PIG-5276: --- [~nkollar]: can we not call clean before jar when upgrading

[jira] [Comment Edited] (PIG-3655) BinStorage and InterStorage approach to record markers is broken

2017-07-26 Thread liyunzhang_intel (JIRA)
[ https://issues.apache.org/jira/browse/PIG-3655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16102348#comment-16102348 ] liyunzhang_intel edited comment on PIG-3655 at 7/26/17 10:01 PM: - [~szita]:

[jira] [Commented] (PIG-3655) BinStorage and InterStorage approach to record markers is broken

2017-07-26 Thread liyunzhang_intel (JIRA)
[ https://issues.apache.org/jira/browse/PIG-3655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16102348#comment-16102348 ] liyunzhang_intel commented on PIG-3655: --- [~szita]: can you provide simple script which i can reproduce

[jira] [Commented] (PIG-3655) BinStorage and InterStorage approach to record markers is broken

2017-07-26 Thread liyunzhang_intel (JIRA)
[ https://issues.apache.org/jira/browse/PIG-3655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16101355#comment-16101355 ] liyunzhang_intel commented on PIG-3655: --- [~szita]: Can you add a filter to filter the empty tuple in

[jira] [Updated] (PIG-5246) Modify bin/pig about SPARK_HOME, SPARK_ASSEMBLY_JAR after upgrading spark to 2

2017-07-24 Thread liyunzhang_intel (JIRA)
[ https://issues.apache.org/jira/browse/PIG-5246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liyunzhang_intel updated PIG-5246: -- Resolution: Fixed Status: Resolved (was: Patch Available) > Modify bin/pig about

[jira] [Commented] (PIG-5246) Modify bin/pig about SPARK_HOME, SPARK_ASSEMBLY_JAR after upgrading spark to 2

2017-07-24 Thread liyunzhang_intel (JIRA)
[ https://issues.apache.org/jira/browse/PIG-5246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16099380#comment-16099380 ] liyunzhang_intel commented on PIG-5246: --- [~nkollar]: commit to trunk, thanks for [~nkollar],

[jira] [Commented] (PIG-5157) Upgrade to Spark 2.0

2017-07-23 Thread liyunzhang_intel (JIRA)
[ https://issues.apache.org/jira/browse/PIG-5157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16097845#comment-16097845 ] liyunzhang_intel commented on PIG-5157: --- [~szita]: {quote} it looks like you've missed adding an

[jira] [Commented] (PIG-5246) Modify bin/pig about SPARK_HOME, SPARK_ASSEMBLY_JAR after upgrading spark to 2

2017-07-19 Thread liyunzhang_intel (JIRA)
[ https://issues.apache.org/jira/browse/PIG-5246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16094066#comment-16094066 ] liyunzhang_intel commented on PIG-5246: --- [~nkollar]: as PIG-5157 is resolved, please help review

[jira] [Updated] (PIG-5157) Upgrade to Spark 2.0

2017-07-18 Thread liyunzhang_intel (JIRA)
[ https://issues.apache.org/jira/browse/PIG-5157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liyunzhang_intel updated PIG-5157: -- Attachment: PIG-5157_15.patch > Upgrade to Spark 2.0 > > >

[jira] [Updated] (PIG-5157) Upgrade to Spark 2.0

2017-07-18 Thread liyunzhang_intel (JIRA)
[ https://issues.apache.org/jira/browse/PIG-5157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liyunzhang_intel updated PIG-5157: -- Resolution: Fixed Status: Resolved (was: Patch Available) > Upgrade to Spark 2.0 >

[jira] [Commented] (PIG-5157) Upgrade to Spark 2.0

2017-07-18 Thread liyunzhang_intel (JIRA)
[ https://issues.apache.org/jira/browse/PIG-5157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16092460#comment-16092460 ] liyunzhang_intel commented on PIG-5157: --- [~nkollar]: commit PIG-5157_15.patch to the trunk, thanks for

[jira] [Commented] (PIG-5080) Support store alias as spark table

2017-07-18 Thread liyunzhang_intel (JIRA)
[ https://issues.apache.org/jira/browse/PIG-5080?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16091220#comment-16091220 ] liyunzhang_intel commented on PIG-5080: --- [~jeffzhang]: thanks for your patch. What's the benefit of

[jira] [Commented] (PIG-5157) Upgrade to Spark 2.0

2017-07-17 Thread liyunzhang_intel (JIRA)
[ https://issues.apache.org/jira/browse/PIG-5157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16089381#comment-16089381 ] liyunzhang_intel commented on PIG-5157: --- [~nkollar]: left comment on review board, just a small fix!

[jira] [Commented] (PIG-5246) Modify bin/pig about SPARK_HOME, SPARK_ASSEMBLY_JAR after upgrading spark to 2

2017-07-16 Thread liyunzhang_intel (JIRA)
[ https://issues.apache.org/jira/browse/PIG-5246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16089217#comment-16089217 ] liyunzhang_intel commented on PIG-5246: --- [~nkollar]: the problem about basic script passed on

[jira] [Updated] (PIG-5246) Modify bin/pig about SPARK_HOME, SPARK_ASSEMBLY_JAR after upgrading spark to 2

2017-07-14 Thread liyunzhang_intel (JIRA)
[ https://issues.apache.org/jira/browse/PIG-5246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liyunzhang_intel updated PIG-5246: -- Attachment: PIG-5246_4.patch [~nkollar]: changes in PIG-5246_4.patch: {code}

[jira] [Updated] (PIG-5157) Upgrade to Spark 2.0

2017-07-13 Thread liyunzhang_intel (JIRA)
[ https://issues.apache.org/jira/browse/PIG-5157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liyunzhang_intel updated PIG-5157: -- Attachment: SkewedJoinInput2.txt SkewedJoinInput1.txt > Upgrade to Spark 2.0 >

[jira] [Commented] (PIG-5157) Upgrade to Spark 2.0

2017-07-13 Thread liyunzhang_intel (JIRA)
[ https://issues.apache.org/jira/browse/PIG-5157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16085443#comment-16085443 ] liyunzhang_intel commented on PIG-5157: --- [~nkollar]: attached SkewedJoinInput1.txt,

[jira] [Commented] (PIG-5157) Upgrade to Spark 2.0

2017-07-09 Thread liyunzhang_intel (JIRA)
[ https://issues.apache.org/jira/browse/PIG-5157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16079868#comment-16079868 ] liyunzhang_intel commented on PIG-5157: --- [~nkollar]: sorry to reply so late. here is the result after

[jira] [Commented] (PIG-5157) Upgrade to Spark 2.0

2017-06-30 Thread liyunzhang_intel (JIRA)
[ https://issues.apache.org/jira/browse/PIG-5157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16069728#comment-16069728 ] liyunzhang_intel commented on PIG-5157: --- [~nkollar]: with PIG-5157_13.patch on review board. the test

[jira] [Commented] (PIG-5157) Upgrade to Spark 2.0

2017-06-26 Thread liyunzhang_intel (JIRA)
[ https://issues.apache.org/jira/browse/PIG-5157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16062805#comment-16062805 ] liyunzhang_intel commented on PIG-5157: --- [~nkollar]: with PIG-5157_11.patch , pass in TestGrunt unit

[jira] [Comment Edited] (PIG-5157) Upgrade to Spark 2.0

2017-06-23 Thread liyunzhang_intel (JIRA)
[ https://issues.apache.org/jira/browse/PIG-5157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16060574#comment-16060574 ] liyunzhang_intel edited comment on PIG-5157 at 6/23/17 8:29 AM:

[jira] [Commented] (PIG-5157) Upgrade to Spark 2.0

2017-06-23 Thread liyunzhang_intel (JIRA)
[ https://issues.apache.org/jira/browse/PIG-5157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16060574#comment-16060574 ] liyunzhang_intel commented on PIG-5157: --- [~nkollar]:looks good, but met some problem when testing in

[jira] [Updated] (PIG-5246) Modify bin/pig about SPARK_HOME, SPARK_ASSEMBLY_JAR after upgrading spark to 2

2017-06-21 Thread liyunzhang_intel (JIRA)
[ https://issues.apache.org/jira/browse/PIG-5246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liyunzhang_intel updated PIG-5246: -- Attachment: PIG-5246.3.patch [~nkollar],[~szita],[~rohini],[~jeffzhang]: update PIG-5246.3.patch

[jira] [Commented] (PIG-5157) Upgrade to Spark 2.0

2017-06-21 Thread liyunzhang_intel (JIRA)
[ https://issues.apache.org/jira/browse/PIG-5157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16057075#comment-16057075 ] liyunzhang_intel commented on PIG-5157: --- [~nkollar]: after using the patch and test a simple query in

[jira] [Commented] (PIG-5246) Modify bin/pig about SPARK_HOME, SPARK_ASSEMBLY_JAR after upgrading spark to 2

2017-06-20 Thread liyunzhang_intel (JIRA)
[ https://issues.apache.org/jira/browse/PIG-5246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16056789#comment-16056789 ] liyunzhang_intel commented on PIG-5246: --- [~rohini]: bq.You can cherry pick jars to include in the

[jira] [Commented] (PIG-5157) Upgrade to Spark 2.0

2017-06-19 Thread liyunzhang_intel (JIRA)
[ https://issues.apache.org/jira/browse/PIG-5157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16053644#comment-16053644 ] liyunzhang_intel commented on PIG-5157: --- [~nkollar]: It builds successful in my env when using ant

[jira] [Comment Edited] (PIG-5246) Modify bin/pig about SPARK_HOME, SPARK_ASSEMBLY_JAR after upgrading spark to 2

2017-06-18 Thread liyunzhang_intel (JIRA)
[ https://issues.apache.org/jira/browse/PIG-5246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16053472#comment-16053472 ] liyunzhang_intel edited comment on PIG-5246 at 6/19/17 4:50 AM: [~rohini]:

[jira] [Commented] (PIG-5246) Modify bin/pig about SPARK_HOME, SPARK_ASSEMBLY_JAR after upgrading spark to 2

2017-06-18 Thread liyunzhang_intel (JIRA)
[ https://issues.apache.org/jira/browse/PIG-5246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16053472#comment-16053472 ] liyunzhang_intel commented on PIG-5246: --- [~rohini]:bq. I would suggest checking for presence of

[jira] [Commented] (PIG-5246) Modify bin/pig about SPARK_HOME, SPARK_ASSEMBLY_JAR after upgrading spark to 2

2017-06-16 Thread liyunzhang_intel (JIRA)
[ https://issues.apache.org/jira/browse/PIG-5246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16051474#comment-16051474 ] liyunzhang_intel commented on PIG-5246: --- [~nkollar]: bq. For Spark 2.x do we have to add all jar

[jira] [Commented] (PIG-5246) Modify bin/pig about SPARK_HOME, SPARK_ASSEMBLY_JAR after upgrading spark to 2

2017-06-15 Thread liyunzhang_intel (JIRA)
[ https://issues.apache.org/jira/browse/PIG-5246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16050164#comment-16050164 ] liyunzhang_intel commented on PIG-5246: --- [~nkollar]: can you help review PIG-5246_2.patch? in

[jira] [Commented] (PIG-5157) Upgrade to Spark 2.0

2017-06-15 Thread liyunzhang_intel (JIRA)
[ https://issues.apache.org/jira/browse/PIG-5157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16050123#comment-16050123 ] liyunzhang_intel commented on PIG-5157: --- [~nkollar]: after i download the latest patch from RB, how to

[jira] [Commented] (PIG-5157) Upgrade to Spark 2.0

2017-06-13 Thread liyunzhang_intel (JIRA)
[ https://issues.apache.org/jira/browse/PIG-5157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16047531#comment-16047531 ] liyunzhang_intel commented on PIG-5157: --- [~nkollar]: leave some comments on review board. can you

[jira] [Comment Edited] (PIG-5157) Upgrade to Spark 2.0

2017-06-13 Thread liyunzhang_intel (JIRA)
[ https://issues.apache.org/jira/browse/PIG-5157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16047531#comment-16047531 ] liyunzhang_intel edited comment on PIG-5157 at 6/13/17 7:36 AM: [~nkollar]:

[jira] [Commented] (PIG-5246) Modify bin/pig about SPARK_HOME, SPARK_ASSEMBLY_JAR after upgrading spark to 2

2017-06-04 Thread liyunzhang_intel (JIRA)
[ https://issues.apache.org/jira/browse/PIG-5246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16036470#comment-16036470 ] liyunzhang_intel commented on PIG-5246: --- [~jeffzhang]: thanks for suggestion bq. Why copying the

[jira] [Created] (PIG-5252) Get properties from $SPARK_HOME/conf/spark-defaults.conf not from pig.properties

2017-06-04 Thread liyunzhang_intel (JIRA)
liyunzhang_intel created PIG-5252: - Summary: Get properties from $SPARK_HOME/conf/spark-defaults.conf not from pig.properties Key: PIG-5252 URL: https://issues.apache.org/jira/browse/PIG-5252

[jira] [Comment Edited] (PIG-5246) Modify bin/pig about SPARK_HOME, SPARK_ASSEMBLY_JAR after upgrading spark to 2

2017-06-04 Thread liyunzhang_intel (JIRA)
[ https://issues.apache.org/jira/browse/PIG-5246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16036459#comment-16036459 ] liyunzhang_intel edited comment on PIG-5246 at 6/5/17 1:59 AM: --- [~rohini]:

[jira] [Commented] (PIG-5246) Modify bin/pig about SPARK_HOME, SPARK_ASSEMBLY_JAR after upgrading spark to 2

2017-06-04 Thread liyunzhang_intel (JIRA)
[ https://issues.apache.org/jira/browse/PIG-5246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16036459#comment-16036459 ] liyunzhang_intel commented on PIG-5246: --- [~rohini]: thanks for suggestion, for spark1 and spark2, it

[jira] [Resolved] (PIG-5215) Merge changes from review board to spark branch

2017-06-04 Thread liyunzhang_intel (JIRA)
[ https://issues.apache.org/jira/browse/PIG-5215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liyunzhang_intel resolved PIG-5215. --- Resolution: Fixed > Merge changes from review board to spark branch >

[jira] [Commented] (PIG-5215) Merge changes from review board to spark branch

2017-06-04 Thread liyunzhang_intel (JIRA)
[ https://issues.apache.org/jira/browse/PIG-5215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16036445#comment-16036445 ] liyunzhang_intel commented on PIG-5215: --- [~nkollar]: thanks for remind, close it > Merge changes from

[jira] [Commented] (PIG-5246) Modify bin/pig about SPARK_HOME, SPARK_ASSEMBLY_JAR after upgrading spark to 2

2017-06-02 Thread liyunzhang_intel (JIRA)
[ https://issues.apache.org/jira/browse/PIG-5246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16034284#comment-16034284 ] liyunzhang_intel commented on PIG-5246: --- [~szita], [~nkollar], [~rohini] and [~jeffzhang]: It is not

[jira] [Comment Edited] (PIG-5246) Modify bin/pig about SPARK_HOME, SPARK_ASSEMBLY_JAR after upgrading spark to 2

2017-06-01 Thread liyunzhang_intel (JIRA)
[ https://issues.apache.org/jira/browse/PIG-5246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16032609#comment-16032609 ] liyunzhang_intel edited comment on PIG-5246 at 6/1/17 7:57 AM: --- [~nkollar]: in

[jira] [Updated] (PIG-5246) Modify bin/pig about SPARK_HOME, SPARK_ASSEMBLY_JAR after upgrading spark to 2

2017-06-01 Thread liyunzhang_intel (JIRA)
[ https://issues.apache.org/jira/browse/PIG-5246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liyunzhang_intel updated PIG-5246: -- Attachment: PIG-5246.1.patch [~nkollar]: in PIG-5246.1.patch, modify {{spark16}} to {{spark1}}

[jira] [Updated] (PIG-5246) Modify bin/pig about SPARK_HOME, SPARK_ASSEMBLY_JAR after upgrading spark to 2

2017-06-01 Thread liyunzhang_intel (JIRA)
[ https://issues.apache.org/jira/browse/PIG-5246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liyunzhang_intel updated PIG-5246: -- Status: Patch Available (was: Open) > Modify bin/pig about SPARK_HOME, SPARK_ASSEMBLY_JAR after

[jira] [Updated] (PIG-5246) Modify bin/pig about SPARK_HOME, SPARK_ASSEMBLY_JAR after upgrading spark to 2

2017-06-01 Thread liyunzhang_intel (JIRA)
[ https://issues.apache.org/jira/browse/PIG-5246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liyunzhang_intel updated PIG-5246: -- Attachment: PIG-5246.patch [~nkollar], [~szita]: help review in spark2, spark-assembly*.jar does

[jira] [Commented] (PIG-5157) Upgrade to Spark 2.0

2017-06-01 Thread liyunzhang_intel (JIRA)
[ https://issues.apache.org/jira/browse/PIG-5157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16032514#comment-16032514 ] liyunzhang_intel commented on PIG-5157: --- [~nkollar]: have tested that we can remove JobLogger in

[jira] [Commented] (PIG-5247) Investigate stopOnFailure feature with Spark execution engine

2017-05-31 Thread liyunzhang_intel (JIRA)
[ https://issues.apache.org/jira/browse/PIG-5247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16032292#comment-16032292 ] liyunzhang_intel commented on PIG-5247: --- [~szita]: what i am confused is currently this feature is

[jira] [Commented] (PIG-5245) TestGrunt.testStopOnFailure is flaky

2017-05-31 Thread liyunzhang_intel (JIRA)
[ https://issues.apache.org/jira/browse/PIG-5245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16030847#comment-16030847 ] liyunzhang_intel commented on PIG-5245: --- [~rohini]: stop_on_failure is implemented in spark mode in

[jira] [Created] (PIG-5246) Modify bin/pig about SPARK_HOME, SPARK_ASSEMBLY_JAR after upgrading spark to 2

2017-05-31 Thread liyunzhang_intel (JIRA)
liyunzhang_intel created PIG-5246: - Summary: Modify bin/pig about SPARK_HOME, SPARK_ASSEMBLY_JAR after upgrading spark to 2 Key: PIG-5246 URL: https://issues.apache.org/jira/browse/PIG-5246 Project:

[jira] [Comment Edited] (PIG-5157) Upgrade to Spark 2.0

2017-05-31 Thread liyunzhang_intel (JIRA)
[ https://issues.apache.org/jira/browse/PIG-5157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16030799#comment-16030799 ] liyunzhang_intel edited comment on PIG-5157 at 5/31/17 7:54 AM: [~nkollar]:

[jira] [Commented] (PIG-5157) Upgrade to Spark 2.0

2017-05-31 Thread liyunzhang_intel (JIRA)
[ https://issues.apache.org/jira/browse/PIG-5157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16030799#comment-16030799 ] liyunzhang_intel commented on PIG-5157: --- [~nkollar]: bq. in JobMetricsListener.java there's a huge

[jira] [Commented] (PIG-5157) Upgrade to Spark 2.0

2017-05-29 Thread liyunzhang_intel (JIRA)
[ https://issues.apache.org/jira/browse/PIG-5157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16028651#comment-16028651 ] liyunzhang_intel commented on PIG-5157: --- [~nkollar]: will review tomorrow as Monday and Tuesday, i am

[jira] [Commented] (PIG-5215) Merge changes from review board to spark branch

2017-05-26 Thread liyunzhang_intel (JIRA)
[ https://issues.apache.org/jira/browse/PIG-5215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16025908#comment-16025908 ] liyunzhang_intel commented on PIG-5215: --- [~szita]: the only possible change is PIG-5167. If [~nkollar]

[jira] [Comment Edited] (PIG-5215) Merge changes from review board to spark branch

2017-05-26 Thread liyunzhang_intel (JIRA)
[ https://issues.apache.org/jira/browse/PIG-5215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16025902#comment-16025902 ] liyunzhang_intel edited comment on PIG-5215 at 5/26/17 7:01 AM: [~szita]: i

[jira] [Commented] (PIG-5215) Merge changes from review board to spark branch

2017-05-26 Thread liyunzhang_intel (JIRA)
[ https://issues.apache.org/jira/browse/PIG-5215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16025902#comment-16025902 ] liyunzhang_intel commented on PIG-5215: --- [~szita]: sorry, the commit fail, i will commit soon. >

[jira] [Commented] (PIG-5167) Limit_4 is failing with spark exec type

2017-05-25 Thread liyunzhang_intel (JIRA)
[ https://issues.apache.org/jira/browse/PIG-5167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16025723#comment-16025723 ] liyunzhang_intel commented on PIG-5167: --- [~nkollar]: my suggestion is 1. add a new verify_pig_script

[jira] [Updated] (PIG-5215) Merge changes from review board to spark branch

2017-05-25 Thread liyunzhang_intel (JIRA)
[ https://issues.apache.org/jira/browse/PIG-5215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liyunzhang_intel updated PIG-5215: -- Attachment: PIG-5215.5.patch [~szita]: thanks for fix. Include PIG-5215.4.fixes.patch and

[jira] [Reopened] (PIG-5167) Limit_4 is failing with spark exec type

2017-05-25 Thread liyunzhang_intel (JIRA)
[ https://issues.apache.org/jira/browse/PIG-5167?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liyunzhang_intel reopened PIG-5167: --- reopen it as Rohini suggested to fix it in another method: bq.Testing distinct + orderby + limit

[jira] [Updated] (PIG-5215) Merge changes from review board to spark branch

2017-05-25 Thread liyunzhang_intel (JIRA)
[ https://issues.apache.org/jira/browse/PIG-5215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liyunzhang_intel updated PIG-5215: -- Attachment: PIG-5215.4.patch [~sztia]: update latest PIG-5215.4.patch. > Merge changes from

[jira] [Updated] (PIG-5215) Merge changes from review board to spark branch

2017-05-25 Thread liyunzhang_intel (JIRA)
[ https://issues.apache.org/jira/browse/PIG-5215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liyunzhang_intel updated PIG-5215: -- Attachment: (was: PIG-5215.4.patch) > Merge changes from review board to spark branch >

[jira] [Updated] (PIG-5215) Merge changes from review board to spark branch

2017-05-25 Thread liyunzhang_intel (JIRA)
[ https://issues.apache.org/jira/browse/PIG-5215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liyunzhang_intel updated PIG-5215: -- Attachment: PIG-5215.4.patch > Merge changes from review board to spark branch >

[jira] [Created] (PIG-5241) Specify the hdfs path directly to spark and avoid the unnecessary download and upload in SparkLauncher.java

2017-05-25 Thread liyunzhang_intel (JIRA)
liyunzhang_intel created PIG-5241: - Summary: Specify the hdfs path directly to spark and avoid the unnecessary download and upload in SparkLauncher.java Key: PIG-5241 URL:

[jira] [Commented] (PIG-5135) HDFS bytes read stats are always 0 in Spark mode

2017-05-24 Thread liyunzhang_intel (JIRA)
[ https://issues.apache.org/jira/browse/PIG-5135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16024218#comment-16024218 ] liyunzhang_intel commented on PIG-5135: --- [~szita]: bq.I've checked this, it seems that

[jira] [Created] (PIG-5240) Fix TestPigRunner#simpleMultiQueryTest3 in spark mode for wrong inputStats

2017-05-24 Thread liyunzhang_intel (JIRA)
liyunzhang_intel created PIG-5240: - Summary: Fix TestPigRunner#simpleMultiQueryTest3 in spark mode for wrong inputStats Key: PIG-5240 URL: https://issues.apache.org/jira/browse/PIG-5240 Project: Pig

[jira] [Updated] (PIG-5239) Investigate why there are duplicated A[3,4] inTestLocationInPhysicalPlan#test in spark mode

2017-05-24 Thread liyunzhang_intel (JIRA)
[ https://issues.apache.org/jira/browse/PIG-5239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liyunzhang_intel updated PIG-5239: -- Issue Type: Sub-task (was: Bug) Parent: PIG-4059 > Investigate why there are duplicated

[jira] [Created] (PIG-5239) Investigate why there are duplicated A[3,4] inTestLocationInPhysicalPlan#test in spark mode

2017-05-24 Thread liyunzhang_intel (JIRA)
liyunzhang_intel created PIG-5239: - Summary: Investigate why there are duplicated A[3,4] inTestLocationInPhysicalPlan#test in spark mode Key: PIG-5239 URL: https://issues.apache.org/jira/browse/PIG-5239

[jira] [Commented] (PIG-5157) Upgrade to Spark 2.0

2017-05-24 Thread liyunzhang_intel (JIRA)
[ https://issues.apache.org/jira/browse/PIG-5157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16022446#comment-16022446 ] liyunzhang_intel commented on PIG-5157: --- [~nkollar]: bq. the optimizations offered (project Tungsten

[jira] [Commented] (PIG-5157) Upgrade to Spark 2.0

2017-05-21 Thread liyunzhang_intel (JIRA)
[ https://issues.apache.org/jira/browse/PIG-5157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16019070#comment-16019070 ] liyunzhang_intel commented on PIG-5157: --- [~zjffdu] and [~rohini]: thanks for your suggestion.

[jira] [Commented] (PIG-5157) Upgrade to Spark 2.0

2017-05-19 Thread liyunzhang_intel (JIRA)
[ https://issues.apache.org/jira/browse/PIG-5157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16017007#comment-16017007 ] liyunzhang_intel commented on PIG-5157: --- [~rohini],[~xuefuz],[~zjffdu]: Should we support spark2 or

[jira] [Commented] (PIG-5207) BugFix e2e tests fail on spark

2017-05-19 Thread liyunzhang_intel (JIRA)
[ https://issues.apache.org/jira/browse/PIG-5207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16016999#comment-16016999 ] liyunzhang_intel commented on PIG-5207: --- [~rohini]: can you spend some time to view the modification

[jira] [Assigned] (PIG-5199) exclude jline in spark dependency

2017-05-18 Thread liyunzhang_intel (JIRA)
[ https://issues.apache.org/jira/browse/PIG-5199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liyunzhang_intel reassigned PIG-5199: - Assignee: Adam Szita (was: liyunzhang_intel) > exclude jline in spark dependency >

[jira] [Commented] (PIG-5199) exclude jline in spark dependency

2017-05-18 Thread liyunzhang_intel (JIRA)
[ https://issues.apache.org/jira/browse/PIG-5199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16015301#comment-16015301 ] liyunzhang_intel commented on PIG-5199: --- [~szita]: 1 question, it seems that pig-0.17-0.SNAPSHOT.jar

[jira] [Updated] (PIG-5199) exclude jline in spark dependency

2017-05-18 Thread liyunzhang_intel (JIRA)
[ https://issues.apache.org/jira/browse/PIG-5199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liyunzhang_intel updated PIG-5199: -- Resolution: Fixed Status: Resolved (was: Patch Available) [~szita]: LGTM, commit to the

[jira] [Commented] (PIG-5135) HDFS bytes read stats are always 0 in Spark mode

2017-05-18 Thread liyunzhang_intel (JIRA)
[ https://issues.apache.org/jira/browse/PIG-5135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16015285#comment-16015285 ] liyunzhang_intel commented on PIG-5135: --- [~rohini]: there is some problem in last checkin causing the

[jira] [Issue Comment Deleted] (PIG-5228) Orc_2 is failing with spark exec type

2017-05-15 Thread liyunzhang_intel (JIRA)
[ https://issues.apache.org/jira/browse/PIG-5228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liyunzhang_intel updated PIG-5228: -- Comment: was deleted (was: [~szita] and [~rohini]: commit to the spark branch, thanks for

[jira] [Commented] (PIG-5228) Orc_2 is failing with spark exec type

2017-05-15 Thread liyunzhang_intel (JIRA)
[ https://issues.apache.org/jira/browse/PIG-5228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16010032#comment-16010032 ] liyunzhang_intel commented on PIG-5228: --- [~szita] and [~rohini]: commit to the spark branch, thanks

[jira] [Updated] (PIG-5215) Merge changes from review board to spark branch

2017-05-07 Thread liyunzhang_intel (JIRA)
[ https://issues.apache.org/jira/browse/PIG-5215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liyunzhang_intel updated PIG-5215: -- Issue Type: Bug (was: Sub-task) Parent: (was: PIG-4059) > Merge changes from review

[jira] [Commented] (PIG-5218) Jyhton_Checkin_3 fails with spark exec type

2017-05-07 Thread liyunzhang_intel (JIRA)
[ https://issues.apache.org/jira/browse/PIG-5218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16000247#comment-16000247 ] liyunzhang_intel commented on PIG-5218: --- [~rohini]: commit to the branch. [~szita]: thanks for

[jira] [Commented] (PIG-5199) exclude jline in spark dependency

2017-05-05 Thread liyunzhang_intel (JIRA)
[ https://issues.apache.org/jira/browse/PIG-5199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15997962#comment-15997962 ] liyunzhang_intel commented on PIG-5199: --- [~szita]: based on current branch code(d6bf437), if we change

[jira] [Commented] (PIG-5215) Merge changes from review board to spark branch

2017-05-05 Thread liyunzhang_intel (JIRA)
[ https://issues.apache.org/jira/browse/PIG-5215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15997920#comment-15997920 ] liyunzhang_intel commented on PIG-5215: --- [~szita]: thanks for review. Commit to branch. > Merge

[jira] [Updated] (PIG-5215) Merge changes from review board to spark branch

2017-05-04 Thread liyunzhang_intel (JIRA)
[ https://issues.apache.org/jira/browse/PIG-5215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liyunzhang_intel updated PIG-5215: -- Attachment: PIG-5215.3.patch [~szita]: help review PIG-5215.3.patch changes: 1.remove unecessary

[jira] [Updated] (PIG-4854) Merge spark branch to trunk

2017-05-03 Thread liyunzhang_intel (JIRA)
[ https://issues.apache.org/jira/browse/PIG-4854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liyunzhang_intel updated PIG-4854: -- Resolution: Fixed Status: Resolved (was: Patch Available) duplicate with PIG-5215 >

[jira] [Commented] (PIG-5228) Orc_2 is failing with spark exec type

2017-05-01 Thread liyunzhang_intel (JIRA)
[ https://issues.apache.org/jira/browse/PIG-5228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15991940#comment-15991940 ] liyunzhang_intel commented on PIG-5228: --- [~szita]: {quote} liyunzhang_intel this still fails on my

[jira] [Commented] (PIG-5228) Orc_2 is failing with spark exec type

2017-04-28 Thread liyunzhang_intel (JIRA)
[ https://issues.apache.org/jira/browse/PIG-5228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15988443#comment-15988443 ] liyunzhang_intel commented on PIG-5228: --- [~szita]: Orc_2 does not fail in my env with branch code(base

  1   2   3   4   5   6   7   8   9   10   >