[jira] [Commented] (PIG-5167) Limit_4 is failing with spark exec type

[ https://issues.apache.org/jira/browse/PIG-5167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16025723#comment-16025723 ] liyunzhang_intel commented on PIG-5167: --- [~nkollar]: my suggestion is 1. add a new verify_pig_script

[jira] [Updated] (PIG-5215) Merge changes from review board to spark branch

[ https://issues.apache.org/jira/browse/PIG-5215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liyunzhang_intel updated PIG-5215: -- Attachment: PIG-5215.5.patch [~szita]: thanks for fix. Include PIG-5215.4.fixes.patch and

[jira] [Reopened] (PIG-5167) Limit_4 is failing with spark exec type

[ https://issues.apache.org/jira/browse/PIG-5167?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liyunzhang_intel reopened PIG-5167: --- reopen it as Rohini suggested to fix it in another method: bq.Testing distinct + orderby + limit

[jira] [Commented] (PIG-5194) HiveUDF fails with Spark exec type

[ https://issues.apache.org/jira/browse/PIG-5194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16025453#comment-16025453 ] Adam Szita commented on PIG-5194: - Thanks for the review [~daijy]. As discussed with [~rohini] I'll commit

[jira] [Commented] (PIG-5215) Merge changes from review board to spark branch

[ https://issues.apache.org/jira/browse/PIG-5215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16025449#comment-16025449 ] Adam Szita commented on PIG-5215: - [~kellyzly] The result of the unit test (run on spark exec type only)

[jira] [Updated] (PIG-5215) Merge changes from review board to spark branch

[ https://issues.apache.org/jira/browse/PIG-5215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Adam Szita updated PIG-5215: Attachment: PIG-5215.4.TestCombinerFix.patch > Merge changes from review board to spark branch >

[jira] [Commented] (PIG-4662) New optimizer rule: filter nulls before inner joins

[ https://issues.apache.org/jira/browse/PIG-4662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16025387#comment-16025387 ] Daniel Dai commented on PIG-4662: - I don't think it would make noticeable performance difference going

[jira] [Resolved] (PIG-5231) PigStorage with -schema may produce inconsistent outputs with more fields

[ https://issues.apache.org/jira/browse/PIG-5231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Koji Noguchi resolved PIG-5231. --- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 0.17.0 Thanks for the review

[jira] [Resolved] (PIG-5224) Extra foreach from ColumnPrune preventing Accumulator usage

[ https://issues.apache.org/jira/browse/PIG-5224?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Koji Noguchi resolved PIG-5224. --- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 0.17.0 Thanks for the review

[jira] [Commented] (PIG-4662) New optimizer rule: filter nulls before inner joins

[ https://issues.apache.org/jira/browse/PIG-4662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16025317#comment-16025317 ] Rohini Palaniswamy commented on PIG-4662: - bq. I prefer to do it in optimizer, it seems to be more

[jira] [Commented] (PIG-5194) HiveUDF fails with Spark exec type

[ https://issues.apache.org/jira/browse/PIG-5194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16025249#comment-16025249 ] Daniel Dai commented on PIG-5194: - +1 for the HiveUDAF change, thanks for catching this! > HiveUDF fails

[jira] [Commented] (PIG-5231) PigStorage with -schema may produce inconsistent outputs with more fields

[ https://issues.apache.org/jira/browse/PIG-5231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16025155#comment-16025155 ] Daniel Dai commented on PIG-5231: - Vote for 3. We pick the first schema in dirs in all LoadFunc, such as

[jira] [Assigned] (PIG-5241) Specify the hdfs path directly to spark and avoid the unnecessary download and upload in SparkLauncher.java

[ https://issues.apache.org/jira/browse/PIG-5241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nandor Kollar reassigned PIG-5241: -- Assignee: Nandor Kollar > Specify the hdfs path directly to spark and avoid the unnecessary

[jira] [Commented] (PIG-5224) Extra foreach from ColumnPrune preventing Accumulator usage

[ https://issues.apache.org/jira/browse/PIG-5224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16024946#comment-16024946 ] Koji Noguchi commented on PIG-5224: --- bq. That's only if user write "foreach" statement carefully. If he

[jira] [Commented] (PIG-5224) Extra foreach from ColumnPrune preventing Accumulator usage

[ https://issues.apache.org/jira/browse/PIG-5224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16024919#comment-16024919 ] Daniel Dai commented on PIG-5224: - bq. Well, if next LOForEach is not removing all the columns which are not

Re: [ANNOUNCE] Welcome new Pig Committer - Adam Szita

Congratulation Adam! Well deserved. On 5/25/17, 1:48 AM, "Adam Szita" wrote: Thanks for all the support! Adam On 23 May 2017 at 10:56, gaurav gupta wrote: > Congratulations Adam :) > > On Tue, May 23, 2017 at

[jira] [Commented] (PIG-5215) Merge changes from review board to spark branch

[ https://issues.apache.org/jira/browse/PIG-5215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16024862#comment-16024862 ] Adam Szita commented on PIG-5215: - [~kellyzly] I've attached [^PIG-5215.4.fixes.patch] which you can apply

[jira] [Updated] (PIG-5215) Merge changes from review board to spark branch

[ https://issues.apache.org/jira/browse/PIG-5215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Adam Szita updated PIG-5215: Attachment: PIG-5215.4.fixes.patch > Merge changes from review board to spark branch >

[jira] [Commented] (PIG-5135) HDFS bytes read stats are always 0 in Spark mode

[ https://issues.apache.org/jira/browse/PIG-5135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16024858#comment-16024858 ] Adam Szita commented on PIG-5135: - [~kellyzly] I just realized you already did this change in one of your

[jira] [Resolved] (PIG-5235) Typecast with as-clause fails for tuple/bag with an empty schema

[ https://issues.apache.org/jira/browse/PIG-5235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Koji Noguchi resolved PIG-5235. --- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 0.17.0 Thanks for the review

[jira] [Created] (PIG-5243) describe with typecast on as-clause shows the types before the typecasting

Koji Noguchi created PIG-5243: - Summary: describe with typecast on as-clause shows the types before the typecasting Key: PIG-5243 URL: https://issues.apache.org/jira/browse/PIG-5243 Project: Pig

[jira] [Updated] (PIG-5224) Extra foreach from ColumnPrune preventing Accumulator usage

[ https://issues.apache.org/jira/browse/PIG-5224?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Koji Noguchi updated PIG-5224: -- Attachment: pig-5224-v2.patch {quote} The inserted LOForEach remove all the columns which are not used in

[jira] [Updated] (PIG-5201) Null handling on FLATTEN

[ https://issues.apache.org/jira/browse/PIG-5201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Koji Noguchi updated PIG-5201: -- Fix Version/s: 0.17.0 It'll be nice if we can fix this for 0.17 since this leads to incorrect outputs

Re: Review Request 57317: Support Pig On Spark

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/57317/ --- (Updated May 25, 2017, 2:03 p.m.) Review request for pig, Daniel Dai and

[jira] [Commented] (PIG-5135) HDFS bytes read stats are always 0 in Spark mode

[ https://issues.apache.org/jira/browse/PIG-5135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16024736#comment-16024736 ] Adam Szita commented on PIG-5135: - [~kellyzly]: fair point. I've done the suggested modifications in

[jira] [Updated] (PIG-5135) HDFS bytes read stats are always 0 in Spark mode

[ https://issues.apache.org/jira/browse/PIG-5135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Adam Szita updated PIG-5135: Attachment: PIG-5135.smallfixes.patch > HDFS bytes read stats are always 0 in Spark mode >

[jira] [Updated] (PIG-5238) Fix datetime related test issues after PIG-4748

[ https://issues.apache.org/jira/browse/PIG-5238?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Koji Noguchi updated PIG-5238: -- Fix Version/s: 0.17.0 > Fix datetime related test issues after PIG-4748 >

[jira] [Updated] (PIG-5240) Fix TestPigRunner#simpleMultiQueryTest3 in spark mode for wrong inputStats

[ https://issues.apache.org/jira/browse/PIG-5240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Adam Szita updated PIG-5240: Summary: Fix TestPigRunner#simpleMultiQueryTest3 in spark mode for wrong inputStats (was: Fix TestPigRunner

[jira] [Updated] (PIG-5240) Fix TestPigRunner in spark mode for wrong inputStats

[ https://issues.apache.org/jira/browse/PIG-5240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Adam Szita updated PIG-5240: Summary: Fix TestPigRunner in spark mode for wrong inputStats (was: Fix TestPigRunner#simpleMultiQueryTest3

[jira] [Updated] (PIG-5240) Fix TestPigRunner in spark mode

[ https://issues.apache.org/jira/browse/PIG-5240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Adam Szita updated PIG-5240: Summary: Fix TestPigRunner in spark mode (was: Fix TestPigRunner in spark mode for wrong inputStats) > Fix

Build failed in Jenkins: Pig-trunk-commit #2489

See -- [...truncated 167.20 KB...] A contrib/piggybank/java/src/main/java/org/apache/pig/piggybank/evaluation/datetime A

Build failed in Jenkins: Pig-trunk-commit #2488

See -- [...truncated 173.03 KB...] A test/org/apache/pig/test/pigunit A test/org/apache/pig/test/pigunit/TestPigTest.java A test/org/apache/pig/test/pigunit/pig A

Build failed in Jenkins: Pig-trunk-commit #2487

See -- [...truncated 166.97 KB...] A contrib/piggybank/java/src/main/java/org/apache/pig/piggybank/evaluation/math/EXPM1.java A

Build failed in Jenkins: Pig-trunk-commit #2486

See Changes: [szita] PIG-5238: Fix datetime related test issues after PIG-4748 (szita) [szita] PIG-3103: make mockito a test dependency (instead of compile) (nkollar via szita)

[jira] [Created] (PIG-5242) Evaluate DataFrame API for Pig on Spark

Nandor Kollar created PIG-5242: -- Summary: Evaluate DataFrame API for Pig on Spark Key: PIG-5242 URL: https://issues.apache.org/jira/browse/PIG-5242 Project: Pig Issue Type: Improvement

[jira] [Updated] (PIG-5238) Fix datetime related test issues after PIG-4748

[ https://issues.apache.org/jira/browse/PIG-5238?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Adam Szita updated PIG-5238: Resolution: Fixed Status: Resolved (was: Patch Available) > Fix datetime related test issues after

[jira] [Commented] (PIG-5238) Fix datetime related test issues after PIG-4748

[ https://issues.apache.org/jira/browse/PIG-5238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16024461#comment-16024461 ] Adam Szita commented on PIG-5238: - [^PIG-5238.0.patch] committed to trunk, thanks for the review [~nkollar],

Re: Review Request 57317: Support Pig On Spark

> On May 24, 2017, 8:55 p.m., Rohini Palaniswamy wrote: > > test/e2e/pig/tests/nightly.conf > > Lines 2307 (patched) > > > > > > Testing distinct + orderby + limit serves the same purpose as orderby + > > limit

[jira] [Commented] (PIG-3103) make mockito a test dependency (instead of compile)

[ https://issues.apache.org/jira/browse/PIG-3103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16024443#comment-16024443 ] Adam Szita commented on PIG-3103: - [^PIG-3103.patch] committed to trunk, thanks [~nkollar]! > make mockito

[jira] [Updated] (PIG-3103) make mockito a test dependency (instead of compile)

[ https://issues.apache.org/jira/browse/PIG-3103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Adam Szita updated PIG-3103: Resolution: Fixed Status: Resolved (was: Patch Available) > make mockito a test dependency (instead

Re: [ANNOUNCE] Welcome new Pig Committer - Adam Szita

Thanks for all the support! Adam On 23 May 2017 at 10:56, gaurav gupta wrote: > Congratulations Adam :) > > On Tue, May 23, 2017 at 11:54 AM, Jeff Zhang wrote: > > > [image: Boxbe] This message is eligible > > for

[jira] [Updated] (PIG-5215) Merge changes from review board to spark branch

[ https://issues.apache.org/jira/browse/PIG-5215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liyunzhang_intel updated PIG-5215: -- Attachment: PIG-5215.4.patch [~sztia]: update latest PIG-5215.4.patch. > Merge changes from

[jira] [Updated] (PIG-5215) Merge changes from review board to spark branch

[ https://issues.apache.org/jira/browse/PIG-5215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liyunzhang_intel updated PIG-5215: -- Attachment: (was: PIG-5215.4.patch) > Merge changes from review board to spark branch >

Re: Review Request 57317: Support Pig On Spark

> On May 24, 2017, 8:55 p.m., Rohini Palaniswamy wrote: > > src/org/apache/pig/backend/hadoop/executionengine/spark/SparkPigContext.java > > Lines 51 (patched) > > > > > > Shouldn't default parallelism returned if

Re: Review Request 57317: Support Pig On Spark

> On March 21, 2017, 8:36 p.m., Rohini Palaniswamy wrote: > > src/org/apache/pig/backend/hadoop/executionengine/spark/SparkLauncher.java > > Lines 179 (patched) > > > > > > You can reuse ScriptState id

[jira] [Comment Edited] (PIG-3021) Split results missing records when there is null values in the column comparison

[ https://issues.apache.org/jira/browse/PIG-3021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16024340#comment-16024340 ] Nian Ji edited comment on PIG-3021 at 5/25/17 7:27 AM: --- [~daijy], thank you. I will

[jira] [Commented] (PIG-3021) Split results missing records when there is null values in the column comparison

[ https://issues.apache.org/jira/browse/PIG-3021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16024340#comment-16024340 ] Nian Ji commented on PIG-3021: -- Daniel Dai, thank you. I will create another Jira and add a patch with adding

Build failed in Jenkins: Pig-trunk-commit #2485

See Changes: [daijy] PIG-3021: Split results missing records when there is null values in the column comparison -- [...truncated 166.79 KB...] A

[jira] [Updated] (PIG-5157) Upgrade to Spark 2.0

[ https://issues.apache.org/jira/browse/PIG-5157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nandor Kollar updated PIG-5157: --- Status: Patch Available (was: Open) > Upgrade to Spark 2.0 > > >

Re: Review Request 59530: PIG-5157 Upgrade to Spark 2.0

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/59530/ --- (Updated May 25, 2017, 7:13 a.m.) Review request for pig, liyun zhang, Rohini

[jira] [Updated] (PIG-5215) Merge changes from review board to spark branch

[ https://issues.apache.org/jira/browse/PIG-5215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liyunzhang_intel updated PIG-5215: -- Attachment: PIG-5215.4.patch > Merge changes from review board to spark branch >

[jira] [Resolved] (PIG-3021) Split results missing records when there is null values in the column comparison

[ https://issues.apache.org/jira/browse/PIG-3021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Dai resolved PIG-3021. - Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 0.17.0 +1 for PIG-3021-4.patch. Patch

[jira] [Commented] (PIG-5224) Extra foreach from ColumnPrune preventing Accumulator usage

[ https://issues.apache.org/jira/browse/PIG-5224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16024320#comment-16024320 ] Daniel Dai commented on PIG-5224: - The inserted LOForEach remove all the columns which are not used in the

[jira] [Commented] (PIG-5235) Typecast with as-clause fails for tuple/bag with an empty schema

[ https://issues.apache.org/jira/browse/PIG-5235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16024288#comment-16024288 ] Daniel Dai commented on PIG-5235: - +1 > Typecast with as-clause fails for tuple/bag with an empty schema >

[jira] [Created] (PIG-5241) Specify the hdfs path directly to spark and avoid the unnecessary download and upload in SparkLauncher.java

liyunzhang_intel created PIG-5241: - Summary: Specify the hdfs path directly to spark and avoid the unnecessary download and upload in SparkLauncher.java Key: PIG-5241 URL:

[jira] Subscription: PIG patch available

Issue Subscription Filter: PIG patch available (38 issues) Subscriber: pigdaily Key Summary PIG-5238Fix datetime related test issues after PIG-4748 https://issues.apache.org/jira/browse/PIG-5238 PIG-5236json simple jar not included automatically while trying to load