Does Pig Re-Use FileInputLoadFuncs Objects?

2010-08-31 Thread Russell Jurney
Pardon the cross-post: Does Pig ever re-use FileInputLoadFunc objects? We suspect state is being retained between different stores, but we don't actually know this. Figured I'd ask to verify the hunch. Our load func for our in-house format works fine with Pig scripts normally... but I have a pig

[jira] Commented: (PIG-1501) need to investigate the impact of compression on pig performance

2010-08-31 Thread Yan Zhou (JIRA)
[ https://issues.apache.org/jira/browse/PIG-1501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12904868#action_12904868 ] Yan Zhou commented on PIG-1501: --- To be more eaccurate, the default compression would be gzip if

[jira] Updated: (PIG-1373) We need to add jdiff output to docs on the website

2010-08-31 Thread Olga Natkovich (JIRA)
[ https://issues.apache.org/jira/browse/PIG-1373?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Olga Natkovich updated PIG-1373: Status: Resolved (was: Patch Available) Resolution: Fixed > We need to add jdiff output to d

[jira] Updated: (PIG-1319) New logical optimization rules

2010-08-31 Thread Olga Natkovich (JIRA)
[ https://issues.apache.org/jira/browse/PIG-1319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Olga Natkovich updated PIG-1319: Fix Version/s: 0.9.0 (was: 0.8.0) > New logical optimization rules > -

[jira] Updated: (PIG-747) Logical to Physical Plan Translation fails when temporary alias are created within foreach

2010-08-31 Thread Olga Natkovich (JIRA)
[ https://issues.apache.org/jira/browse/PIG-747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Olga Natkovich updated PIG-747: --- Fix Version/s: 0.9.0 (was: 0.8.0) > Logical to Physical Plan Translation fails wh

[jira] Resolved: (PIG-1537) Column pruner causes wrong results when using both Custom Store UDF and PigStorage

2010-08-31 Thread Olga Natkovich (JIRA)
[ https://issues.apache.org/jira/browse/PIG-1537?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Olga Natkovich resolved PIG-1537. - Resolution: Fixed > Column pruner causes wrong results when using both Custom Store UDF and > PigS

[jira] Resolved: (PIG-1588) Parameter pre-processing of values containing pig positional variables ($0, $1 etc)

2010-08-31 Thread Olga Natkovich (JIRA)
[ https://issues.apache.org/jira/browse/PIG-1588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Olga Natkovich resolved PIG-1588. - Resolution: Duplicate This is duplicate of https://issues.apache.org/jira/browse/PIG-1586 and at th

[jira] Created: (PIG-1588) Parameter pre-processing of values containing pig positional variables ($0, $1 etc)

2010-08-31 Thread Laukik Chitnis (JIRA)
Parameter pre-processing of values containing pig positional variables ($0, $1 etc) --- Key: PIG-1588 URL: https://issues.apache.org/jira/browse/PIG-1588 Project: Pig

[jira] Assigned: (PIG-1586) Parameter subsitution using -param option runs into problems when substituing entire pig statements in a shell script (maybe this is a bash problem)

2010-08-31 Thread Olga Natkovich (JIRA)
[ https://issues.apache.org/jira/browse/PIG-1586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Olga Natkovich reassigned PIG-1586: --- Assignee: Viraj Bhat Viraj volunteered to print the line that pig gets as part of parameter su

[jira] Created: (PIG-1587) Cloning utility functions for new logical plan

2010-08-31 Thread Daniel Dai (JIRA)
Cloning utility functions for new logical plan -- Key: PIG-1587 URL: https://issues.apache.org/jira/browse/PIG-1587 Project: Pig Issue Type: Improvement Components: impl Affects Versi

[jira] Updated: (PIG-1586) Parameter subsitution using -param option runs into problems when substituing entire pig statements in a shell script (maybe this is a bash problem)

2010-08-31 Thread Viraj Bhat (JIRA)
[ https://issues.apache.org/jira/browse/PIG-1586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Viraj Bhat updated PIG-1586: Description: I have a Pig script as a template: {code} register Countwords.jar; A = $INPUT; B = FOREACH A GE

[jira] Created: (PIG-1586) Parameter subsitution using -param option runs into problems when substituing entire pig statements in a shell script (maybe this is a bash problem)

2010-08-31 Thread Viraj Bhat (JIRA)
Parameter subsitution using -param option runs into problems when substituing entire pig statements in a shell script (maybe this is a bash problem)

[jira] Commented: (PIG-1501) need to investigate the impact of compression on pig performance

2010-08-31 Thread Olga Natkovich (JIRA)
[ https://issues.apache.org/jira/browse/PIG-1501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12904848#action_12904848 ] Olga Natkovich commented on PIG-1501: - Ashutosh, The reason it is off by default is beca

[jira] Updated: (PIG-1572) change default datatype when relations are used as scalar to bytearray

2010-08-31 Thread Thejas M Nair (JIRA)
[ https://issues.apache.org/jira/browse/PIG-1572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thejas M Nair updated PIG-1572: --- Attachment: PIG-1572.2.patch PIG-1572.2.patch - Fixed loss of lineage information in translation durin

[jira] Commented: (PIG-1501) need to investigate the impact of compression on pig performance

2010-08-31 Thread Ashutosh Chauhan (JIRA)
[ https://issues.apache.org/jira/browse/PIG-1501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12904843#action_12904843 ] Ashutosh Chauhan commented on PIG-1501: --- If its not backward-incompatible then is there

[jira] Created: (PIG-1585) Add new properties to help and documentation

2010-08-31 Thread Olga Natkovich (JIRA)
Add new properties to help and documentation Key: PIG-1585 URL: https://issues.apache.org/jira/browse/PIG-1585 Project: Pig Issue Type: Bug Reporter: Olga Natkovich Assigne

[jira] Commented: (PIG-1506) Need to clarify the difference between null handling in JOIN and COGROUP

2010-08-31 Thread Scott Carey (JIRA)
[ https://issues.apache.org/jira/browse/PIG-1506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12904835#action_12904835 ] Scott Carey commented on PIG-1506: -- I have just confirmed that on 0.7 it works fine, but 0.5

[jira] Updated: (PIG-1501) need to investigate the impact of compression on pig performance

2010-08-31 Thread Yan Zhou (JIRA)
[ https://issues.apache.org/jira/browse/PIG-1501?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yan Zhou updated PIG-1501: -- Release Note: This feature will save HDFS space used to store the intermediate data used by PIG and potentially

[jira] Updated: (PIG-1501) need to investigate the impact of compression on pig performance

2010-08-31 Thread Yan Zhou (JIRA)
[ https://issues.apache.org/jira/browse/PIG-1501?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yan Zhou updated PIG-1501: -- Release Note: This feature will save HDFS space used to store the intermediate data used by PIG and potentially

[jira] Commented: (PIG-1506) Need to clarify the difference between null handling in JOIN and COGROUP

2010-08-31 Thread Olga Natkovich (JIRA)
[ https://issues.apache.org/jira/browse/PIG-1506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12904829#action_12904829 ] Olga Natkovich commented on PIG-1506: - I verified that 0.8 code does deal correctly with

[jira] Updated: (PIG-1583) piggybank unit test TestLookupInFiles is broken

2010-08-31 Thread Giridharan Kesavan (JIRA)
[ https://issues.apache.org/jira/browse/PIG-1583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Giridharan Kesavan updated PIG-1583: Status: Open (was: Patch Available) submitting to hudson > piggybank unit test TestLookupI

[jira] Updated: (PIG-1583) piggybank unit test TestLookupInFiles is broken

2010-08-31 Thread Giridharan Kesavan (JIRA)
[ https://issues.apache.org/jira/browse/PIG-1583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Giridharan Kesavan updated PIG-1583: Status: Patch Available (was: Open) > piggybank unit test TestLookupInFiles is broken >

[jira] Created: (PIG-1584) deal with inner cogroup

2010-08-31 Thread Olga Natkovich (JIRA)
deal with inner cogroup --- Key: PIG-1584 URL: https://issues.apache.org/jira/browse/PIG-1584 Project: Pig Issue Type: Bug Reporter: Olga Natkovich Fix For: 0.9.0 The current implementation of i

[jira] Commented: (PIG-1506) Need to clarify the difference between null handling in JOIN and COGROUP

2010-08-31 Thread Scott Carey (JIRA)
[ https://issues.apache.org/jira/browse/PIG-1506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12904819#action_12904819 ] Scott Carey commented on PIG-1506: -- The SQL behavior of the above for an outer join would be

[jira] Commented: (PIG-1399) Logical Optimizer: Expression optimizor rule

2010-08-31 Thread Alan Gates (JIRA)
[ https://issues.apache.org/jira/browse/PIG-1399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12904808#action_12904808 ] Alan Gates commented on PIG-1399: - [exec] +1 overall. [exec] [exec] +1

[jira] Commented: (PIG-1506) Need to clarify the difference between null handling in JOIN and COGROUP

2010-08-31 Thread Olga Natkovich (JIRA)
[ https://issues.apache.org/jira/browse/PIG-1506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12904785#action_12904785 ] Olga Natkovich commented on PIG-1506: - This is what we need to document: In the case of

[jira] Updated: (PIG-1583) piggybank unit test TestLookupInFiles is broken

2010-08-31 Thread Xuefu Zhang (JIRA)
[ https://issues.apache.org/jira/browse/PIG-1583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuefu Zhang updated PIG-1583: - Status: Patch Available (was: Open) > piggybank unit test TestLookupInFiles is broken > --

[jira] Commented: (PIG-1583) piggybank unit test TestLookupInFiles is broken

2010-08-31 Thread Xuefu Zhang (JIRA)
[ https://issues.apache.org/jira/browse/PIG-1583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12904783#action_12904783 ] Xuefu Zhang commented on PIG-1583: -- +1 Patch Looks Good. > piggybank unit test TestLookupIn

RE: Branching for Pig 0.8 release

2010-08-31 Thread Olga Natkovich
The branch has been created. Only bug fixes related to 0.8 release should be committed there. When committing on the branch, please, make sure to make (and test) the corresponding changes on the trunk. Thanks, Olga -Original Message- From: Olga Natkovich [mailto:ol...@yahoo-inc.com] S

Branching for Pig 0.8 release

2010-08-31 Thread Olga Natkovich
Hi, I am about to branch for release. Please, hold off your commits till I am done. I will send a follow up email at that time. Thanks, Olga

[jira] Updated: (PIG-1583) piggybank unit test TestLookupInFiles is broken

2010-08-31 Thread Daniel Dai (JIRA)
[ https://issues.apache.org/jira/browse/PIG-1583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Dai updated PIG-1583: Attachment: PIG-1583-1.patch > piggybank unit test TestLookupInFiles is broken >

[jira] Updated: (PIG-1583) piggybank unit test TestLookupInFiles is broken

2010-08-31 Thread Daniel Dai (JIRA)
[ https://issues.apache.org/jira/browse/PIG-1583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Dai updated PIG-1583: Attachment: (was: PIG-1583-1.patch) > piggybank unit test TestLookupInFiles is broken > -

[jira] Updated: (PIG-1583) piggybank unit test TestLookupInFiles is broken

2010-08-31 Thread Daniel Dai (JIRA)
[ https://issues.apache.org/jira/browse/PIG-1583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Dai updated PIG-1583: Attachment: PIG-1583-1.patch > piggybank unit test TestLookupInFiles is broken >

[jira] Created: (PIG-1583) piggybank unit test TestLookupInFiles is broken

2010-08-31 Thread Daniel Dai (JIRA)
piggybank unit test TestLookupInFiles is broken --- Key: PIG-1583 URL: https://issues.apache.org/jira/browse/PIG-1583 Project: Pig Issue Type: Bug Components: impl Affects Versions: 0

[jira] Updated: (PIG-1582) upgrade commons-logging version with ivy

2010-08-31 Thread Giridharan Kesavan (JIRA)
[ https://issues.apache.org/jira/browse/PIG-1582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Giridharan Kesavan updated PIG-1582: Status: Resolved (was: Patch Available) Fix Version/s: 0.8.0 Resolution: Fi

[jira] Updated: (PIG-1582) upgrade commons-logging version with ivy

2010-08-31 Thread Giridharan Kesavan (JIRA)
[ https://issues.apache.org/jira/browse/PIG-1582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Giridharan Kesavan updated PIG-1582: Status: Patch Available (was: Open) > upgrade commons-logging version with ivy > ---

[jira] Updated: (PIG-1582) upgrade commons-logging version with ivy

2010-08-31 Thread Giridharan Kesavan (JIRA)
[ https://issues.apache.org/jira/browse/PIG-1582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Giridharan Kesavan updated PIG-1582: Attachment: pig-1582.patch > upgrade commons-logging version with ivy > -

[jira] Assigned: (PIG-1582) upgrade commons-logging version with ivy

2010-08-31 Thread Giridharan Kesavan (JIRA)
[ https://issues.apache.org/jira/browse/PIG-1582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Giridharan Kesavan reassigned PIG-1582: --- Assignee: Giridharan Kesavan > upgrade commons-logging version with ivy > -

[jira] Created: (PIG-1582) upgrade commons-logging version with ivy

2010-08-31 Thread Giridharan Kesavan (JIRA)
upgrade commons-logging version with ivy Key: PIG-1582 URL: https://issues.apache.org/jira/browse/PIG-1582 Project: Pig Issue Type: Improvement Components: build Reporter: Giridh

[jira] Updated: (PIG-1494) PIG Logical Optimization: Use CNF in PushUpFilter

2010-08-31 Thread Olga Natkovich (JIRA)
[ https://issues.apache.org/jira/browse/PIG-1494?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Olga Natkovich updated PIG-1494: Fix Version/s: (was: 0.8.0) > PIG Logical Optimization: Use CNF in PushUpFilter > ---

[jira] Updated: (PIG-1494) PIG Logical Optimization: Use CNF in PushUpFilter

2010-08-31 Thread Olga Natkovich (JIRA)
[ https://issues.apache.org/jira/browse/PIG-1494?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Olga Natkovich updated PIG-1494: Unlinking from 0.8 since we are about to branch for release > PIG Logical Optimization: Use CNF in PushU

[jira] Resolved: (PIG-1530) PIG Logical Optimization: Push LOFilter above LOCogroup

2010-08-31 Thread Olga Natkovich (JIRA)
[ https://issues.apache.org/jira/browse/PIG-1530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Olga Natkovich resolved PIG-1530. - Resolution: Duplicate Xuefu is addressing this issue as part of https://issues.apache.org/jira/bro

[jira] Updated: (PIG-1429) Add Boolean Data Type to Pig

2010-08-31 Thread Olga Natkovich (JIRA)
[ https://issues.apache.org/jira/browse/PIG-1429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Olga Natkovich updated PIG-1429: Fix Version/s: (was: 0.8.0) Unlinking because we are branching for release today > Add Boolean D

[jira] Updated: (PIG-1549) Provide utility to construct CNF form of predicates

2010-08-31 Thread Olga Natkovich (JIRA)
[ https://issues.apache.org/jira/browse/PIG-1549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Olga Natkovich updated PIG-1549: Fix Version/s: (was: 0.8.0) Unlinking from 0.8 release since we are about to branch > Provide ut

[jira] Updated: (PIG-1314) Add DateTime Support to Pig

2010-08-31 Thread Olga Natkovich (JIRA)
[ https://issues.apache.org/jira/browse/PIG-1314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Olga Natkovich updated PIG-1314: Fix Version/s: (was: 0.8.0) Unlinking from 0.8 since we are branching today > Add DateTime Suppo

[jira] Commented: (PIG-794) Use Avro serialization in Pig

2010-08-31 Thread Doug Cutting (JIRA)
[ https://issues.apache.org/jira/browse/PIG-794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12904687#action_12904687 ] Doug Cutting commented on PIG-794: -- A few comments about the attached code: - is there a rea

[jira] Commented: (PIG-794) Use Avro serialization in Pig

2010-08-31 Thread Scott Carey (JIRA)
[ https://issues.apache.org/jira/browse/PIG-794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12904683#action_12904683 ] Scott Carey commented on PIG-794: - bq. The performance of InterRecordWriter is much better th

[jira] Commented: (PIG-794) Use Avro serialization in Pig

2010-08-31 Thread Scott Carey (JIRA)
[ https://issues.apache.org/jira/browse/PIG-794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12904680#action_12904680 ] Scott Carey commented on PIG-794: - So a summary of the differences I can see quickly are: h5.

[jira] Commented: (PIG-794) Use Avro serialization in Pig

2010-08-31 Thread Scott Carey (JIRA)
[ https://issues.apache.org/jira/browse/PIG-794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12904674#action_12904674 ] Scott Carey commented on PIG-794: - AVRO-592 creates an AvroStorage class for writing and readi

[jira] Commented: (PIG-794) Use Avro serialization in Pig

2010-08-31 Thread Dmitriy V. Ryaboy (JIRA)
[ https://issues.apache.org/jira/browse/PIG-794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12904615#action_12904615 ] Dmitriy V. Ryaboy commented on PIG-794: --- Jeff, have you checkoed out Scott Carey's work

[jira] Commented: (PIG-794) Use Avro serialization in Pig

2010-08-31 Thread Jeff Zhang (JIRA)
[ https://issues.apache.org/jira/browse/PIG-794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12904575#action_12904575 ] Jeff Zhang commented on PIG-794: Attach the updated patch Avro_Strorage_3.patch ( I found one

[jira] Updated: (PIG-794) Use Avro serialization in Pig

2010-08-31 Thread Jeff Zhang (JIRA)
[ https://issues.apache.org/jira/browse/PIG-794?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Zhang updated PIG-794: --- Attachment: AvroStorage_3.patch > Use Avro serialization in Pig > - > >

[jira] Commented: (PIG-794) Use Avro serialization in Pig

2010-08-31 Thread Jeff Zhang (JIRA)
[ https://issues.apache.org/jira/browse/PIG-794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12904555#action_12904555 ] Jeff Zhang commented on PIG-794: Besides the above experiment, I also did a experiment to comp

[jira] Updated: (PIG-794) Use Avro serialization in Pig

2010-08-31 Thread Jeff Zhang (JIRA)
[ https://issues.apache.org/jira/browse/PIG-794?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Zhang updated PIG-794: --- Attachment: AvroTest.java > Use Avro serialization in Pig > - > > Ke

[jira] Commented: (PIG-794) Use Avro serialization in Pig

2010-08-31 Thread Jeff Zhang (JIRA)
[ https://issues.apache.org/jira/browse/PIG-794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12904551#action_12904551 ] Jeff Zhang commented on PIG-794: I did some experiment on Avro, Avro_Storage_2.patch is the de

[jira] Updated: (PIG-794) Use Avro serialization in Pig

2010-08-31 Thread Jeff Zhang (JIRA)
[ https://issues.apache.org/jira/browse/PIG-794?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Zhang updated PIG-794: --- Attachment: AvroStorage_2.patch > Use Avro serialization in Pig > - > >