[jira] [Updated] (PIG-1949) e2e test harness should use bin/pig rather than calling java directly

[Alan Gates (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PIG-1949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alan Gates updated PIG-1949:


Attachment: PIG-1949.patch

Fixes to TestDriverPig to make it work with bin/pig instead of using Java 
directly.  I also yanked the code out of TestDriverPig that mimicked the 
runTests function in TestDriver.

> e2e test harness should use bin/pig rather than calling java directly
> -
>
> Key: PIG-1949
> URL: https://issues.apache.org/jira/browse/PIG-1949
> Project: Pig
>  Issue Type: Bug
>  Components: tools
>Reporter: Alan Gates
>Assignee: Alan Gates
>Priority: Minor
> Fix For: 0.10
>
> Attachments: PIG-1949.patch
>
>
> Currently TestDriverPig.pm uses java directly to invoke Pig.  It should use 
> the bash shell script bin/pig instead.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (PIG-1949) e2e test harness should use bin/pig rather than calling java directly

[Alan Gates (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PIG-1949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alan Gates updated PIG-1949:


Fix Version/s: 0.10
   Status: Patch Available  (was: Open)

> e2e test harness should use bin/pig rather than calling java directly
> -
>
> Key: PIG-1949
> URL: https://issues.apache.org/jira/browse/PIG-1949
> Project: Pig
>  Issue Type: Bug
>  Components: tools
>Reporter: Alan Gates
>Assignee: Alan Gates
>Priority: Minor
> Fix For: 0.10
>
> Attachments: PIG-1949.patch
>
>
> Currently TestDriverPig.pm uses java directly to invoke Pig.  It should use 
> the bash shell script bin/pig instead.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (PIG-2016) -dot option does not work with explain and new logical plan

[Daniel Dai (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PIG-2016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Dai updated PIG-2016:


Attachment: PIG-2016-1.patch

> -dot option does not work with explain and new logical plan
> ---
>
> Key: PIG-2016
> URL: https://issues.apache.org/jira/browse/PIG-2016
> Project: Pig
>  Issue Type: Bug
>  Components: impl
>Affects Versions: 0.8.0, 0.8.1, 0.9.0
>Reporter: Alan Gates
>Assignee: Daniel Dai
>Priority: Minor
> Fix For: 0.9.0
>
> Attachments: PIG-2016-1.patch
>
>
> If you specify -dot in explain, it is supposed to produce a file with the 
> graphs in .dot format.  While the physical plan and map reduce plan are 
> correctly output in .dot format, the new logical plan is still output in text 
> format.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

Jenkins build is back to normal : Pig-trunk-commit #772

[Apache Jenkins Server]

See 

[jira] [Commented] (PIG-1314) Add DateTime Support to Pig

[Jeremy Hanna (JIRA)]

[ 
https://issues.apache.org/jira/browse/PIG-1314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13027250#comment-13027250
 ] 

Jeremy Hanna commented on PIG-1314:
---

I think this would be nice also when outputting from pig scripts using 
DBStorage to an RDBMS - to be able to serialize properly to the db's timestamp 
or date type (without extra UDF work).

> Add DateTime Support to Pig
> ---
>
> Key: PIG-1314
> URL: https://issues.apache.org/jira/browse/PIG-1314
> Project: Pig
>  Issue Type: Bug
>  Components: data
>Affects Versions: 0.7.0
>Reporter: Russell Jurney
>Assignee: Russell Jurney
>   Original Estimate: 672h
>  Remaining Estimate: 672h
>
> Hadoop/Pig are primarily used to parse log data, and most logs have a 
> timestamp component.  Therefore Pig should support dates as a primitive.
> Can someone familiar with adding types to pig comment on how hard this is?  
> We're looking at doing this, rather than use UDFs.  Is this a patch that 
> would be accepted?

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Resolved] (PIG-2018) NPE for co-group with group-by column having complex schema and different load functions for each input

[Thejas M Nair (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PIG-2018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thejas M Nair resolved PIG-2018.


Resolution: Fixed

Patch committed to 0.9 branch and trunk.

> NPE for co-group with group-by column having complex schema and different 
> load functions for each input
> ---
>
> Key: PIG-2018
> URL: https://issues.apache.org/jira/browse/PIG-2018
> Project: Pig
>  Issue Type: Bug
>Reporter: Thejas M Nair
>Assignee: Thejas M Nair
> Fix For: 0.9.0
>
> Attachments: PIG-2018.1.patch
>
>
> {code}
> l1 = load 'x' using PigStorage(':') as (a : (i : int),b,c);
> l2 = load 'x' as (a,b,c);
> cg = cogroup l1 by a, l2 by a;
> explain cg;
> Gives -
> ERROR 1067: Unable to explain alias cg
> org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1067: Unable to 
> explain alias cg
> at org.apache.pig.PigServer.explain(PigServer.java:1075)
> at 
> org.apache.pig.tools.grunt.GruntParser.explainCurrentBatch(GruntParser.java:381)
> at 
> org.apache.pig.tools.grunt.GruntParser.processExplain(GruntParser.java:313)
> at 
> org.apache.pig.tools.grunt.GruntParser.processExplain(GruntParser.java:276)
> at 
> org.apache.pig.tools.pigscript.parser.PigScriptParser.Explain(PigScriptParser.java:665)
> at 
> org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:325)
> at 
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:176)
> at 
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:152)
> at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:90)
> at org.apache.pig.Main.run(Main.java:554)
> at org.apache.pig.Main.main(Main.java:109)
> Caused by: java.lang.NullPointerException
> at 
> org.apache.pig.newplan.logical.visitor.LineageFindRelVisitor.mapMatchLoadFuncToUid(LineageFindRelVisitor.java:528)
> at 
> org.apache.pig.newplan.logical.visitor.LineageFindRelVisitor.visit(LineageFindRelVisitor.java:287)
> at 
> org.apache.pig.newplan.logical.relational.LOCogroup.accept(LOCogroup.java:235)
> at 
> org.apache.pig.newplan.DependencyOrderWalker.walk(DependencyOrderWalker.java:75)
> at org.apache.pig.newplan.PlanVisitor.visit(PlanVisitor.java:50)
> at 
> org.apache.pig.newplan.logical.visitor.CastLineageSetter.(CastLineageSetter.java:57)
> at org.apache.pig.PigServer$Graph.compile(PigServer.java:1683)
> at org.apache.pig.PigServer$Graph.compile(PigServer.java:1659)
> at org.apache.pig.PigServer$Graph.access$200(PigServer.java:1389)
> at org.apache.pig.PigServer.buildStorePlan(PigServer.java:1277)
> at org.apache.pig.PigServer.explain(PigServer.java:1038)
> ... 10 more
> {code}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (PIG-2015) Explain writes out logical plan twice

[Daniel Dai (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PIG-2015?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Dai updated PIG-2015:


  Resolution: Fixed
Assignee: Alan Gates
Hadoop Flags: [Reviewed]
  Status: Resolved  (was: Patch Available)

Commit this patch since PIG-2016 need it. Committed to both trunk and 0.9 
branch.

> Explain writes out logical plan twice
> -
>
> Key: PIG-2015
> URL: https://issues.apache.org/jira/browse/PIG-2015
> Project: Pig
>  Issue Type: Bug
>  Components: impl
>Affects Versions: 0.9.0
>Reporter: Alan Gates
>Assignee: Alan Gates
>Priority: Minor
> Fix For: 0.9.0
>
> Attachments: PIG-2015.patch
>
>
> Running explain on a script writes out the logical plan twice, the physical 
> plan once, and the map reduce plan once.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

Build failed in Jenkins: Pig-trunk #996

[Apache Jenkins Server]

See 

Changes:

[daijy] PIG-2017: consumeMap() fails with EmptyStackException (thedatachef via 
daijy)

--
[...truncated 34044 lines...]
[junit] (10:LOLoad={name: chararray,details: (age: bytearray,gpa: 
bytearray),field3: (a: bytearray,b: bytearray)}==>15)
[junit] (15:LOForEach={name: chararray,details: (age: bytearray,gpa: 
bytearray),field3: (a: bytearray,b: bytearray)}==>TERMINAL)
[junit] 
[junit] 
[junit] Checking DONE!
[junit] testSUM1
[junit] 11/04/29 22:31:53 INFO executionengine.HExecutionEngine: Connecting 
to hadoop file system at: file:///
[junit] 11/04/29 22:31:53 INFO executionengine.HExecutionEngine: Connecting 
to hadoop file system at: file:///
[junit] 11/04/29 22:31:53 INFO executionengine.HExecutionEngine: Connecting 
to hadoop file system at: file:///
[junit] testSUM2
[junit] 11/04/29 22:31:53 INFO executionengine.HExecutionEngine: Connecting 
to hadoop file system at: file:///
[junit] 11/04/29 22:31:53 INFO executionengine.HExecutionEngine: Connecting 
to hadoop file system at: file:///
[junit] Actual plan after type check:
[junit] *Type Graph***
[junit] (0:LOLoad={null}==>3)
[junit] (3:LOCogroup={group: (bytearray,bytearray),0: {null}}==>13)
[junit] 
[junit] (1:LOProject=bytearray==>TERMINAL)
[junit] 
[junit] (2:LOProject=bytearray==>TERMINAL)
[junit] (13:LOForEach={bytearray,bytearray,double}==>TERMINAL)
[junit] 
[junit] (9:LOProject=(bytearray,bytearray)==>TERMINAL)
[junit] 
[junit] (11:LOProject=(null)==>12)
[junit] (12:LOProject=(bytearray)==>10)
[junit] (10:LOUserFunc=double==>TERMINAL)
[junit] 
[junit] testGenerate1
[junit] 11/04/29 22:31:53 INFO executionengine.HExecutionEngine: Connecting 
to hadoop file system at: file:///
[junit] 11/04/29 22:31:53 INFO executionengine.HExecutionEngine: Connecting 
to hadoop file system at: file:///
[junit] *MessageCollector dump***
[junit] Warning:int is implicitly cast to float under LOAdd Operator
[junit] Warning:long is implicitly cast to float under LOAdd Operator
[junit] Warning:int is implicitly cast to double under LOAdd Operator
[junit] Actual plan after type check:
[junit] *Type Graph***
[junit] (0:LOLoad={name: chararray,age: int,gpa: double}==>18)
[junit] (18:LOForEach={float,double}==>TERMINAL)
[junit] 
[junit] (12:LOConst=int==>19)
[junit] (19:LOCast=float==>10)
[junit] (13:LOConst=float==>10)
[junit] (10:LOAdd=float==>14)
[junit] (11:LOConst=long==>20)
[junit] (20:LOCast=float==>14)
[junit] (14:LOAdd=float==>TERMINAL)
[junit] 
[junit] (17:LOProject=double==>16)
[junit] (15:LOConst=int==>21)
[junit] (21:LOCast=double==>16)
[junit] (16:LOAdd=double==>TERMINAL)
[junit] 
[junit] Tests run: 8, Failures: 0, Errors: 0, Time elapsed: 0.8 sec
[junit] Running org.apache.pig.test.TestTypeCheckingValidatorNoSchema
[junit] testUnion1
[junit] 11/04/29 22:31:54 WARN conf.Configuration: DEPRECATED: 
hadoop-site.xml found in the classpath. Usage of hadoop-site.xml is deprecated. 
Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to override 
properties of core-default.xml, mapred-default.xml and hdfs-default.xml 
respectively
[junit] *Type Graph***
[junit] (0:LOLoad={null}==>2)
[junit] (1:LOLoad={null}==>2)
[junit] (2:LOUnion={null}==>TERMINAL)
[junit] 
[junit] testUnion2
[junit] *Type Graph***
[junit] (3:LOLoad={a: int,b: long,c: bytearray}==>5)
[junit] (4:LOLoad={null}==>5)
[junit] (5:LOUnion={null}==>TERMINAL)
[junit] 
[junit] testSplitWithInnerPlan1
[junit] *MessageCollector dump***
[junit] Warning:bytearray is implicitly cast to int under LOLesserThanEqual 
Operator
[junit] *Type Graph***
[junit] (6:LOLoad={null}==>13)
[junit] (13:LOSplit={null}==>14,15)
[junit] (7:LOProject=bytearray==>9)
[junit] (8:LOProject=bytearray==>9)
[junit] (9:LONotEqual=boolean==>TERMINAL)
[junit] (10:LOProject=bytearray==>16)
[junit] (16:LOCast=int==>12)
[junit] (11:LOConst=int==>12)
[junit] (12:LOLesserThanEqual=boolean==>TERMINAL)
[junit] 
[junit] testSplitWithInnerPlan2
[junit] *MessageCollector dump***
[junit] Warning:bytearray is implicitly cast to int under LOAdd Operator
[junit] Error:In alias null, incompatible types in Subtract Operator left 
hand side:int right hand side:chararray
[junit] *Type Graph***
[junit] (17:LOLoad={null}==>26)
[junit] (26:LOSplit={null}==>27,28)
[junit] (18:LOProject=bytearray==>20)
[junit] (19:LOProject=bytearray==>20)
[junit] (20:LONotEqual=boolean==>TERMINAL)
[junit] (21:LOProject=bytearray

Build failed in Jenkins: Pig-trunk-commit #771

[Apache Jenkins Server]

See 

Changes:

[daijy] PIG-2017: consumeMap() fails with EmptyStackException (thedatachef via 
daijy)

--
[...truncated 38650 lines...]
[junit] (10:LOLoad={name: chararray,details: (age: bytearray,gpa: 
bytearray),field3: (a: bytearray,b: bytearray)}==>15)
[junit] (15:LOForEach={name: chararray,details: (age: bytearray,gpa: 
bytearray),field3: (a: bytearray,b: bytearray)}==>TERMINAL)
[junit] 
[junit] 
[junit] Checking DONE!
[junit] testSUM1
[junit] 11/04/29 22:30:42 INFO executionengine.HExecutionEngine: Connecting 
to hadoop file system at: file:///
[junit] 11/04/29 22:30:42 INFO executionengine.HExecutionEngine: Connecting 
to hadoop file system at: file:///
[junit] 11/04/29 22:30:42 INFO executionengine.HExecutionEngine: Connecting 
to hadoop file system at: file:///
[junit] testSUM2
[junit] 11/04/29 22:30:42 INFO executionengine.HExecutionEngine: Connecting 
to hadoop file system at: file:///
[junit] 11/04/29 22:30:42 INFO executionengine.HExecutionEngine: Connecting 
to hadoop file system at: file:///
[junit] Actual plan after type check:
[junit] *Type Graph***
[junit] (0:LOLoad={null}==>3)
[junit] (3:LOCogroup={group: (bytearray,bytearray),0: {null}}==>13)
[junit] 
[junit] (1:LOProject=bytearray==>TERMINAL)
[junit] 
[junit] (2:LOProject=bytearray==>TERMINAL)
[junit] (13:LOForEach={bytearray,bytearray,double}==>TERMINAL)
[junit] 
[junit] (9:LOProject=(bytearray,bytearray)==>TERMINAL)
[junit] 
[junit] (11:LOProject=(null)==>12)
[junit] (12:LOProject=(bytearray)==>10)
[junit] (10:LOUserFunc=double==>TERMINAL)
[junit] 
[junit] testGenerate1
[junit] 11/04/29 22:30:42 INFO executionengine.HExecutionEngine: Connecting 
to hadoop file system at: file:///
[junit] 11/04/29 22:30:42 INFO executionengine.HExecutionEngine: Connecting 
to hadoop file system at: file:///
[junit] *MessageCollector dump***
[junit] Warning:int is implicitly cast to float under LOAdd Operator
[junit] Warning:long is implicitly cast to float under LOAdd Operator
[junit] Warning:int is implicitly cast to double under LOAdd Operator
[junit] Actual plan after type check:
[junit] *Type Graph***
[junit] (0:LOLoad={name: chararray,age: int,gpa: double}==>18)
[junit] (18:LOForEach={float,double}==>TERMINAL)
[junit] 
[junit] (12:LOConst=int==>19)
[junit] (19:LOCast=float==>10)
[junit] (13:LOConst=float==>10)
[junit] (10:LOAdd=float==>14)
[junit] (11:LOConst=long==>20)
[junit] (20:LOCast=float==>14)
[junit] (14:LOAdd=float==>TERMINAL)
[junit] 
[junit] (17:LOProject=double==>16)
[junit] (15:LOConst=int==>21)
[junit] (21:LOCast=double==>16)
[junit] (16:LOAdd=double==>TERMINAL)
[junit] 
[junit] Tests run: 8, Failures: 0, Errors: 0, Time elapsed: 0.81 sec
[junit] Running org.apache.pig.test.TestTypeCheckingValidatorNoSchema
[junit] testUnion1
[junit] 11/04/29 22:30:43 WARN conf.Configuration: DEPRECATED: 
hadoop-site.xml found in the classpath. Usage of hadoop-site.xml is deprecated. 
Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to override 
properties of core-default.xml, mapred-default.xml and hdfs-default.xml 
respectively
[junit] *Type Graph***
[junit] (0:LOLoad={null}==>2)
[junit] (1:LOLoad={null}==>2)
[junit] (2:LOUnion={null}==>TERMINAL)
[junit] 
[junit] testUnion2
[junit] *Type Graph***
[junit] (3:LOLoad={a: int,b: long,c: bytearray}==>5)
[junit] (4:LOLoad={null}==>5)
[junit] (5:LOUnion={null}==>TERMINAL)
[junit] 
[junit] testSplitWithInnerPlan1
[junit] *MessageCollector dump***
[junit] Warning:bytearray is implicitly cast to int under LOLesserThanEqual 
Operator
[junit] *Type Graph***
[junit] (6:LOLoad={null}==>13)
[junit] (13:LOSplit={null}==>14,15)
[junit] (7:LOProject=bytearray==>9)
[junit] (8:LOProject=bytearray==>9)
[junit] (9:LONotEqual=boolean==>TERMINAL)
[junit] (10:LOProject=bytearray==>16)
[junit] (16:LOCast=int==>12)
[junit] (11:LOConst=int==>12)
[junit] (12:LOLesserThanEqual=boolean==>TERMINAL)
[junit] 
[junit] testSplitWithInnerPlan2
[junit] *MessageCollector dump***
[junit] Warning:bytearray is implicitly cast to int under LOAdd Operator
[junit] Error:In alias null, incompatible types in Subtract Operator left 
hand side:int right hand side:chararray
[junit] *Type Graph***
[junit] (17:LOLoad={null}==>26)
[junit] (26:LOSplit={null}==>27,28)
[junit] (18:LOProject=bytearray==>20)
[junit] (19:LOProject=bytearray==>20)
[junit] (20:LONotEqual=boolean==>TERMINAL)
[junit] (21:LOProject=b

[jira] [Updated] (PIG-1998) Allow macro to return void

[Richard Ding (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PIG-1998?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Richard Ding updated PIG-1998:
--

Attachment: PIG-1998_3.patch

The purpose of this validation is to give user an early warning when an alias 
in the returns clause doesn't appear in the macro as $. It performs a 
simple parsing using StreamTokenizer.

> Allow macro to return void
> --
>
> Key: PIG-1998
> URL: https://issues.apache.org/jira/browse/PIG-1998
> Project: Pig
>  Issue Type: Improvement
>  Components: impl
>Affects Versions: 0.9.0
>Reporter: Richard Ding
>Assignee: Richard Ding
> Fix For: 0.9.0
>
> Attachments: PIG-1998_1.patch, PIG-1998_2.patch, PIG-1998_3.patch
>
>
> Pig macro is allowed to not have output alias. But this property isn't clear 
> from macro definition and macro invocation (macro inline). Here we propose to 
> make it clear:
> 1. If a macro doesn't output any alias, it must specify void as return value. 
> For example:
> {code}  
> define mymacro(...) returns void {
>... ...
> };
> {code}
> 2. If a macro doesn't output any alias, it must be invoked without return 
> value. For example, to invoke above macro, just specify:
> {code}
> mymacro(...);
> {code}
> 3. Any non-void return alias in the macro definition must exist in the macro 
> body and be prefixed with $. For example:
> {code}  
> define mymacro(...) returns B {
>... ...
>$B = filter ...;
> };
> {code}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (PIG-1998) Allow macro to return void

[Richard Ding (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PIG-1998?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Richard Ding updated PIG-1998:
--

Status: Patch Available  (was: Open)

> Allow macro to return void
> --
>
> Key: PIG-1998
> URL: https://issues.apache.org/jira/browse/PIG-1998
> Project: Pig
>  Issue Type: Improvement
>  Components: impl
>Affects Versions: 0.9.0
>Reporter: Richard Ding
>Assignee: Richard Ding
> Fix For: 0.9.0
>
> Attachments: PIG-1998_1.patch, PIG-1998_2.patch, PIG-1998_3.patch
>
>
> Pig macro is allowed to not have output alias. But this property isn't clear 
> from macro definition and macro invocation (macro inline). Here we propose to 
> make it clear:
> 1. If a macro doesn't output any alias, it must specify void as return value. 
> For example:
> {code}  
> define mymacro(...) returns void {
>... ...
> };
> {code}
> 2. If a macro doesn't output any alias, it must be invoked without return 
> value. For example, to invoke above macro, just specify:
> {code}
> mymacro(...);
> {code}
> 3. Any non-void return alias in the macro definition must exist in the macro 
> body and be prefixed with $. For example:
> {code}  
> define mymacro(...) returns B {
>... ...
>$B = filter ...;
> };
> {code}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (PIG-2023) lineage tracking for casting should compare LoadCaster returned from LoadFunc instead of comparing the FuncSpec

[Thejas M Nair (JIRA)]

lineage tracking  for casting should compare LoadCaster returned from LoadFunc 
instead of comparing the FuncSpec


 Key: PIG-2023
 URL: https://issues.apache.org/jira/browse/PIG-2023
 Project: Pig
  Issue Type: Improvement
Reporter: Thejas M Nair
 Fix For: 0.10


When lineage of a column is tracked for the purpose of finding the LoadCaster 
associated with a column, and it finds that a column has two possible sources, 
it associates a LoadCaster (through a LoadFunc) only if the funcspec for 
LoadFunc in both cases are the same. But it is possible that the two LoadFunc 
with different func spec actually use the same LoadCaster (for example the 
default of Utf8StorageConverter). If the LoadFunc funcspec don't match, the 
LoadCaster returned by the LoadFunc should also be compred. If they are equal, 
this LoadCaster should be associated with the column . The LoadCaster 
implementation would need to override equals().

For example, in this case the columns in relation u use the same LoadCaster -
{code}
l1 = load 'x' using PigStorage(',') as (a,b);
l2 = load 'y' using PigStorage(':') as (a,b);
u = union l1,l2;
{code}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (PIG-2018) NPE for co-group with group-by column having complex schema and different load functions for each input

[Daniel Dai (JIRA)]

[ 
https://issues.apache.org/jira/browse/PIG-2018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13027199#comment-13027199
 ] 

Daniel Dai commented on PIG-2018:
-

+1

> NPE for co-group with group-by column having complex schema and different 
> load functions for each input
> ---
>
> Key: PIG-2018
> URL: https://issues.apache.org/jira/browse/PIG-2018
> Project: Pig
>  Issue Type: Bug
>Reporter: Thejas M Nair
>Assignee: Thejas M Nair
> Fix For: 0.9.0
>
> Attachments: PIG-2018.1.patch
>
>
> {code}
> l1 = load 'x' using PigStorage(':') as (a : (i : int),b,c);
> l2 = load 'x' as (a,b,c);
> cg = cogroup l1 by a, l2 by a;
> explain cg;
> Gives -
> ERROR 1067: Unable to explain alias cg
> org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1067: Unable to 
> explain alias cg
> at org.apache.pig.PigServer.explain(PigServer.java:1075)
> at 
> org.apache.pig.tools.grunt.GruntParser.explainCurrentBatch(GruntParser.java:381)
> at 
> org.apache.pig.tools.grunt.GruntParser.processExplain(GruntParser.java:313)
> at 
> org.apache.pig.tools.grunt.GruntParser.processExplain(GruntParser.java:276)
> at 
> org.apache.pig.tools.pigscript.parser.PigScriptParser.Explain(PigScriptParser.java:665)
> at 
> org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:325)
> at 
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:176)
> at 
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:152)
> at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:90)
> at org.apache.pig.Main.run(Main.java:554)
> at org.apache.pig.Main.main(Main.java:109)
> Caused by: java.lang.NullPointerException
> at 
> org.apache.pig.newplan.logical.visitor.LineageFindRelVisitor.mapMatchLoadFuncToUid(LineageFindRelVisitor.java:528)
> at 
> org.apache.pig.newplan.logical.visitor.LineageFindRelVisitor.visit(LineageFindRelVisitor.java:287)
> at 
> org.apache.pig.newplan.logical.relational.LOCogroup.accept(LOCogroup.java:235)
> at 
> org.apache.pig.newplan.DependencyOrderWalker.walk(DependencyOrderWalker.java:75)
> at org.apache.pig.newplan.PlanVisitor.visit(PlanVisitor.java:50)
> at 
> org.apache.pig.newplan.logical.visitor.CastLineageSetter.(CastLineageSetter.java:57)
> at org.apache.pig.PigServer$Graph.compile(PigServer.java:1683)
> at org.apache.pig.PigServer$Graph.compile(PigServer.java:1659)
> at org.apache.pig.PigServer$Graph.access$200(PigServer.java:1389)
> at org.apache.pig.PigServer.buildStorePlan(PigServer.java:1277)
> at org.apache.pig.PigServer.explain(PigServer.java:1038)
> ... 10 more
> {code}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (PIG-1990) support casting of complex types with empty inner schema to complex type with non-empty inner schema

[Thejas M Nair (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PIG-1990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thejas M Nair updated PIG-1990:
---

Attachment: PIG-1990.1.patch

> support casting of complex types with empty inner schema to complex type with 
> non-empty inner schema
> 
>
> Key: PIG-1990
> URL: https://issues.apache.org/jira/browse/PIG-1990
> Project: Pig
>  Issue Type: Bug
>Affects Versions: 0.9.0
>Reporter: Thejas M Nair
>Assignee: Thejas M Nair
> Fix For: 0.9.0
>
> Attachments: PIG-1990.1.patch
>
>
> Use case like the following should be supported - 
> {code}
> a = load '1.txt' as (t:tuple());
> b = foreach a generate (tuple(int))t;
> {code}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Resolved] (PIG-2017) consumeMap() fails with EmptyStackException

[Daniel Dai (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PIG-2017?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Dai resolved PIG-2017.
-

   Resolution: Fixed
Fix Version/s: 0.9.0
 Assignee: Jacob Perkins
 Hadoop Flags: [Reviewed]

Test patch pass:
 [exec] +1 overall.  
 [exec] 
 [exec] +1 @author.  The patch does not contain any @author tags.
 [exec] 
 [exec] +1 tests included.  The patch appears to include 3 new or 
modified tests.
 [exec] 
 [exec] +1 javadoc.  The javadoc tool did not generate any warning 
messages.
 [exec] 
 [exec] +1 javac.  The applied patch does not increase the total number 
of javac compiler warnings.
 [exec] 
 [exec] +1 findbugs.  The patch does not introduce any new Findbugs 
warnings.
 [exec] 
 [exec] +1 release audit.  The applied patch does not increase the 
total number of release audit warnings.

Unit test pass

Patch committed to both trunk and 0.9 branch. Thanks Jacob!

> consumeMap() fails with EmptyStackException
> ---
>
> Key: PIG-2017
> URL: https://issues.apache.org/jira/browse/PIG-2017
> Project: Pig
>  Issue Type: Bug
>Reporter: Jacob Perkins
>Assignee: Jacob Perkins
> Fix For: 0.9.0
>
> Attachments: utf8storagepatch.txt, utf8storagepatch_withtests.txt
>
>
> If a map is read in its serialized form, eg: [key#value], then the 
> consumeMap() method of Utf8StorageConverter fails for the following maps:
> {code:none}
> [a#)]
> [a#}]
> [a#"take a look at my lovely curly brace, }"]
> [a#'oh look, a closed parenthesis! )']
> {code}
> There are a couple of options:
> 1. Define an escape sequence (ie. quotes or a backslash)
> 2. Call it a bad record, go get a beer, and move on.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (PIG-1775) Removal of old logical plan

[Xuefu Zhang (JIRA)]

[ 
https://issues.apache.org/jira/browse/PIG-1775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13027184#comment-13027184
 ] 

Xuefu Zhang commented on PIG-1775:
--

Test-patch run:

 [exec] +1 overall.
 [exec]
 [exec] +1 @author.  The patch does not contain any @author tags.
 [exec]
 [exec] +1 tests included.  The patch appears to include 227 new or 
modified tests.
 [exec]
 [exec] +1 javadoc.  The javadoc tool did not generate any warning 
messages.
 [exec]
 [exec] +1 javac.  The applied patch does not increase the total number 
of javac compiler warnings.
 [exec]
 [exec] +1 findbugs.  The patch does not introduce any new Findbugs 
warnings.
 [exec]
 [exec] +1 release audit.  The applied patch does not increase the 
total number of release audit warnings.


> Removal of old logical plan
> ---
>
> Key: PIG-1775
> URL: https://issues.apache.org/jira/browse/PIG-1775
> Project: Pig
>  Issue Type: Improvement
>Affects Versions: 0.9.0
>Reporter: Yan Zhou
>Assignee: Xuefu Zhang
> Fix For: 0.9.0
>
> Attachments: PIG-1775.patch
>
>
> The new logical plan will only be used and the old logical plan will be 
> removed once the new one is stable enough. It is scheduled for the 0.9 
> release.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (PIG-2012) Comments at the begining of the file throws off line numbers in errors

[Richard Ding (JIRA)]

[ 
https://issues.apache.org/jira/browse/PIG-2012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13027174#comment-13027174
 ] 

Richard Ding commented on PIG-2012:
---

test-patch result:

{code}
 [exec] -1 overall.  
 [exec] 
 [exec] +1 @author.  The patch does not contain any @author tags.
 [exec] 
 [exec] +1 tests included.  The patch appears to include 3 new or 
modified tests.
 [exec] 
 [exec] +1 javadoc.  The javadoc tool did not generate any warning 
messages.
 [exec] 
 [exec] -1 javac.  The applied patch generated 964 javac compiler 
warnings (more than the trunk's current 963 warnings).
 [exec] 
 [exec] +1 findbugs.  The patch does not introduce any new Findbugs 
warnings.
 [exec] 
 [exec] +1 release audit.  The applied patch does not increase the 
total number of release audit warnings.
{code}

> Comments at the begining of the file throws off line numbers in errors
> --
>
> Key: PIG-2012
> URL: https://issues.apache.org/jira/browse/PIG-2012
> Project: Pig
>  Issue Type: Bug
>  Components: impl
>Affects Versions: 0.9.0
>Reporter: Alan Gates
>Assignee: Richard Ding
> Fix For: 0.9.0
>
> Attachments: PIG-2012_1.patch, macro.pig
>
>
> The preprocessor does not appear to be handling leading comments properly 
> when calculating line numbers for error messages.  In the attached script, 
> the error is reported to be on line 7.  It is actually on line 10.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (PIG-2012) Comments at the begining of the file throws off line numbers in errors

[Richard Ding (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PIG-2012?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Richard Ding updated PIG-2012:
--

Status: Patch Available  (was: Open)

> Comments at the begining of the file throws off line numbers in errors
> --
>
> Key: PIG-2012
> URL: https://issues.apache.org/jira/browse/PIG-2012
> Project: Pig
>  Issue Type: Bug
>  Components: impl
>Affects Versions: 0.9.0
>Reporter: Alan Gates
>Assignee: Richard Ding
> Fix For: 0.9.0
>
> Attachments: PIG-2012_1.patch, macro.pig
>
>
> The preprocessor does not appear to be handling leading comments properly 
> when calculating line numbers for error messages.  In the attached script, 
> the error is reported to be on line 7.  It is actually on line 10.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (PIG-1827) When passing a parameter to Pig, if the value contains $ it has to be escaped for no apparent reason

[Richard Ding (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PIG-1827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Richard Ding updated PIG-1827:
--

Status: Patch Available  (was: Open)

> When passing a parameter to Pig, if the value contains $ it has to be escaped 
> for no apparent reason
> 
>
> Key: PIG-1827
> URL: https://issues.apache.org/jira/browse/PIG-1827
> Project: Pig
>  Issue Type: Bug
>  Components: impl
>Affects Versions: 0.8.0
>Reporter: Julien Le Dem
>Assignee: Richard Ding
> Fix For: 0.9.0
>
> Attachments: PIG-1827-1.patch, PIG-1827_2.patch
>
>


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (PIG-2022) Filter expression plan not able to refer to input tuple fields

[Xuefu Zhang (JIRA)]

Filter expression plan not able to refer to input tuple fields
--

 Key: PIG-2022
 URL: https://issues.apache.org/jira/browse/PIG-2022
 Project: Pig
  Issue Type: Improvement
Reporter: Xuefu Zhang
Assignee: Xuefu Zhang


For the following query:

A = load 'x' as ( a:bag{T:tuple(u:int,v:chararray)}, b:int );
B = foreach A { T = filter a by u > b; generate T; };
Store B into 'y';

Pig currently gives the following error:

2011-04-29 12:43:23,979 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 
1025:  Invalid field projection. Projected field [b] does 
not exist in schema: u:int,v:chararray.

However, this is a reasonable usage. Without this support, user has to flatten 
the bag, do filtering, and then re-bag. Therefore, it's very useful to allow 
user to refer fields from the input relation in the nested plan.


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (PIG-2020) Valid query fails to validate

[Xuefu Zhang (JIRA)]

[ 
https://issues.apache.org/jira/browse/PIG-2020?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13027158#comment-13027158
 ] 

Xuefu Zhang commented on PIG-2020:
--

After some discussion, it's found that the real problem is that Pig didn't 
automatically cast $0 to a bag type, which it should. $1 and $2 are referring 
to fields in the bag.

This behavior is inconsistent with other types because we insert type casting 
for other types.

The error message is confusing too, because the bag's schema is unknown rather 
than of size 1.

> Valid query fails to validate
> -
>
> Key: PIG-2020
> URL: https://issues.apache.org/jira/browse/PIG-2020
> Project: Pig
>  Issue Type: Bug
>Affects Versions: 0.8.0, 0.9.0
>Reporter: Xuefu Zhang
>Assignee: Xuefu Zhang
> Fix For: 0.9.0
>
>
> The following query seems valid:
> A = load 'x';
> B = foreach A { T = filter $0 by $1 > $2; generate T; };
> Store B into 'y';
> However, the query fails due to validation error in 0.8:
> 2011-04-28 09:08:06,846 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 
> 1026: Attempt to fetch field 1 from schema of size 1
> Similar error is given in 0.9.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (PIG-1775) Removal of old logical plan

[Xuefu Zhang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PIG-1775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xuefu Zhang updated PIG-1775:
-

Attachment: PIG-1775.patch

First patch, migrating old test case to use new logical plan. Some obsolete 
test cases are removed.

> Removal of old logical plan
> ---
>
> Key: PIG-1775
> URL: https://issues.apache.org/jira/browse/PIG-1775
> Project: Pig
>  Issue Type: Improvement
>Affects Versions: 0.9.0
>Reporter: Yan Zhou
>Assignee: Xuefu Zhang
> Fix For: 0.9.0
>
> Attachments: PIG-1775.patch
>
>
> The new logical plan will only be used and the old logical plan will be 
> removed once the new one is stable enough. It is scheduled for the 0.9 
> release.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Assigned] (PIG-2016) -dot option does not work with explain and new logical plan

[Daniel Dai (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PIG-2016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Dai reassigned PIG-2016:
---

Assignee: Daniel Dai

> -dot option does not work with explain and new logical plan
> ---
>
> Key: PIG-2016
> URL: https://issues.apache.org/jira/browse/PIG-2016
> Project: Pig
>  Issue Type: Bug
>  Components: impl
>Affects Versions: 0.8.0, 0.8.1, 0.9.0
>Reporter: Alan Gates
>Assignee: Daniel Dai
>Priority: Minor
> Fix For: 0.9.0
>
>
> If you specify -dot in explain, it is supposed to produce a file with the 
> graphs in .dot format.  While the physical plan and map reduce plan are 
> correctly output in .dot format, the new logical plan is still output in text 
> format.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (PIG-2018) NPE for co-group with group-by column having complex schema and different load functions for each input

[Thejas M Nair (JIRA)]

[ 
https://issues.apache.org/jira/browse/PIG-2018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13027021#comment-13027021
 ] 

Thejas M Nair commented on PIG-2018:


unit tests and test-patch  passed for PIG-2018.1.patch .


> NPE for co-group with group-by column having complex schema and different 
> load functions for each input
> ---
>
> Key: PIG-2018
> URL: https://issues.apache.org/jira/browse/PIG-2018
> Project: Pig
>  Issue Type: Bug
>Reporter: Thejas M Nair
>Assignee: Thejas M Nair
> Fix For: 0.9.0
>
> Attachments: PIG-2018.1.patch
>
>
> {code}
> l1 = load 'x' using PigStorage(':') as (a : (i : int),b,c);
> l2 = load 'x' as (a,b,c);
> cg = cogroup l1 by a, l2 by a;
> explain cg;
> Gives -
> ERROR 1067: Unable to explain alias cg
> org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1067: Unable to 
> explain alias cg
> at org.apache.pig.PigServer.explain(PigServer.java:1075)
> at 
> org.apache.pig.tools.grunt.GruntParser.explainCurrentBatch(GruntParser.java:381)
> at 
> org.apache.pig.tools.grunt.GruntParser.processExplain(GruntParser.java:313)
> at 
> org.apache.pig.tools.grunt.GruntParser.processExplain(GruntParser.java:276)
> at 
> org.apache.pig.tools.pigscript.parser.PigScriptParser.Explain(PigScriptParser.java:665)
> at 
> org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:325)
> at 
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:176)
> at 
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:152)
> at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:90)
> at org.apache.pig.Main.run(Main.java:554)
> at org.apache.pig.Main.main(Main.java:109)
> Caused by: java.lang.NullPointerException
> at 
> org.apache.pig.newplan.logical.visitor.LineageFindRelVisitor.mapMatchLoadFuncToUid(LineageFindRelVisitor.java:528)
> at 
> org.apache.pig.newplan.logical.visitor.LineageFindRelVisitor.visit(LineageFindRelVisitor.java:287)
> at 
> org.apache.pig.newplan.logical.relational.LOCogroup.accept(LOCogroup.java:235)
> at 
> org.apache.pig.newplan.DependencyOrderWalker.walk(DependencyOrderWalker.java:75)
> at org.apache.pig.newplan.PlanVisitor.visit(PlanVisitor.java:50)
> at 
> org.apache.pig.newplan.logical.visitor.CastLineageSetter.(CastLineageSetter.java:57)
> at org.apache.pig.PigServer$Graph.compile(PigServer.java:1683)
> at org.apache.pig.PigServer$Graph.compile(PigServer.java:1659)
> at org.apache.pig.PigServer$Graph.access$200(PigServer.java:1389)
> at org.apache.pig.PigServer.buildStorePlan(PigServer.java:1277)
> at org.apache.pig.PigServer.explain(PigServer.java:1038)
> ... 10 more
> {code}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira