[jira] [Updated] (PIG-2927) SHIP and use JRuby gems in JRuby UDFs

[Russell Jurney (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PIG-2927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Russell Jurney updated PIG-2927:


Fix Version/s: 0.11

> SHIP and use JRuby gems in JRuby UDFs
> -
>
> Key: PIG-2927
> URL: https://issues.apache.org/jira/browse/PIG-2927
> Project: Pig
>  Issue Type: New Feature
>  Components: parser
>Affects Versions: 0.11
> Environment: JRuby UDFs
>Reporter: Russell Jurney
>Assignee: Jonathan Coveney
>Priority: Minor
> Fix For: 0.11
>
> Attachments: PIG-2927-0.patch, PIG-2927-1.patch, PIG-2927-2.patch, 
> PIG-2927-3.patch
>
>
> It would be great to use JRuby gems in JRuby UDFs without installing them on 
> all machines on the cluster. Some way to SHIP them automatically with the job 
> would be great.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (PIG-2927) SHIP and use JRuby gems in JRuby UDFs

[Russell Jurney (JIRA)]

[ 
https://issues.apache.org/jira/browse/PIG-2927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13476644#comment-13476644
 ] 

Russell Jurney commented on PIG-2927:
-

It builds! :) Ok now I test.

> SHIP and use JRuby gems in JRuby UDFs
> -
>
> Key: PIG-2927
> URL: https://issues.apache.org/jira/browse/PIG-2927
> Project: Pig
>  Issue Type: New Feature
>  Components: parser
>Affects Versions: 0.11
> Environment: JRuby UDFs
>Reporter: Russell Jurney
>Assignee: Jonathan Coveney
>Priority: Minor
> Attachments: PIG-2927-0.patch, PIG-2927-1.patch, PIG-2927-2.patch, 
> PIG-2927-3.patch
>
>
> It would be great to use JRuby gems in JRuby UDFs without installing them on 
> all machines on the cluster. Some way to SHIP them automatically with the job 
> would be great.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (PIG-2927) SHIP and use JRuby gems in JRuby UDFs

[Jonathan Coveney (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PIG-2927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jonathan Coveney updated PIG-2927:
--

Attachment: PIG-2927-3.patch

Wasn't maven. Was me overlooking the file in the git branch. Woops. Thanks for 
your patience, Russell.

> SHIP and use JRuby gems in JRuby UDFs
> -
>
> Key: PIG-2927
> URL: https://issues.apache.org/jira/browse/PIG-2927
> Project: Pig
>  Issue Type: New Feature
>  Components: parser
>Affects Versions: 0.11
> Environment: JRuby UDFs
>Reporter: Russell Jurney
>Assignee: Jonathan Coveney
>Priority: Minor
> Attachments: PIG-2927-0.patch, PIG-2927-1.patch, PIG-2927-2.patch, 
> PIG-2927-3.patch
>
>
> It would be great to use JRuby gems in JRuby UDFs without installing them on 
> all machines on the cluster. Some way to SHIP them automatically with the job 
> would be great.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (PIG-2927) SHIP and use JRuby gems in JRuby UDFs

[Russell Jurney (JIRA)]

[ 
https://issues.apache.org/jira/browse/PIG-2927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13476623#comment-13476623
 ] 

Russell Jurney commented on PIG-2927:
-

Maven is on drugs. I try with stuff I had, no work. I pull from trunk, apply 
again, no work. Why can't it see the tar package?

compile-sources:
[javac] /Users/rjurney/Software/pig-trunk/build.xml:504: warning: 
'includeantruntime' was not set, defaulting to build.sysclasspath=last; set to 
false for repeatable builds
[javac] Compiling 16 source files to 
/Users/rjurney/Software/pig-trunk/build/classes
[javac] 
/Users/rjurney/Software/pig-trunk/src/org/apache/pig/scripting/jruby/JrubyScriptEngine.java:51:
 package org.apache.pig.tar does not exist
[javac] import org.apache.pig.tar.TarUtils;
[javac]  ^
[javac] 
/Users/rjurney/Software/pig-trunk/src/org/apache/pig/scripting/jruby/JrubyScriptEngine.java:170:
 cannot find symbol
[javac] symbol  : variable TarUtils
[javac] location: class org.apache.pig.scripting.jruby.JrubyScriptEngine
[javac] TarUtils.tarFile(GEM_DIR_BASE_NAME, f.getParentFile(), 
new File(f, "lib"), os);
[javac] ^
[javac] Note: 
/Users/rjurney/Software/pig-trunk/src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/JobControlCompiler.java
 uses or overrides a deprecated API.
[javac] Note: Recompile with -Xlint:deprecation for details.
[javac] 2 errors

BUILD FAILED
/Users/rjurney/Software/pig-trunk/build.xml:451: The following error occurred 
while executing this line:
/Users/rjurney/Software/pig-trunk/build.xml:504: Compile failed; see the 
compiler error output for details.

Total time: 14 seconds



> SHIP and use JRuby gems in JRuby UDFs
> -
>
> Key: PIG-2927
> URL: https://issues.apache.org/jira/browse/PIG-2927
> Project: Pig
>  Issue Type: New Feature
>  Components: parser
>Affects Versions: 0.11
> Environment: JRuby UDFs
>Reporter: Russell Jurney
>Assignee: Jonathan Coveney
>Priority: Minor
> Attachments: PIG-2927-0.patch, PIG-2927-1.patch, PIG-2927-2.patch
>
>
> It would be great to use JRuby gems in JRuby UDFs without installing them on 
> all machines on the cluster. Some way to SHIP them automatically with the job 
> would be great.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] Subscription: PIG patch available

[jira]

Issue Subscription
Filter: PIG patch available (36 issues)

Subscriber: pigdaily

Key Summary
PIG-2976Reduce HBaseStorage logging
https://issues.apache.org/jira/browse/PIG-2976
PIG-2968ColumnMapKeyPrune fails to prune a subtree inside foreach
https://issues.apache.org/jira/browse/PIG-2968
PIG-2966Test failures on CentOS 6 because MALLOC_ARENA_MAX is not set
https://issues.apache.org/jira/browse/PIG-2966
PIG-2960Increase the timeout for unit test
https://issues.apache.org/jira/browse/PIG-2960
PIG-2959Add a pig.cmd for Pig to run under Windows
https://issues.apache.org/jira/browse/PIG-2959
PIG-2958Pig tests do not appear to have a logger attached
https://issues.apache.org/jira/browse/PIG-2958
PIG-2957TetsScriptUDF fail due to volume prefix in jar
https://issues.apache.org/jira/browse/PIG-2957
PIG-2956Invalid cache specification for some streaming statement
https://issues.apache.org/jira/browse/PIG-2956
PIG-2955 Fix bunch of Pig e2e tests on Windows 
https://issues.apache.org/jira/browse/PIG-2955
PIG-2954 TestParamSubPreproc still depends on "bash" to run 
https://issues.apache.org/jira/browse/PIG-2954
PIG-2953"which" utility does not exist on Windows
https://issues.apache.org/jira/browse/PIG-2953
PIG-2943DevTests, Refactor Windows checks to use new Util.WINDOWS method 
for code health
https://issues.apache.org/jira/browse/PIG-2943
PIG-2942DevTests, TestLoad has a false failure on Windows
https://issues.apache.org/jira/browse/PIG-2942
PIG-2940HBaseStorage store fails in secure cluster
https://issues.apache.org/jira/browse/PIG-2940
PIG-2931$ signs in the replacement string make parameter substitution fail
https://issues.apache.org/jira/browse/PIG-2931
PIG-2904Scripting UDFs should allow DEFINE statements to pass parameters to 
the UDF's constructor
https://issues.apache.org/jira/browse/PIG-2904
PIG-2898Parallel execution of e2e tests
https://issues.apache.org/jira/browse/PIG-2898
PIG-2881Add SUBTRACT eval function
https://issues.apache.org/jira/browse/PIG-2881
PIG-2873Converting bin/pig shell script to python
https://issues.apache.org/jira/browse/PIG-2873
PIG-2834MultiStorage requires unused constructor argument
https://issues.apache.org/jira/browse/PIG-2834
PIG-2824Pushing checking number of fields into LoadFunc
https://issues.apache.org/jira/browse/PIG-2824
PIG-2801grunt "sh" command should invoke the shell implicitly instead of 
calling exec directly with the command tokens
https://issues.apache.org/jira/browse/PIG-2801
PIG-2798pig streaming tests assume interpreters are auto-resolved
https://issues.apache.org/jira/browse/PIG-2798
PIG-2796Local temporary paths are not always valid HDFS path names.
https://issues.apache.org/jira/browse/PIG-2796
PIG-2795Fix test cases that generate pig scripts with "load " + pathStr to 
encode "\" in the path
https://issues.apache.org/jira/browse/PIG-2795
PIG-2778Add 'matches' operator to predicate pushdown
https://issues.apache.org/jira/browse/PIG-2778
PIG-2661Pig uses an extra job for loading data in Pigmix L9
https://issues.apache.org/jira/browse/PIG-2661
PIG-2657Print warning if using wrong jython version
https://issues.apache.org/jira/browse/PIG-2657
PIG-2495Using merge JOIN from a HBaseStorage produces an error
https://issues.apache.org/jira/browse/PIG-2495
PIG-2417Streaming UDFs -  allow users to easily write UDFs in scripting 
languages with no JVM implementation.
https://issues.apache.org/jira/browse/PIG-2417
PIG-2405svn tags/release-0.9.1: some unit test case failed with open JDK
https://issues.apache.org/jira/browse/PIG-2405
PIG-2362Rework Ant build.xml to use macrodef instead of antcall
https://issues.apache.org/jira/browse/PIG-2362
PIG-2312NPE when relation and column share the same name and used in Nested 
Foreach 
https://issues.apache.org/jira/browse/PIG-2312
PIG-1942script UDF (jython) should utilize the intended output schema to 
more directly convert Py objects to Pig objects
https://issues.apache.org/jira/browse/PIG-1942
PIG-1283COUNT on null bag causes failure
https://issues.apache.org/jira/browse/PIG-1283
PIG-1237Piggybank MutliStorage - specify field to write in output
https://issues.apache.org/jira/browse/PIG-1237

You may edit this subscription at:
https://issues.apache.org/jira/secure/FilterSubscription!default.jspa?subId=13225&filterId=12322384

[jira] [Assigned] (PIG-2965) RANDOM should allow seed initialization for ease of testing

[Dmitriy V. Ryaboy (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PIG-2965?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Dmitriy V. Ryaboy reassigned PIG-2965:
--

Assignee: Jonathan Coveney  (was: Dmitriy V. Ryaboy)

> RANDOM should allow seed initialization for ease of testing
> ---
>
> Key: PIG-2965
> URL: https://issues.apache.org/jira/browse/PIG-2965
> Project: Pig
>  Issue Type: Bug
>Reporter: Aneesh Sharma
>Assignee: Jonathan Coveney
>  Labels: newbie
> Attachments: PIG-2965-0.patch
>
>


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (PIG-2586) A better plan/data flow visualizer

[Aniket Mokashi (JIRA)]

[ 
https://issues.apache.org/jira/browse/PIG-2586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13476599#comment-13476599
 ] 

Aniket Mokashi commented on PIG-2586:
-

Do we have a patch for this?

> A better plan/data flow visualizer
> --
>
> Key: PIG-2586
> URL: https://issues.apache.org/jira/browse/PIG-2586
> Project: Pig
>  Issue Type: Improvement
>  Components: impl
>Reporter: Daniel Dai
>  Labels: gsoc2012
>
> Pig supports a dot graph style plan to visualize the 
> logical/physical/mapreduce plan (explain with -dot option, see 
> http://ofps.oreilly.com/titles/9781449302641/developing_and_testing.html). 
> However, dot graph takes extra step to generate the plan graph and the 
> quality of the output is not good. It's better we can implement a better 
> visualizer for Pig. It should:
> 1. show operator type and alias
> 2. turn on/off output schema
> 3. dive into foreach inner plan on demand
> 4. provide a way to show operator source code, eg, tooltip of an operator 
> (plan don't currently have this information, but you can assume this is in 
> place)
> 5. besides visualize logical/physical/mapreduce plan, visualize the script 
> itself is also useful
> 6. may rely on some java graphic library such as Swing
> This is a candidate project for Google summer of code 2012. More information 
> about the program can be found at 
> https://cwiki.apache.org/confluence/display/PIG/GSoc2012

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (PIG-2794) Pig test: add utils to simplify testing on Windows

[John Gordon (JIRA)]

[ 
https://issues.apache.org/jira/browse/PIG-2794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13476585#comment-13476585
 ] 

John Gordon commented on PIG-2794:
--

Thank you!

> Pig test: add utils to simplify testing on Windows
> --
>
> Key: PIG-2794
> URL: https://issues.apache.org/jira/browse/PIG-2794
> Project: Pig
>  Issue Type: Sub-task
>  Components: tools
>Affects Versions: 0.10.0
> Environment: Windows
>Reporter: John Gordon
>Assignee: John Gordon
> Fix For: 0.11, 0.10.1
>
> Attachments: 0007-Utils-to-support-windows-port.patch, 
> trunkUtilpatch.patch
>
>
> Add facilities to simplify testing on Windows for things like path management 
> and determining if the OS is Windows or not.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (PIG-2927) SHIP and use JRuby gems in JRuby UDFs

[Jonathan Coveney (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PIG-2927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jonathan Coveney updated PIG-2927:
--

Attachment: PIG-2927-2.patch

Russell, thanks for your help on this :) I think it was picking up the dep from 
my environment. I think it should pull it from maven now.

> SHIP and use JRuby gems in JRuby UDFs
> -
>
> Key: PIG-2927
> URL: https://issues.apache.org/jira/browse/PIG-2927
> Project: Pig
>  Issue Type: New Feature
>  Components: parser
>Affects Versions: 0.11
> Environment: JRuby UDFs
>Reporter: Russell Jurney
>Assignee: Jonathan Coveney
>Priority: Minor
> Attachments: PIG-2927-0.patch, PIG-2927-1.patch, PIG-2927-2.patch
>
>
> It would be great to use JRuby gems in JRuby UDFs without installing them on 
> all machines on the cluster. Some way to SHIP them automatically with the job 
> would be great.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (PIG-2976) Reduce HBaseStorage logging

[Bill Graham (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PIG-2976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bill Graham updated PIG-2976:
-

Status: Patch Available  (was: Open)

> Reduce HBaseStorage logging
> ---
>
> Key: PIG-2976
> URL: https://issues.apache.org/jira/browse/PIG-2976
> Project: Pig
>  Issue Type: Improvement
>Reporter: Aneesh Sharma
>Assignee: Bill Graham
> Attachments: PIG-2976_1.patch
>
>
> The current logging in HBaseStorage is very verbose, filling the log with 
> lines like these:
> 2012-10-05 00:59:01,158 INFO 
> org.apache.pig.backend.hadoop.hbase.HBaseStorage: Adding family:descriptor 
> filters with values extracted:user_logins
> It would be nice to reduce this verbosity!

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (PIG-2794) Pig test: add utils to simplify testing on Windows

[Alan Gates (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PIG-2794?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alan Gates updated PIG-2794:


   Resolution: Fixed
Fix Version/s: 0.10.1
   Status: Resolved  (was: Patch Available)

Patch checked into 0.10 branch, 0.11 branch, and trunk.

> Pig test: add utils to simplify testing on Windows
> --
>
> Key: PIG-2794
> URL: https://issues.apache.org/jira/browse/PIG-2794
> Project: Pig
>  Issue Type: Sub-task
>  Components: tools
>Affects Versions: 0.10.0
> Environment: Windows
>Reporter: John Gordon
>Assignee: John Gordon
> Fix For: 0.11, 0.10.1
>
> Attachments: 0007-Utils-to-support-windows-port.patch, 
> trunkUtilpatch.patch
>
>
> Add facilities to simplify testing on Windows for things like path management 
> and determining if the OS is Windows or not.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (PIG-2976) Reduce HBaseStorage logging

[Bill Graham (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PIG-2976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bill Graham updated PIG-2976:
-

Attachment: PIG-2976_1.patch

Attaching a patch. Since this is a trivial 6-line logging change that fixes an 
annoying issue, I'll commit to both trunk and pig 11 branch unless anyone 
objects.

> Reduce HBaseStorage logging
> ---
>
> Key: PIG-2976
> URL: https://issues.apache.org/jira/browse/PIG-2976
> Project: Pig
>  Issue Type: Improvement
>Reporter: Aneesh Sharma
>Assignee: Bill Graham
> Attachments: PIG-2976_1.patch
>
>
> The current logging in HBaseStorage is very verbose, filling the log with 
> lines like these:
> 2012-10-05 00:59:01,158 INFO 
> org.apache.pig.backend.hadoop.hbase.HBaseStorage: Adding family:descriptor 
> filters with values extracted:user_logins
> It would be nice to reduce this verbosity!

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Resolved] (PIG-1314) Add DateTime Support to Pig

[Thejas M Nair (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PIG-1314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thejas M Nair resolved PIG-1314.


Resolution: Fixed

As Dmitriy suggested, closing this jira and opened new ones for remaining work 
- PIG-2980, PIG-2981, PIG-2982 .


> Add DateTime Support to Pig
> ---
>
> Key: PIG-1314
> URL: https://issues.apache.org/jira/browse/PIG-1314
> Project: Pig
>  Issue Type: Bug
>  Components: data
>Affects Versions: 0.7.0
>Reporter: Russell Jurney
>Assignee: Zhijie Shen
>  Labels: gsoc2012
> Attachments: joda_vs_builtin.zip, PIG-1314-1.patch, PIG-1314-2.patch, 
> PIG-1314-3.patch, PIG-1314-4.patch, PIG-1314-5.patch, PIG-1314-6.patch, 
> PIG-1314-7.patch
>
>   Original Estimate: 672h
>  Remaining Estimate: 672h
>
> Hadoop/Pig are primarily used to parse log data, and most logs have a 
> timestamp component.  Therefore Pig should support dates as a primitive.
> Can someone familiar with adding types to pig comment on how hard this is?  
> We're looking at doing this, rather than use UDFs.  Is this a patch that 
> would be accepted?
> This is a candidate project for Google summer of code 2012. More information 
> about the program can be found at 
> https://cwiki.apache.org/confluence/display/PIG/GSoc2012

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (PIG-2982) add unit tests for DateTime type that test setting timezone

[Thejas M Nair (JIRA)]

Thejas M Nair created PIG-2982:
--

 Summary: add unit tests for DateTime type that test setting 
timezone
 Key: PIG-2982
 URL: https://issues.apache.org/jira/browse/PIG-2982
 Project: Pig
  Issue Type: Test
Reporter: Thejas M Nair
 Fix For: 0.11


The default timezone can be set for the new DateTime type. We need to add unit 
tests that test this functionality. 


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (PIG-2927) SHIP and use JRuby gems in JRuby UDFs

[Russell Jurney (JIRA)]

[ 
https://issues.apache.org/jira/browse/PIG-2927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13476559#comment-13476559
 ] 

Russell Jurney commented on PIG-2927:
-

No build :(

 [echo] *** Building Main Sources ***
 [echo] *** To compile with all warnings enabled, supply -Dall.warnings=1 
on command line ***
 [echo] *** If all.warnings property is supplied, 
compile-sources-all-warnings target will be executed ***
 [echo] *** Else, compile-sources (which only warns about deprecations) 
target will be executed ***

compile-sources:
[javac] /Users/rjurney/Software/pig-trunk/build.xml:503: warning: 
'includeantruntime' was not set, defaulting to build.sysclasspath=last; set to 
false for repeatable builds
[javac] Compiling 14 source files to 
/Users/rjurney/Software/pig-trunk/build/classes
[javac] 
/Users/rjurney/Software/pig-trunk/src/org/apache/pig/scripting/jruby/JrubyScriptEngine.java:51:
 package org.apache.pig.tar does not exist
[javac] import org.apache.pig.tar.TarUtils;
[javac]  ^
[javac] 
/Users/rjurney/Software/pig-trunk/src/org/apache/pig/scripting/jruby/JrubyScriptEngine.java:170:
 cannot find symbol
[javac] symbol  : variable TarUtils
[javac] location: class org.apache.pig.scripting.jruby.JrubyScriptEngine
[javac] TarUtils.tarFile(GEM_DIR_BASE_NAME, f.getParentFile(), 
new File(f, "lib"), os);
[javac] ^
[javac] Note: 
/Users/rjurney/Software/pig-trunk/src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/JobControlCompiler.java
 uses or overrides a deprecated API.
[javac] Note: Recompile with -Xlint:deprecation for details.
[javac] 2 errors

BUILD FAILED
/Users/rjurney/Software/pig-trunk/build.xml:451: The following error occurred 
while executing this line:
/Users/rjurney/Software/pig-trunk/build.xml:503: Compile failed; see the 
compiler error output for details.

Total time: 1 minute 0 seconds

> SHIP and use JRuby gems in JRuby UDFs
> -
>
> Key: PIG-2927
> URL: https://issues.apache.org/jira/browse/PIG-2927
> Project: Pig
>  Issue Type: New Feature
>  Components: parser
>Affects Versions: 0.11
> Environment: JRuby UDFs
>Reporter: Russell Jurney
>Assignee: Jonathan Coveney
>Priority: Minor
> Attachments: PIG-2927-0.patch, PIG-2927-1.patch
>
>
> It would be great to use JRuby gems in JRuby UDFs without installing them on 
> all machines on the cluster. Some way to SHIP them automatically with the job 
> would be great.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (PIG-2981) add e2e tests for DateTime data type

[Thejas M Nair (JIRA)]

Thejas M Nair created PIG-2981:
--

 Summary: add e2e tests for DateTime  data type
 Key: PIG-2981
 URL: https://issues.apache.org/jira/browse/PIG-2981
 Project: Pig
  Issue Type: Test
Reporter: Thejas M Nair
 Fix For: 0.11


e2e tests for DateTime datatype need to be added.


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (PIG-2980) documentation for DateTime datatype

[Thejas M Nair (JIRA)]

Thejas M Nair created PIG-2980:
--

 Summary: documentation for DateTime datatype
 Key: PIG-2980
 URL: https://issues.apache.org/jira/browse/PIG-2980
 Project: Pig
  Issue Type: Bug
  Components: documentation
Reporter: Thejas M Nair
 Fix For: 0.11


Documentation for new DateTime type needs to be added.


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Build failed in Jenkins: Pig-trunk #1336

[Apache Jenkins Server]

See 

Changes:

[dvryaboy] Add test/org/apache/pig/test/junit for PIG-2908

[dvryaboy] PIG-2928: Fix e2e test failures in trunk: FilterBoolean_23/24

[dvryaboy] PIG-2908: Fix unit tests to work with jdk7

[jcoveney] PIG-2971: Add new parameter to specify the streaming environment 
(jcoveney)

[jcoveney] PIG-2963: Illustrate command and POPackageLite (cheolsoo via 
jcoveney)

--
[...truncated 6962 lines...]
 [findbugs]   org.apache.hadoop.fs.FSDataInputStream
 [findbugs]   org.python.core.PyObject
 [findbugs]   jline.History
 [findbugs]   org.jruby.embed.internal.LocalContextProvider
 [findbugs]   org.apache.hadoop.io.BooleanWritable
 [findbugs]   org.apache.log4j.Logger
 [findbugs]   org.apache.hadoop.hbase.filter.FamilyFilter
 [findbugs]   org.codehaus.jackson.annotate.JsonPropertyOrder
 [findbugs]   groovy.lang.Tuple
 [findbugs]   org.antlr.runtime.IntStream
 [findbugs]   org.apache.hadoop.util.ReflectionUtils
 [findbugs]   org.apache.hadoop.fs.ContentSummary
 [findbugs]   org.jruby.runtime.builtin.IRubyObject
 [findbugs]   org.jruby.RubyInteger
 [findbugs]   org.python.core.PyTuple
 [findbugs]   org.mortbay.log.Log
 [findbugs]   org.apache.hadoop.conf.Configuration
 [findbugs]   com.google.common.base.Joiner
 [findbugs]   org.apache.hadoop.mapreduce.lib.input.FileSplit
 [findbugs]   org.apache.hadoop.mapred.Counters$Counter
 [findbugs]   com.jcraft.jsch.Channel
 [findbugs]   org.apache.hadoop.mapred.JobPriority
 [findbugs]   org.apache.commons.cli.Options
 [findbugs]   org.apache.hadoop.mapred.JobID
 [findbugs]   org.apache.hadoop.util.bloom.BloomFilter
 [findbugs]   org.python.core.PyFrame
 [findbugs]   org.apache.hadoop.hbase.filter.CompareFilter
 [findbugs]   org.apache.hadoop.util.VersionInfo
 [findbugs]   org.python.core.PyString
 [findbugs]   org.apache.hadoop.io.Text$Comparator
 [findbugs]   org.jruby.runtime.Block
 [findbugs]   org.antlr.runtime.MismatchedSetException
 [findbugs]   org.apache.hadoop.io.BytesWritable
 [findbugs]   org.apache.hadoop.fs.FsShell
 [findbugs]   org.joda.time.Months
 [findbugs]   org.mozilla.javascript.ImporterTopLevel
 [findbugs]   org.apache.hadoop.hbase.mapreduce.TableOutputFormat
 [findbugs]   org.apache.hadoop.mapred.TaskReport
 [findbugs]   org.apache.hadoop.security.UserGroupInformation
 [findbugs]   org.antlr.runtime.tree.RewriteRuleSubtreeStream
 [findbugs]   org.apache.commons.cli.HelpFormatter
 [findbugs]   com.google.common.collect.Maps
 [findbugs]   org.joda.time.ReadableInstant
 [findbugs]   org.mozilla.javascript.NativeObject
 [findbugs]   org.apache.hadoop.hbase.HConstants
 [findbugs]   org.apache.hadoop.io.serializer.Deserializer
 [findbugs]   org.antlr.runtime.FailedPredicateException
 [findbugs]   org.apache.hadoop.io.compress.CompressionCodec
 [findbugs]   org.jruby.RubyNil
 [findbugs]   org.apache.hadoop.fs.FileStatus
 [findbugs]   org.apache.hadoop.hbase.client.Result
 [findbugs]   org.apache.hadoop.mapreduce.JobContext
 [findbugs]   org.codehaus.jackson.JsonGenerator
 [findbugs]   org.apache.hadoop.mapreduce.TaskAttemptContext
 [findbugs]   org.apache.hadoop.io.BytesWritable$Comparator
 [findbugs]   org.apache.hadoop.io.LongWritable$Comparator
 [findbugs]   org.codehaus.jackson.map.util.LRUMap
 [findbugs]   org.apache.hadoop.hbase.util.Bytes
 [findbugs]   org.antlr.runtime.MismatchedTokenException
 [findbugs]   org.codehaus.jackson.JsonParser
 [findbugs]   com.jcraft.jsch.UserInfo
 [findbugs]   org.python.core.PyException
 [findbugs]   org.apache.commons.cli.ParseException
 [findbugs]   org.apache.hadoop.io.compress.CompressionOutputStream
 [findbugs]   org.apache.hadoop.hbase.filter.WritableByteArrayComparable
 [findbugs]   org.antlr.runtime.tree.CommonTreeNodeStream
 [findbugs]   org.apache.log4j.Level
 [findbugs]   org.apache.hadoop.hbase.client.Scan
 [findbugs]   org.jruby.anno.JRubyMethod
 [findbugs]   org.apache.hadoop.mapreduce.Job
 [findbugs]   com.google.common.util.concurrent.Futures
 [findbugs]   org.apache.commons.logging.LogFactory
 [findbugs]   org.apache.commons.collections.IteratorUtils
 [findbugs]   org.apache.commons.codec.binary.Base64
 [findbugs]   org.codehaus.jackson.map.ObjectMapper
 [findbugs]   org.apache.hadoop.fs.FileSystem
 [findbugs]   org.jruby.embed.LocalContextScope
 [findbugs]   org.apache.hadoop.hbase.filter.FilterList$Operator
 [findbugs]   org.jruby.RubySymbol
 [findbugs]   org.apache.hadoop.hbase.io.ImmutableBytesWritable
 [findbugs]   org.apache.hadoop.io.serializer.SerializationFactory
 [findbugs]   org.antlr.runtime.tree.TreeAdaptor
 [findbugs]   org.apache.hadoop.mapred.RunningJob
 [findbugs]   org.antlr.runtime.CommonTokenStream
 [findbugs]   org.apache.hadoop.io.DataInputBuffer
 [findbugs]   org.apache.hadoop.io.file.tfile.TFile
 [findbugs]   org.apache.commons.cli.GnuParser
 [findbugs]   org.mozilla.javascript.Context
 [findbugs]   org.apache.hadoop.io.FloatWritable
 [findbugs]   org.antlr.runtime.tree.RewriteE

[jira] [Commented] (PIG-2927) SHIP and use JRuby gems in JRuby UDFs

[Russell Jurney (JIRA)]

[ 
https://issues.apache.org/jira/browse/PIG-2927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13476545#comment-13476545
 ] 

Russell Jurney commented on PIG-2927:
-

Thanks, patch applied. Testing now.

> SHIP and use JRuby gems in JRuby UDFs
> -
>
> Key: PIG-2927
> URL: https://issues.apache.org/jira/browse/PIG-2927
> Project: Pig
>  Issue Type: New Feature
>  Components: parser
>Affects Versions: 0.11
> Environment: JRuby UDFs
>Reporter: Russell Jurney
>Assignee: Jonathan Coveney
>Priority: Minor
> Attachments: PIG-2927-0.patch, PIG-2927-1.patch
>
>
> It would be great to use JRuby gems in JRuby UDFs without installing them on 
> all machines on the cluster. Some way to SHIP them automatically with the job 
> would be great.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (PIG-2927) SHIP and use JRuby gems in JRuby UDFs

[Jonathan Coveney (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PIG-2927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jonathan Coveney updated PIG-2927:
--

Attachment: PIG-2927-1.patch

> SHIP and use JRuby gems in JRuby UDFs
> -
>
> Key: PIG-2927
> URL: https://issues.apache.org/jira/browse/PIG-2927
> Project: Pig
>  Issue Type: New Feature
>  Components: parser
>Affects Versions: 0.11
> Environment: JRuby UDFs
>Reporter: Russell Jurney
>Assignee: Jonathan Coveney
>Priority: Minor
> Attachments: PIG-2927-0.patch, PIG-2927-1.patch
>
>
> It would be great to use JRuby gems in JRuby UDFs without installing them on 
> all machines on the cluster. Some way to SHIP them automatically with the job 
> would be great.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (PIG-2927) SHIP and use JRuby gems in JRuby UDFs

[Jonathan Coveney (JIRA)]

[ 
https://issues.apache.org/jira/browse/PIG-2927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13476540#comment-13476540
 ] 

Jonathan Coveney commented on PIG-2927:
---

Russell, I pulled trunk and made a new patch. Should apply fine now.

> SHIP and use JRuby gems in JRuby UDFs
> -
>
> Key: PIG-2927
> URL: https://issues.apache.org/jira/browse/PIG-2927
> Project: Pig
>  Issue Type: New Feature
>  Components: parser
>Affects Versions: 0.11
> Environment: JRuby UDFs
>Reporter: Russell Jurney
>Assignee: Jonathan Coveney
>Priority: Minor
> Attachments: PIG-2927-0.patch, PIG-2927-1.patch
>
>
> It would be great to use JRuby gems in JRuby UDFs without installing them on 
> all machines on the cluster. Some way to SHIP them automatically with the job 
> would be great.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (PIG-2908) Fix unit tests to work with jdk7

[Dmitriy V. Ryaboy (JIRA)]

[ 
https://issues.apache.org/jira/browse/PIG-2908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13476534#comment-13476534
 ] 

Dmitriy V. Ryaboy commented on PIG-2908:


Done

> Fix unit tests to work with jdk7
> 
>
> Key: PIG-2908
> URL: https://issues.apache.org/jira/browse/PIG-2908
> Project: Pig
>  Issue Type: Sub-task
>Affects Versions: 0.10.0
>Reporter: Rohini Palaniswamy
>Assignee: Rohini Palaniswamy
>Priority: Minor
> Fix For: 0.11, 0.10.1
>
> Attachments: PIG-2908-branch10-1.patch, PIG-2908-branch10.patch, 
> PIG-2908-trunk-1.patch, PIG-2908-trunk.patch
>
>
> Fix unit tests that fail when compiling with jdk7 as they depend on the order 
> of tests run - TestGFCross, TestJsonLoaderStorage, TestLogToPhyCompiler, 
> TestMRCompiler, TestMergeJoinOuter and TestNewPlanLogToPhyTranslationVisitor. 
> jdk7 returns methods in a different order during reflection.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (PIG-2908) Fix unit tests to work with jdk7

[Rohini Palaniswamy (JIRA)]

[ 
https://issues.apache.org/jira/browse/PIG-2908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13476533#comment-13476533
 ] 

Rohini Palaniswamy commented on PIG-2908:
-

Yes. OrderedJunitRunner

> Fix unit tests to work with jdk7
> 
>
> Key: PIG-2908
> URL: https://issues.apache.org/jira/browse/PIG-2908
> Project: Pig
>  Issue Type: Sub-task
>Affects Versions: 0.10.0
>Reporter: Rohini Palaniswamy
>Assignee: Rohini Palaniswamy
>Priority: Minor
> Fix For: 0.11, 0.10.1
>
> Attachments: PIG-2908-branch10-1.patch, PIG-2908-branch10.patch, 
> PIG-2908-trunk-1.patch, PIG-2908-trunk.patch
>
>
> Fix unit tests that fail when compiling with jdk7 as they depend on the order 
> of tests run - TestGFCross, TestJsonLoaderStorage, TestLogToPhyCompiler, 
> TestMRCompiler, TestMergeJoinOuter and TestNewPlanLogToPhyTranslationVisitor. 
> jdk7 returns methods in a different order during reflection.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (PIG-2908) Fix unit tests to work with jdk7

[Dmitriy V. Ryaboy (JIRA)]

[ 
https://issues.apache.org/jira/browse/PIG-2908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13476531#comment-13476531
 ] 

Dmitriy V. Ryaboy commented on PIG-2908:


test/org/apache/pig/test/junit ?

> Fix unit tests to work with jdk7
> 
>
> Key: PIG-2908
> URL: https://issues.apache.org/jira/browse/PIG-2908
> Project: Pig
>  Issue Type: Sub-task
>Affects Versions: 0.10.0
>Reporter: Rohini Palaniswamy
>Assignee: Rohini Palaniswamy
>Priority: Minor
> Fix For: 0.11, 0.10.1
>
> Attachments: PIG-2908-branch10-1.patch, PIG-2908-branch10.patch, 
> PIG-2908-trunk-1.patch, PIG-2908-trunk.patch
>
>
> Fix unit tests that fail when compiling with jdk7 as they depend on the order 
> of tests run - TestGFCross, TestJsonLoaderStorage, TestLogToPhyCompiler, 
> TestMRCompiler, TestMergeJoinOuter and TestNewPlanLogToPhyTranslationVisitor. 
> jdk7 returns methods in a different order during reflection.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (PIG-2908) Fix unit tests to work with jdk7

[Rohini Palaniswamy (JIRA)]

[ 
https://issues.apache.org/jira/browse/PIG-2908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13476530#comment-13476530
 ] 

Rohini Palaniswamy commented on PIG-2908:
-

You had missed the svn add step. Can you commit the missing new test file?

> Fix unit tests to work with jdk7
> 
>
> Key: PIG-2908
> URL: https://issues.apache.org/jira/browse/PIG-2908
> Project: Pig
>  Issue Type: Sub-task
>Affects Versions: 0.10.0
>Reporter: Rohini Palaniswamy
>Assignee: Rohini Palaniswamy
>Priority: Minor
> Fix For: 0.11, 0.10.1
>
> Attachments: PIG-2908-branch10-1.patch, PIG-2908-branch10.patch, 
> PIG-2908-trunk-1.patch, PIG-2908-trunk.patch
>
>
> Fix unit tests that fail when compiling with jdk7 as they depend on the order 
> of tests run - TestGFCross, TestJsonLoaderStorage, TestLogToPhyCompiler, 
> TestMRCompiler, TestMergeJoinOuter and TestNewPlanLogToPhyTranslationVisitor. 
> jdk7 returns methods in a different order during reflection.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (PIG-2928) Fix e2e test failures in trunk: FilterBoolean_23/24

[Dmitriy V. Ryaboy (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PIG-2928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Dmitriy V. Ryaboy updated PIG-2928:
---

Resolution: Fixed
Status: Resolved  (was: Patch Available)

Committed to 0.11 branch and trunk.
Thanks Cheolsoo.

> Fix e2e test failures in trunk: FilterBoolean_23/24
> ---
>
> Key: PIG-2928
> URL: https://issues.apache.org/jira/browse/PIG-2928
> Project: Pig
>  Issue Type: Sub-task
>Affects Versions: 0.10.0
>Reporter: Cheolsoo Park
>Assignee: Cheolsoo Park
> Fix For: 0.11
>
> Attachments: PIG-2928.patch
>
>
> To reproduce the failure, please run
> {code}
> ant -Dhadoopversion=20 ... -Dtest-e2e-deploy-local
> ant -Dhadoopversion=20 ... -Dtest-e2e-local -Dtests.to.run="-t 
> FilterBoolean_23"
> {code}
> This is a regression from PIG-1314.
> PIG-1314 reverted the following change made by PIG-2593 that had made it 
> possible to filter by a boolean value.
> {code}
> --- src/org/apache/pig/parser/QueryParser.g
> +++ src/org/apache/pig/parser/QueryParser.g
> @@ -391,13 +391,16 @@ or_cond : and_cond  ( OR^ and_cond )*
>  and_cond : unary_cond ( AND^ unary_cond )*
>  ;
>  
> -unary_cond : LEFT_PAREN! cond RIGHT_PAREN!
> -   | not_cond
> -   | expr rel_op^ expr
> +unary_cond : expr rel_op^ expr
> +   | LEFT_PAREN! cond RIGHT_PAREN!
> +   | not_cond   
> | func_eval
> | null_check_cond
> +   | bool_cond   
>  ;
>  
> +bool_cond: expr -> ^(BOOL_COND expr);
> +
>  not_cond : NOT^ unary_cond
>  ;
> @@ -784,6 +787,7 @@ eid : rel_str_op
>  | TRUE
>  | FALSE
>  | REALIAS
> +| BOOL_COND
>  ;
> {code}
> This makes FilterBoolean_23/24 in e2e test fail.
> If this change is intended, we should disable FilterBoolean_23/24.
> If not, we should put the reverted change back to the grammar.
> Thanks!

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (PIG-2908) Fix unit tests to work with jdk7

[Rohini Palaniswamy (JIRA)]

[ 
https://issues.apache.org/jira/browse/PIG-2908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13476521#comment-13476521
 ] 

Rohini Palaniswamy commented on PIG-2908:
-

Thanks Dmitriy for committing this one. 

> Fix unit tests to work with jdk7
> 
>
> Key: PIG-2908
> URL: https://issues.apache.org/jira/browse/PIG-2908
> Project: Pig
>  Issue Type: Sub-task
>Affects Versions: 0.10.0
>Reporter: Rohini Palaniswamy
>Assignee: Rohini Palaniswamy
>Priority: Minor
> Fix For: 0.11, 0.10.1
>
> Attachments: PIG-2908-branch10-1.patch, PIG-2908-branch10.patch, 
> PIG-2908-trunk-1.patch, PIG-2908-trunk.patch
>
>
> Fix unit tests that fail when compiling with jdk7 as they depend on the order 
> of tests run - TestGFCross, TestJsonLoaderStorage, TestLogToPhyCompiler, 
> TestMRCompiler, TestMergeJoinOuter and TestNewPlanLogToPhyTranslationVisitor. 
> jdk7 returns methods in a different order during reflection.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (PIG-1314) Add DateTime Support to Pig

[Dmitriy V. Ryaboy (JIRA)]

[ 
https://issues.apache.org/jira/browse/PIG-1314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13476519#comment-13476519
 ] 

Dmitriy V. Ryaboy commented on PIG-1314:


A chunk of this is committed, and it's not clear what's left to do. Can we 
close this and create a new ticket for the remaining work?

> Add DateTime Support to Pig
> ---
>
> Key: PIG-1314
> URL: https://issues.apache.org/jira/browse/PIG-1314
> Project: Pig
>  Issue Type: Bug
>  Components: data
>Affects Versions: 0.7.0
>Reporter: Russell Jurney
>Assignee: Zhijie Shen
>  Labels: gsoc2012
> Attachments: joda_vs_builtin.zip, PIG-1314-1.patch, PIG-1314-2.patch, 
> PIG-1314-3.patch, PIG-1314-4.patch, PIG-1314-5.patch, PIG-1314-6.patch, 
> PIG-1314-7.patch
>
>   Original Estimate: 672h
>  Remaining Estimate: 672h
>
> Hadoop/Pig are primarily used to parse log data, and most logs have a 
> timestamp component.  Therefore Pig should support dates as a primitive.
> Can someone familiar with adding types to pig comment on how hard this is?  
> We're looking at doing this, rather than use UDFs.  Is this a patch that 
> would be accepted?
> This is a candidate project for Google summer of code 2012. More information 
> about the program can be found at 
> https://cwiki.apache.org/confluence/display/PIG/GSoc2012

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (PIG-2908) Fix unit tests to work with jdk7

[Dmitriy V. Ryaboy (JIRA)]

[ 
https://issues.apache.org/jira/browse/PIG-2908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13476516#comment-13476516
 ] 

Dmitriy V. Ryaboy commented on PIG-2908:


Thanks for seeing this important but non-flashy ticket to completion, Rohini!

> Fix unit tests to work with jdk7
> 
>
> Key: PIG-2908
> URL: https://issues.apache.org/jira/browse/PIG-2908
> Project: Pig
>  Issue Type: Sub-task
>Affects Versions: 0.10.0
>Reporter: Rohini Palaniswamy
>Assignee: Rohini Palaniswamy
>Priority: Minor
> Fix For: 0.11, 0.10.1
>
> Attachments: PIG-2908-branch10-1.patch, PIG-2908-branch10.patch, 
> PIG-2908-trunk-1.patch, PIG-2908-trunk.patch
>
>
> Fix unit tests that fail when compiling with jdk7 as they depend on the order 
> of tests run - TestGFCross, TestJsonLoaderStorage, TestLogToPhyCompiler, 
> TestMRCompiler, TestMergeJoinOuter and TestNewPlanLogToPhyTranslationVisitor. 
> jdk7 returns methods in a different order during reflection.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (PIG-2908) Fix unit tests to work with jdk7

[Dmitriy V. Ryaboy (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PIG-2908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Dmitriy V. Ryaboy updated PIG-2908:
---

Resolution: Fixed
Status: Resolved  (was: Patch Available)

Committed to 0.11, 0.10, and trunk branches.

> Fix unit tests to work with jdk7
> 
>
> Key: PIG-2908
> URL: https://issues.apache.org/jira/browse/PIG-2908
> Project: Pig
>  Issue Type: Sub-task
>Affects Versions: 0.10.0
>Reporter: Rohini Palaniswamy
>Assignee: Rohini Palaniswamy
>Priority: Minor
> Fix For: 0.11, 0.10.1
>
> Attachments: PIG-2908-branch10-1.patch, PIG-2908-branch10.patch, 
> PIG-2908-trunk-1.patch, PIG-2908-trunk.patch
>
>
> Fix unit tests that fail when compiling with jdk7 as they depend on the order 
> of tests run - TestGFCross, TestJsonLoaderStorage, TestLogToPhyCompiler, 
> TestMRCompiler, TestMergeJoinOuter and TestNewPlanLogToPhyTranslationVisitor. 
> jdk7 returns methods in a different order during reflection.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (PIG-2979) ant test-e2e-local fails due to missing run-time dependencies in classpath with hadoop-2.0.x

[Cheolsoo Park (JIRA)]

Cheolsoo Park created PIG-2979:
--

 Summary: ant test-e2e-local fails due to missing run-time 
dependencies in classpath with hadoop-2.0.x
 Key: PIG-2979
 URL: https://issues.apache.org/jira/browse/PIG-2979
 Project: Pig
  Issue Type: Sub-task
Reporter: Cheolsoo Park


To reproduce, please run on machine where no Hadoop is installed:
{code}
ant clean
ant -Dharness.old.pig=old_pig -Dharness.cluster.conf=hadoop_conf_dir 
-Dharness.cluster.bin=hadoop_script test-e2e-deploy-local -Dhadoopversion=23
ant -Dharness.old.pig=old_pig -Dharness.cluster.conf=hadoop_conf_dir 
-Dharness.cluster.bin=hadoop_script test-e2e-local -Dhadoopversion=23
{code}

The ant test-e2e-local fails with the following error:
{code}
java.lang.NoClassDefFoundError: org/slf4j/LoggerFactory
at 
org.apache.hadoop.security.authentication.util.KerberosName.(KerberosName.java:42)
at 
org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:211)
at 
org.apache.hadoop.security.UserGroupInformation.isSecurityEnabled(UserGroupInformation.java:274)
at 
org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:531)
at 
org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:512)
{code}

In fact, this is also an issue with running Pig in local mode with the fat jar 
where no hadoop dependencies are available in classpath. For example, the 
following command also fails with the same error:
{code}
ant clean jar -Dhadoopversion=23
./bin/pig -x local
{code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (PIG-2927) SHIP and use JRuby gems in JRuby UDFs

[Russell Jurney (JIRA)]

[ 
https://issues.apache.org/jira/browse/PIG-2927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13476494#comment-13476494
 ] 

Russell Jurney commented on PIG-2927:
-

Patch does not apply to trunk, please advise:

Russells-MacBook-Pro:pig-trunk rjurney$ patch -p0  SHIP and use JRuby gems in JRuby UDFs
> -
>
> Key: PIG-2927
> URL: https://issues.apache.org/jira/browse/PIG-2927
> Project: Pig
>  Issue Type: New Feature
>  Components: parser
>Affects Versions: 0.11
> Environment: JRuby UDFs
>Reporter: Russell Jurney
>Assignee: Jonathan Coveney
>Priority: Minor
> Attachments: PIG-2927-0.patch
>
>
> It would be great to use JRuby gems in JRuby UDFs without installing them on 
> all machines on the cluster. Some way to SHIP them automatically with the job 
> would be great.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (PIG-2978) TestLoadStoreFuncLifeCycle fails with hadoop-2.0.x

[Cheolsoo Park (JIRA)]

Cheolsoo Park created PIG-2978:
--

 Summary: TestLoadStoreFuncLifeCycle fails with hadoop-2.0.x
 Key: PIG-2978
 URL: https://issues.apache.org/jira/browse/PIG-2978
 Project: Pig
  Issue Type: Sub-task
Reporter: Cheolsoo Park


To reproduce, please run:
{code}
ant clean test -Dtestcase=TestLoadStoreFuncLifeCycle -Dhadoopversion=23
{code}

This fails with the following error:
{code}
Error during parsing. Job in state DEFINE instead of RUNNING
org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1000: Error during 
parsing. Job in state DEFINE instead of RUNNING
at org.apache.pig.PigServer$Graph.parseQuery(PigServer.java:1607)
at org.apache.pig.PigServer$Graph.registerQuery(PigServer.java:1546)
at org.apache.pig.PigServer.registerQuery(PigServer.java:516)
at org.apache.pig.PigServer.registerQuery(PigServer.java:529)
at 
org.apache.pig.TestLoadStoreFuncLifeCycle.testLoadStoreFunc(TestLoadStoreFuncLifeCycle.java:332)
Caused by: Failed to parse: Job in state DEFINE instead of RUNNING
at org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:193)
at org.apache.pig.PigServer$Graph.parseQuery(PigServer.java:1599)
Caused by: java.lang.IllegalStateException: Job in state DEFINE instead of 
RUNNING
at org.apache.hadoop.mapreduce.Job.ensureState(Job.java:292)
at org.apache.hadoop.mapreduce.Job.toString(Job.java:456)
at java.lang.String.valueOf(String.java:2826)
at 
org.apache.pig.TestLoadStoreFuncLifeCycle.logCaller(TestLoadStoreFuncLifeCycle.java:270)
at 
org.apache.pig.TestLoadStoreFuncLifeCycle.access$000(TestLoadStoreFuncLifeCycle.java:41)
at 
org.apache.pig.TestLoadStoreFuncLifeCycle$InstrumentedStorage.logCaller(TestLoadStoreFuncLifeCycle.java:54)
at 
org.apache.pig.TestLoadStoreFuncLifeCycle$InstrumentedStorage.getSchema(TestLoadStoreFuncLifeCycle.java:115)
at 
org.apache.pig.newplan.logical.relational.LOLoad.getSchemaFromMetaData(LOLoad.java:174)
at org.apache.pig.newplan.logical.relational.LOLoad.(LOLoad.java:88)
at 
org.apache.pig.parser.LogicalPlanBuilder.buildLoadOp(LogicalPlanBuilder.java:839)
at 
org.apache.pig.parser.LogicalPlanGenerator.load_clause(LogicalPlanGenerator.java:3236)
at 
org.apache.pig.parser.LogicalPlanGenerator.op_clause(LogicalPlanGenerator.java:1315)
at 
org.apache.pig.parser.LogicalPlanGenerator.general_statement(LogicalPlanGenerator.java:799)
at 
org.apache.pig.parser.LogicalPlanGenerator.statement(LogicalPlanGenerator.java:517)
at 
org.apache.pig.parser.LogicalPlanGenerator.query(LogicalPlanGenerator.java:392)
at org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:184)
{code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (PIG-2908) Fix unit tests to work with jdk7

[Rohini Palaniswamy (JIRA)]

[ 
https://issues.apache.org/jira/browse/PIG-2908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13476475#comment-13476475
 ] 

Rohini Palaniswamy commented on PIG-2908:
-

Dmitriy,
  We have a mandate to upgrade to jdk7 by end of Nov and we are going to have 
0.10.1 around for a while before 0.11 gets stabilized and we internally release 
0.11. That's why need it on 0.10 too. 

> Fix unit tests to work with jdk7
> 
>
> Key: PIG-2908
> URL: https://issues.apache.org/jira/browse/PIG-2908
> Project: Pig
>  Issue Type: Sub-task
>Affects Versions: 0.10.0
>Reporter: Rohini Palaniswamy
>Assignee: Rohini Palaniswamy
>Priority: Minor
> Fix For: 0.11, 0.10.1
>
> Attachments: PIG-2908-branch10-1.patch, PIG-2908-branch10.patch, 
> PIG-2908-trunk-1.patch, PIG-2908-trunk.patch
>
>
> Fix unit tests that fail when compiling with jdk7 as they depend on the order 
> of tests run - TestGFCross, TestJsonLoaderStorage, TestLogToPhyCompiler, 
> TestMRCompiler, TestMergeJoinOuter and TestNewPlanLogToPhyTranslationVisitor. 
> jdk7 returns methods in a different order during reflection.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (PIG-2927) SHIP and use JRuby gems in JRuby UDFs

[Russell Jurney (JIRA)]

[ 
https://issues.apache.org/jira/browse/PIG-2927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13476471#comment-13476471
 ] 

Russell Jurney commented on PIG-2927:
-

Testing now...

> SHIP and use JRuby gems in JRuby UDFs
> -
>
> Key: PIG-2927
> URL: https://issues.apache.org/jira/browse/PIG-2927
> Project: Pig
>  Issue Type: New Feature
>  Components: parser
>Affects Versions: 0.11
> Environment: JRuby UDFs
>Reporter: Russell Jurney
>Assignee: Jonathan Coveney
>Priority: Minor
> Attachments: PIG-2927-0.patch
>
>
> It would be great to use JRuby gems in JRuby UDFs without installing them on 
> all machines on the cluster. Some way to SHIP them automatically with the job 
> would be great.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (PIG-2977) TestHBaseStorage fails with hadoop-2.0.x

[Cheolsoo Park (JIRA)]

Cheolsoo Park created PIG-2977:
--

 Summary: TestHBaseStorage fails with hadoop-2.0.x
 Key: PIG-2977
 URL: https://issues.apache.org/jira/browse/PIG-2977
 Project: Pig
  Issue Type: Sub-task
Reporter: Cheolsoo Park


To reproduce, please run:
{code}
ant clean test -Dtestcase=TestHBaseStorage -Dhadoopversion=23
{code}

In the test log, HBase master fails to start with the following error:
{code}
2012-10-15 14:01:28,455 FATAL [Master:0;cheolsoo-mr2-1.ent.cloudera.com:46920] 
master.HMaster (HMaster.java:abort(936)) - Unhandled exception. Starting 
shutdown.
java.lang.NoClassDefFoundError: 
org/apache/hadoop/hdfs/protocol/FSConstants$SafeModeAction
at org.apache.hadoop.hbase.util.FSUtils.waitOnSafeMode(FSUtils.java:280)
at 
org.apache.hadoop.hbase.master.MasterFileSystem.checkRootDir(MasterFileSystem.java:229)
at 
org.apache.hadoop.hbase.master.MasterFileSystem.createInitialFileSystemLayout(MasterFileSystem.java:105)
at 
org.apache.hadoop.hbase.master.MasterFileSystem.(MasterFileSystem.java:90)
at 
org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:341)
at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:277)
at java.lang.Thread.run(Thread.java:662)
{code}

This is because hbase.jar in the maven repository is not compatible with hadoop 
23/2.0 (HBASE-5680).

The proposed fix is to disable this test in MR2.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (PIG-2908) Fix unit tests to work with jdk7

[Dmitriy V. Ryaboy (JIRA)]

[ 
https://issues.apache.org/jira/browse/PIG-2908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13476467#comment-13476467
 ] 

Dmitriy V. Ryaboy commented on PIG-2908:


+1 for trunk and 0.11

Why do we need to apply this to 0.10?

> Fix unit tests to work with jdk7
> 
>
> Key: PIG-2908
> URL: https://issues.apache.org/jira/browse/PIG-2908
> Project: Pig
>  Issue Type: Sub-task
>Affects Versions: 0.10.0
>Reporter: Rohini Palaniswamy
>Assignee: Rohini Palaniswamy
>Priority: Minor
> Fix For: 0.11, 0.10.1
>
> Attachments: PIG-2908-branch10-1.patch, PIG-2908-branch10.patch, 
> PIG-2908-trunk-1.patch, PIG-2908-trunk.patch
>
>
> Fix unit tests that fail when compiling with jdk7 as they depend on the order 
> of tests run - TestGFCross, TestJsonLoaderStorage, TestLogToPhyCompiler, 
> TestMRCompiler, TestMergeJoinOuter and TestNewPlanLogToPhyTranslationVisitor. 
> jdk7 returns methods in a different order during reflection.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (PIG-2976) Reduce HBaseStorage logging

[Aneesh Sharma (JIRA)]

Aneesh Sharma created PIG-2976:
--

 Summary: Reduce HBaseStorage logging
 Key: PIG-2976
 URL: https://issues.apache.org/jira/browse/PIG-2976
 Project: Pig
  Issue Type: Improvement
Reporter: Aneesh Sharma
Assignee: Bill Graham


The current logging in HBaseStorage is very verbose, filling the log with lines 
like these:
2012-10-05 00:59:01,158 INFO org.apache.pig.backend.hadoop.hbase.HBaseStorage: 
Adding family:descriptor filters with values extracted:user_logins

It would be nice to reduce this verbosity!

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (PIG-2971) Add new parameter to specify the streaming environment

[Dmitriy V. Ryaboy (JIRA)]

[ 
https://issues.apache.org/jira/browse/PIG-2971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13476457#comment-13476457
 ] 

Dmitriy V. Ryaboy commented on PIG-2971:


Please update documentation, and a commented out example in the default 
pig.properties.


> Add new parameter to specify the streaming environment
> --
>
> Key: PIG-2971
> URL: https://issues.apache.org/jira/browse/PIG-2971
> Project: Pig
>  Issue Type: Bug
>Reporter: Jonathan Coveney
>Assignee: Jonathan Coveney
> Fix For: 0.11
>
> Attachments: PIG-2971-0.patch
>
>
> The current proposal in PIG-2925 is a bit arbitrary. How do we set what is 
> and is not too big? This instead let's the user set what they need.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (PIG-2975) TestTypedMap.testOrderBy failing with incorrect result

[Koji Noguchi (JIRA)]

[ 
https://issues.apache.org/jira/browse/PIG-2975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13476447#comment-13476447
 ] 

Koji Noguchi commented on PIG-2975:
---

I haven't had time to look into detail but reverting PIG-2862 seems to fix the 
issue.
(Test started failing only after PIG-2936 when BinInterSedes byte-identifier 
started to get picked up by the test.)

> TestTypedMap.testOrderBy failing with incorrect result 
> ---
>
> Key: PIG-2975
> URL: https://issues.apache.org/jira/browse/PIG-2975
> Project: Pig
>  Issue Type: Sub-task
>Affects Versions: 0.11
>Reporter: Koji Noguchi
>Priority: Blocker
> Fix For: 0.11
>
>
> Looked at 
> {noformat}
> junit.framework.AssertionFailedError
> at org.apache.pig.test.TestTypedMap.testOrderBy(TestTypedMap.java:352)
> {noformat}
> This looks like a valid test case failing with incorrect result.
> {noformat}
> % cat test/orderby.txt
> [key#1,key9#23]
> [key#3,key3#2]
> [key#22]
> % cat test/orderby.pig
> a = load 'test/orderby.txt' as (m:[]);
> b = foreach a generate m#'key' as b0;
> dump b;
> c = order b by b0;
> dump c;
> % java ... org.apache.pig.Main-x local test/orderby.pig 
> [dump b]
> (1)
> (3)
> (22)
> ...
> [dump c]
> (1)
> (1)
> (22)
> %
> where did the '(3)' go?
> {noformat}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (PIG-2975) TestTypedMap.testOrderBy failing with incorrect result

[Koji Noguchi (JIRA)]

Koji Noguchi created PIG-2975:
-

 Summary: TestTypedMap.testOrderBy failing with incorrect result 
 Key: PIG-2975
 URL: https://issues.apache.org/jira/browse/PIG-2975
 Project: Pig
  Issue Type: Sub-task
Affects Versions: 0.11
Reporter: Koji Noguchi
Priority: Blocker


Looked at 
{noformat}
junit.framework.AssertionFailedError
at org.apache.pig.test.TestTypedMap.testOrderBy(TestTypedMap.java:352)
{noformat}

This looks like a valid test case failing with incorrect result.

{noformat}
% cat test/orderby.txt
[key#1,key9#23]
[key#3,key3#2]
[key#22]

% cat test/orderby.pig
a = load 'test/orderby.txt' as (m:[]);
b = foreach a generate m#'key' as b0;
dump b;
c = order b by b0;
dump c;

% java ... org.apache.pig.Main-x local test/orderby.pig 
[dump b]
(1)
(3)
(22)

...
[dump c]
(1)
(1)
(22)
%

where did the '(3)' go?
{noformat}



--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (PIG-2971) Add new parameter to specify the streaming environment

[Jonathan Coveney (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PIG-2971?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jonathan Coveney updated PIG-2971:
--

Resolution: Fixed
Status: Resolved  (was: Patch Available)

> Add new parameter to specify the streaming environment
> --
>
> Key: PIG-2971
> URL: https://issues.apache.org/jira/browse/PIG-2971
> Project: Pig
>  Issue Type: Bug
>Reporter: Jonathan Coveney
>Assignee: Jonathan Coveney
> Fix For: 0.11
>
> Attachments: PIG-2971-0.patch
>
>
> The current proposal in PIG-2925 is a bit arbitrary. How do we set what is 
> and is not too big? This instead let's the user set what they need.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Work started] (PIG-2971) Add new parameter to specify the streaming environment

[Jonathan Coveney (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PIG-2971?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on PIG-2971 started by Jonathan Coveney.

> Add new parameter to specify the streaming environment
> --
>
> Key: PIG-2971
> URL: https://issues.apache.org/jira/browse/PIG-2971
> Project: Pig
>  Issue Type: Bug
>Reporter: Jonathan Coveney
>Assignee: Jonathan Coveney
> Fix For: 0.11
>
> Attachments: PIG-2971-0.patch
>
>
> The current proposal in PIG-2925 is a bit arbitrary. How do we set what is 
> and is not too big? This instead let's the user set what they need.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (PIG-2971) Add new parameter to specify the streaming environment

[Jonathan Coveney (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PIG-2971?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jonathan Coveney updated PIG-2971:
--

Status: Patch Available  (was: In Progress)

> Add new parameter to specify the streaming environment
> --
>
> Key: PIG-2971
> URL: https://issues.apache.org/jira/browse/PIG-2971
> Project: Pig
>  Issue Type: Bug
>Reporter: Jonathan Coveney
>Assignee: Jonathan Coveney
> Fix For: 0.11
>
> Attachments: PIG-2971-0.patch
>
>
> The current proposal in PIG-2925 is a bit arbitrary. How do we set what is 
> and is not too big? This instead let's the user set what they need.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (PIG-2971) Add new parameter to specify the streaming environment

[Julien Le Dem (JIRA)]

[ 
https://issues.apache.org/jira/browse/PIG-2971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13476430#comment-13476430
 ] 

Julien Le Dem commented on PIG-2971:


+1 looks good
Thanks!

> Add new parameter to specify the streaming environment
> --
>
> Key: PIG-2971
> URL: https://issues.apache.org/jira/browse/PIG-2971
> Project: Pig
>  Issue Type: Bug
>Reporter: Jonathan Coveney
>Assignee: Jonathan Coveney
> Fix For: 0.11
>
> Attachments: PIG-2971-0.patch
>
>
> The current proposal in PIG-2925 is a bit arbitrary. How do we set what is 
> and is not too big? This instead let's the user set what they need.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (PIG-2925) Extremely long JobConf values should not be added to Streaming environment

[Jonathan Coveney (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PIG-2925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jonathan Coveney updated PIG-2925:
--

Resolution: Won't Fix
Status: Resolved  (was: Patch Available)

We are instead using the method in PIG-2971

> Extremely long JobConf values should not be added to Streaming environment
> --
>
> Key: PIG-2925
> URL: https://issues.apache.org/jira/browse/PIG-2925
> Project: Pig
>  Issue Type: Bug
>Reporter: Dmitriy V. Ryaboy
>Assignee: Dmitriy V. Ryaboy
> Attachments: PIG-2925.patch
>
>
> PIG-2900 made Pig match Hadoop's practice of making all JobConf properties 
> available as environment variables for pig streaming. 
> As it happens, ProcessBuilder creates a long string with all environment 
> variables and their values on it, then runs it; having some of the weightier 
> properties like serialized plans in the environment can make the total 
> command length exceed OS limits. 
> We propose adding a blacklist of properties which should not be added to 
> streaming environment.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Re: Unit test failures

[Rohini Palaniswamy]

Thanks Alan. I had missed looking at that umbrella jira.

On Mon, Oct 15, 2012 at 1:38 PM, Alan Gates  wrote:

> I would like to push the fixes for Windows into 0.11, as they are mostly
> small bug fixes.  There's already an umbrella JIRA for these,
> https://issues.apache.org/jira/browse/PIG-2793
>
> Alan.
>
> On Oct 15, 2012, at 1:26 PM, Rohini Palaniswamy wrote:
>
> > Me and Cheolsoo are kicking off a new run for the tests to check the
> > current failures. Will update the jira with those. Interested folks can
> > pick up the sub-task jiras.
> >
> > Should we create another umbrella ticket just for the windows test
> > failures? HW and Microsoft folks are putting in effort to fix them. Not
> > sure if the plan is to get them into 0.11 or make it part of trunk.
> Either
> > way another umbrella ticket for that would be good.
> >
> > Regards,
> > Rohini
> >
> > On Mon, Oct 15, 2012 at 12:17 PM, Cheolsoo Park  >wrote:
> >
> >> Now we have the umbrella ticket for test failures (unit/e2e):
> >> https://issues.apache.org/jira/browse/PIG-2972
> >>
> >> Please add new jiras as sub-tasks to it.
> >>
> >> Thanks,
> >> Cheolsoo
> >>
> >> On Mon, Oct 15, 2012 at 11:49 AM, Cheolsoo Park  >>> wrote:
> >>
> > Thinking of creating a umbrella ticket to track all current unit
> >>> test failures and get them fixed.
> >>>
> >>> Once we open a umbrella ticket for test failures, everyone should link
> >>> their jiras that are for failing tests to it. I know that there are
> >> already
> >>> many open jiras for failing tests. I have several jiras myself.
> >>>
> >>> Please let me know if you want me to open the umbrella ticket and link
> >>> existing tickets to it.
> >>>
> >>> Thanks,
> >>> Cheolsoo
> >>>
> >>>
> >>> On Mon, Oct 15, 2012 at 11:36 AM, Cheolsoo Park  >>> wrote:
> >>>
>  This will be great! Please feel free to assign me bug fixes.
> 
> 
>  On Mon, Oct 15, 2012 at 11:32 AM, Jonathan Coveney <
> jcove...@gmail.com
> >>> wrote:
> 
> > I think this is an excellent idea, Rohini. We have resources to fix
> >> these
> > failures, so pointing them out should get them fixed. The error we've
> >> run
> > into is errors that are transient or related to a specific build on a
> > specific machine (and thus hard to replicate) vs. more serious
> errors.
> > Both
> > should be fixed, but it's hard to do the former when you have no
> >> control
> > over the CI. Throw in all the false fails from clover issues or
> > what-have-you and we're in a bad state. Cleaning that up would be
> > awesome.
> >
> > 2012/10/15 Julien Le Dem 
> >
> >> I think this is a great idea.
> >> I don't think we have access to the build right now.
> >> Daniel seemed to know more about that.
> >> Julien
> >>
> >> On Mon, Oct 15, 2012 at 11:15 AM, Rohini Palaniswamy <
> >> rohini.adi...@gmail.com> wrote:
> >>
> >>> Hi all,
> >>>Quite a few unit tests are broken in trunk/0.11. Now that 0.11
> >> is
> >>> branched it is more important to have them fixed and pass for
> > stability.
> >>> Also we need to figure out some way to have tests not broken. But
> > with
> >> the
> >>> time unit tests take it is difficult for everyone to run it. I
> >> think
> > the
> >>> first thing to do would be to have the Apache jenkins build fixed
> >> and
> >> treat
> >>> build failures there seriously. I can help out with that if someone
> > can
> >>> guide me with access, modifying jenkins scripts, etc. Any other
> > ideas?
> >>>   Thinking of creating a umbrella ticket to track all current unit
> > test
> >>> failures and get them fixed. Or is there some other process that we
> >> usually
> >>> follow?
> >>>
> >>> Thanks,
> >>> Rohini
> >>>
> >>
> >
> 
> 
> >>>
> >>
>
>

Re: Unit test failures

[Alan Gates]

I would like to push the fixes for Windows into 0.11, as they are mostly small 
bug fixes.  There's already an umbrella JIRA for these, 
https://issues.apache.org/jira/browse/PIG-2793

Alan.

On Oct 15, 2012, at 1:26 PM, Rohini Palaniswamy wrote:

> Me and Cheolsoo are kicking off a new run for the tests to check the
> current failures. Will update the jira with those. Interested folks can
> pick up the sub-task jiras.
> 
> Should we create another umbrella ticket just for the windows test
> failures? HW and Microsoft folks are putting in effort to fix them. Not
> sure if the plan is to get them into 0.11 or make it part of trunk. Either
> way another umbrella ticket for that would be good.
> 
> Regards,
> Rohini
> 
> On Mon, Oct 15, 2012 at 12:17 PM, Cheolsoo Park wrote:
> 
>> Now we have the umbrella ticket for test failures (unit/e2e):
>> https://issues.apache.org/jira/browse/PIG-2972
>> 
>> Please add new jiras as sub-tasks to it.
>> 
>> Thanks,
>> Cheolsoo
>> 
>> On Mon, Oct 15, 2012 at 11:49 AM, Cheolsoo Park >> wrote:
>> 
> Thinking of creating a umbrella ticket to track all current unit
>>> test failures and get them fixed.
>>> 
>>> Once we open a umbrella ticket for test failures, everyone should link
>>> their jiras that are for failing tests to it. I know that there are
>> already
>>> many open jiras for failing tests. I have several jiras myself.
>>> 
>>> Please let me know if you want me to open the umbrella ticket and link
>>> existing tickets to it.
>>> 
>>> Thanks,
>>> Cheolsoo
>>> 
>>> 
>>> On Mon, Oct 15, 2012 at 11:36 AM, Cheolsoo Park >> wrote:
>>> 
 This will be great! Please feel free to assign me bug fixes.
 
 
 On Mon, Oct 15, 2012 at 11:32 AM, Jonathan Coveney >> wrote:
 
> I think this is an excellent idea, Rohini. We have resources to fix
>> these
> failures, so pointing them out should get them fixed. The error we've
>> run
> into is errors that are transient or related to a specific build on a
> specific machine (and thus hard to replicate) vs. more serious errors.
> Both
> should be fixed, but it's hard to do the former when you have no
>> control
> over the CI. Throw in all the false fails from clover issues or
> what-have-you and we're in a bad state. Cleaning that up would be
> awesome.
> 
> 2012/10/15 Julien Le Dem 
> 
>> I think this is a great idea.
>> I don't think we have access to the build right now.
>> Daniel seemed to know more about that.
>> Julien
>> 
>> On Mon, Oct 15, 2012 at 11:15 AM, Rohini Palaniswamy <
>> rohini.adi...@gmail.com> wrote:
>> 
>>> Hi all,
>>>Quite a few unit tests are broken in trunk/0.11. Now that 0.11
>> is
>>> branched it is more important to have them fixed and pass for
> stability.
>>> Also we need to figure out some way to have tests not broken. But
> with
>> the
>>> time unit tests take it is difficult for everyone to run it. I
>> think
> the
>>> first thing to do would be to have the Apache jenkins build fixed
>> and
>> treat
>>> build failures there seriously. I can help out with that if someone
> can
>>> guide me with access, modifying jenkins scripts, etc. Any other
> ideas?
>>>   Thinking of creating a umbrella ticket to track all current unit
> test
>>> failures and get them fixed. Or is there some other process that we
>> usually
>>> follow?
>>> 
>>> Thanks,
>>> Rohini
>>> 
>> 
> 
 
 
>>> 
>> 

Re: [jira] [Commented] (PIG-2963) Illustrate command and POPackageLite

[Alan Gates]

Send email to dev-unsubscr...@pig.apache.org

Alan.

On Oct 15, 2012, at 12:59 PM, Curtis Strite wrote:

> How do I remove myself from this distro?
> 
> 
> 
> Thanks,
> Curtis
> 
> 
> 
> On Mon, Oct 15, 2012 at 2:57 PM, Jonathan Coveney (JIRA) 
> wrote:
> 
>> 
>>[
>> https://issues.apache.org/jira/browse/PIG-2963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13476372#comment-13476372]
>> 
>> Jonathan Coveney commented on PIG-2963:
>> ---
>> 
>> Allan: it comes down to pestering a committer to reviewing and committing,
>> or just hoping they look at it.
>> 
>> Which I just did. Thanks for the fix, Cheolsoo. It's in!
>> 
>>> Illustrate command and POPackageLite
>>> 
>>> 
>>>Key: PIG-2963
>>>URL: https://issues.apache.org/jira/browse/PIG-2963
>>>Project: Pig
>>> Issue Type: Bug
>>>   Reporter: Allan Avendaño
>>>   Assignee: Cheolsoo Park
>>>   Priority: Critical
>>>Fix For: 0.11
>>> 
>>>Attachments: PIG-2963.patch
>>> 
>>> 
>>> While trying to execute a simple script like:
>>> A = LOAD 'test01' AS (f1:chararray,f2:int,f3:chararray);
>>> B = order A by f1;
>>> illustrate B;
>>> or
>>> C = foreach B generate f1, f2;
>>> illustrate C;
>>> I got the following exception:
>>> java.lang.RuntimeException: ReadOnceBag does not support getMemorySize
>> operation
>>>  at
>> org.apache.pig.data.ReadOnceBag.getMemorySize(ReadOnceBag.java:74)
>>>  at org.apache.pig.data.SizeUtil.getPigObjMemSize(SizeUtil.java:61)
>>>  at
>> org.apache.pig.data.DefaultTuple.getMemorySize(DefaultTuple.java:180)
>>>  at
>> org.apache.pig.pen.util.ExampleTuple.getMemorySize(ExampleTuple.java:97)
>>>  at
>> org.apache.pig.data.DefaultAbstractBag.getMemorySize(DefaultAbstractBag.java:148)
>>>  at
>> org.apache.pig.data.DefaultAbstractBag.markSpillableIfNecessary(DefaultAbstractBag.java:100)
>>>  at
>> org.apache.pig.data.DefaultAbstractBag.add(DefaultAbstractBag.java:92)
>>>  at org.apache.pig.pen.Illustrator.addData(Illustrator.java:116)
>>>  at
>> org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POPackageLite.illustratorMarkup(POPackageLite.java:227)
>>>  at
>> org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POPackageLite.getNext(POPackageLite.java:182)
>>>  at
>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapReduce$Reduce.processOnePackageOutput(PigGenericMapReduce.java:422)
>>>  at
>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapReduce$Reduce.reduce(PigGenericMapReduce.java:413)
>>>  at
>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapReduce$Reduce.reduce(PigGenericMapReduce.java:257)
>>>  at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:176)
>>>  at
>> org.apache.pig.pen.LocalMapReduceSimulator.launchPig(LocalMapReduceSimulator.java:235)
>>>  at
>> org.apache.pig.pen.ExampleGenerator.getData(ExampleGenerator.java:257)
>>>  at
>> org.apache.pig.pen.ExampleGenerator.getData(ExampleGenerator.java:238)
>>>  at
>> org.apache.pig.pen.LineageTrimmingVisitor.init(LineageTrimmingVisitor.java:103)
>>>  at
>> org.apache.pig.pen.LineageTrimmingVisitor.(LineageTrimmingVisitor.java:98)
>>>  at
>> org.apache.pig.pen.ExampleGenerator.getExamples(ExampleGenerator.java:166)
>>>  at org.apache.pig.PigServer.getExamples(PigServer.java:1180)
>>>  at
>> org.apache.pig.tools.grunt.GruntParser.processIllustrate(GruntParser.java:738)
>>>  at
>> org.apache.pig.tools.pigscript.parser.PigScriptParser.Illustrate(PigScriptParser.java:626)
>>>  at
>> org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:323)
>>>  at
>> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:193)
>>>  at
>> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:169)
>>>  at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:69)
>>>  at org.apache.pig.Main.run(Main.java:538)
>>>  at org.apache.pig.Main.main(Main.java:154)
>>>  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>  at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>>  at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>  at java.lang.reflect.Method.invoke(Method.java:597)
>>>  at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
>>> ==
>>> At log file, the following:
>>> Pig Stack Trace
>>> ---
>>> ERROR 2997: Encountered IOException. Exception
>>> java.io.IOException: Exception
>>>at org.apache.pig.PigServer.getExamples(PigServer.java:1186)
>>>at
>> org.apache.pig.tools.grunt.GruntPars

Re: Unit test failures

[Rohini Palaniswamy]

Me and Cheolsoo are kicking off a new run for the tests to check the
current failures. Will update the jira with those. Interested folks can
pick up the sub-task jiras.

Should we create another umbrella ticket just for the windows test
failures? HW and Microsoft folks are putting in effort to fix them. Not
sure if the plan is to get them into 0.11 or make it part of trunk. Either
way another umbrella ticket for that would be good.

Regards,
Rohini

On Mon, Oct 15, 2012 at 12:17 PM, Cheolsoo Park wrote:

> Now we have the umbrella ticket for test failures (unit/e2e):
> https://issues.apache.org/jira/browse/PIG-2972
>
> Please add new jiras as sub-tasks to it.
>
> Thanks,
> Cheolsoo
>
> On Mon, Oct 15, 2012 at 11:49 AM, Cheolsoo Park  >wrote:
>
> > >> Thinking of creating a umbrella ticket to track all current unit
> > test failures and get them fixed.
> >
> > Once we open a umbrella ticket for test failures, everyone should link
> > their jiras that are for failing tests to it. I know that there are
> already
> > many open jiras for failing tests. I have several jiras myself.
> >
> > Please let me know if you want me to open the umbrella ticket and link
> > existing tickets to it.
> >
> > Thanks,
> > Cheolsoo
> >
> >
> > On Mon, Oct 15, 2012 at 11:36 AM, Cheolsoo Park  >wrote:
> >
> >> This will be great! Please feel free to assign me bug fixes.
> >>
> >>
> >> On Mon, Oct 15, 2012 at 11:32 AM, Jonathan Coveney  >wrote:
> >>
> >>> I think this is an excellent idea, Rohini. We have resources to fix
> these
> >>> failures, so pointing them out should get them fixed. The error we've
> run
> >>> into is errors that are transient or related to a specific build on a
> >>> specific machine (and thus hard to replicate) vs. more serious errors.
> >>> Both
> >>> should be fixed, but it's hard to do the former when you have no
> control
> >>> over the CI. Throw in all the false fails from clover issues or
> >>> what-have-you and we're in a bad state. Cleaning that up would be
> >>> awesome.
> >>>
> >>> 2012/10/15 Julien Le Dem 
> >>>
> >>> > I think this is a great idea.
> >>> > I don't think we have access to the build right now.
> >>> > Daniel seemed to know more about that.
> >>> > Julien
> >>> >
> >>> > On Mon, Oct 15, 2012 at 11:15 AM, Rohini Palaniswamy <
> >>> > rohini.adi...@gmail.com> wrote:
> >>> >
> >>> > > Hi all,
> >>> > > Quite a few unit tests are broken in trunk/0.11. Now that 0.11
> is
> >>> > > branched it is more important to have them fixed and pass for
> >>> stability.
> >>> > > Also we need to figure out some way to have tests not broken. But
> >>> with
> >>> > the
> >>> > > time unit tests take it is difficult for everyone to run it. I
> think
> >>> the
> >>> > > first thing to do would be to have the Apache jenkins build fixed
> and
> >>> > treat
> >>> > > build failures there seriously. I can help out with that if someone
> >>> can
> >>> > > guide me with access, modifying jenkins scripts, etc. Any other
> >>> ideas?
> >>> > >Thinking of creating a umbrella ticket to track all current unit
> >>> test
> >>> > > failures and get them fixed. Or is there some other process that we
> >>> > usually
> >>> > > follow?
> >>> > >
> >>> > > Thanks,
> >>> > > Rohini
> >>> > >
> >>> >
> >>>
> >>
> >>
> >
>

[jira] [Updated] (PIG-2967) Fix Glob_local test failure for Pig E2E Test Framework

[Rohini Palaniswamy (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PIG-2967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rohini Palaniswamy updated PIG-2967:


Issue Type: Sub-task  (was: Test)
Parent: PIG-2972

> Fix Glob_local test failure for Pig E2E Test Framework
> --
>
> Key: PIG-2967
> URL: https://issues.apache.org/jira/browse/PIG-2967
> Project: Pig
>  Issue Type: Sub-task
>  Components: e2e harness
>Affects Versions: 0.10.1
>Reporter: Sushant Joshi
>Priority: Minor
> Fix For: 0.10.1
>
> Attachments: glob_local.patch
>
>
> The Glob_3_local, Glob_4_local, Glob_5_local E2E tests fails due check sum 
> mismatch with benchmark data.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Re: [jira] [Commented] (PIG-2963) Illustrate command and POPackageLite

[Curtis Strite]

How do I remove myself from this distro?



Thanks,
Curtis



On Mon, Oct 15, 2012 at 2:57 PM, Jonathan Coveney (JIRA) wrote:

>
> [
> https://issues.apache.org/jira/browse/PIG-2963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13476372#comment-13476372]
>
> Jonathan Coveney commented on PIG-2963:
> ---
>
> Allan: it comes down to pestering a committer to reviewing and committing,
> or just hoping they look at it.
>
> Which I just did. Thanks for the fix, Cheolsoo. It's in!
>
> > Illustrate command and POPackageLite
> > 
> >
> > Key: PIG-2963
> > URL: https://issues.apache.org/jira/browse/PIG-2963
> > Project: Pig
> >  Issue Type: Bug
> >Reporter: Allan Avendaño
> >Assignee: Cheolsoo Park
> >Priority: Critical
> > Fix For: 0.11
> >
> > Attachments: PIG-2963.patch
> >
> >
> > While trying to execute a simple script like:
> > A = LOAD 'test01' AS (f1:chararray,f2:int,f3:chararray);
> > B = order A by f1;
> > illustrate B;
> > or
> > C = foreach B generate f1, f2;
> > illustrate C;
> > I got the following exception:
> > java.lang.RuntimeException: ReadOnceBag does not support getMemorySize
> operation
> >   at
> org.apache.pig.data.ReadOnceBag.getMemorySize(ReadOnceBag.java:74)
> >   at org.apache.pig.data.SizeUtil.getPigObjMemSize(SizeUtil.java:61)
> >   at
> org.apache.pig.data.DefaultTuple.getMemorySize(DefaultTuple.java:180)
> >   at
> org.apache.pig.pen.util.ExampleTuple.getMemorySize(ExampleTuple.java:97)
> >   at
> org.apache.pig.data.DefaultAbstractBag.getMemorySize(DefaultAbstractBag.java:148)
> >   at
> org.apache.pig.data.DefaultAbstractBag.markSpillableIfNecessary(DefaultAbstractBag.java:100)
> >   at
> org.apache.pig.data.DefaultAbstractBag.add(DefaultAbstractBag.java:92)
> >   at org.apache.pig.pen.Illustrator.addData(Illustrator.java:116)
> >   at
> org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POPackageLite.illustratorMarkup(POPackageLite.java:227)
> >   at
> org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POPackageLite.getNext(POPackageLite.java:182)
> >   at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapReduce$Reduce.processOnePackageOutput(PigGenericMapReduce.java:422)
> >   at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapReduce$Reduce.reduce(PigGenericMapReduce.java:413)
> >   at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapReduce$Reduce.reduce(PigGenericMapReduce.java:257)
> >   at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:176)
> >   at
> org.apache.pig.pen.LocalMapReduceSimulator.launchPig(LocalMapReduceSimulator.java:235)
> >   at
> org.apache.pig.pen.ExampleGenerator.getData(ExampleGenerator.java:257)
> >   at
> org.apache.pig.pen.ExampleGenerator.getData(ExampleGenerator.java:238)
> >   at
> org.apache.pig.pen.LineageTrimmingVisitor.init(LineageTrimmingVisitor.java:103)
> >   at
> org.apache.pig.pen.LineageTrimmingVisitor.(LineageTrimmingVisitor.java:98)
> >   at
> org.apache.pig.pen.ExampleGenerator.getExamples(ExampleGenerator.java:166)
> >   at org.apache.pig.PigServer.getExamples(PigServer.java:1180)
> >   at
> org.apache.pig.tools.grunt.GruntParser.processIllustrate(GruntParser.java:738)
> >   at
> org.apache.pig.tools.pigscript.parser.PigScriptParser.Illustrate(PigScriptParser.java:626)
> >   at
> org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:323)
> >   at
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:193)
> >   at
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:169)
> >   at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:69)
> >   at org.apache.pig.Main.run(Main.java:538)
> >   at org.apache.pig.Main.main(Main.java:154)
> >   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >   at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> >   at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >   at java.lang.reflect.Method.invoke(Method.java:597)
> >   at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
> > ==
> > At log file, the following:
> > Pig Stack Trace
> > ---
> > ERROR 2997: Encountered IOException. Exception
> > java.io.IOException: Exception
> > at org.apache.pig.PigServer.getExamples(PigServer.java:1186)
> > at
> org.apache.pig.tools.grunt.GruntParser.processIllustrate(GruntParser.java:738)
> > at
> org.apache.pig.tools.pigscript.parser.PigScriptParser.Illustrate(PigScr

[jira] [Updated] (PIG-2846) Can we skip hcat related e2e when hcat is not installed?

[Cheolsoo Park (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PIG-2846?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Cheolsoo Park updated PIG-2846:
---

Issue Type: Sub-task  (was: Test)
Parent: PIG-2972

> Can we skip hcat related e2e when hcat is not installed?
> 
>
> Key: PIG-2846
> URL: https://issues.apache.org/jira/browse/PIG-2846
> Project: Pig
>  Issue Type: Sub-task
>Reporter: Koji Noguchi
>Priority: Trivial
> Attachments: pig-2846-trunk-v1.txt
>
>
> Trying pig e2e for the first time, I see couple of the tests 
> (HCatDDL_1,HCatDDL_2 and Jython_Command_1) failing with 
> bq. java.io.IOException: Cannot run program /usr/local/hcat/bin/hcat:
> bq. java.io.IOException: error=2, No such file or directory
> Is it ok to change the test_harness to skip these tests when hcat does not 
> exist?

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (PIG-2963) Illustrate command and POPackageLite

[Jonathan Coveney (JIRA)]

[ 
https://issues.apache.org/jira/browse/PIG-2963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13476372#comment-13476372
 ] 

Jonathan Coveney commented on PIG-2963:
---

Allan: it comes down to pestering a committer to reviewing and committing, or 
just hoping they look at it.

Which I just did. Thanks for the fix, Cheolsoo. It's in!

> Illustrate command and POPackageLite
> 
>
> Key: PIG-2963
> URL: https://issues.apache.org/jira/browse/PIG-2963
> Project: Pig
>  Issue Type: Bug
>Reporter: Allan Avendaño
>Assignee: Cheolsoo Park
>Priority: Critical
> Fix For: 0.11
>
> Attachments: PIG-2963.patch
>
>
> While trying to execute a simple script like:
> A = LOAD 'test01' AS (f1:chararray,f2:int,f3:chararray);
> B = order A by f1;
> illustrate B;
> or 
> C = foreach B generate f1, f2; 
> illustrate C;
> I got the following exception:
> java.lang.RuntimeException: ReadOnceBag does not support getMemorySize 
> operation
>   at org.apache.pig.data.ReadOnceBag.getMemorySize(ReadOnceBag.java:74)
>   at org.apache.pig.data.SizeUtil.getPigObjMemSize(SizeUtil.java:61)
>   at org.apache.pig.data.DefaultTuple.getMemorySize(DefaultTuple.java:180)
>   at 
> org.apache.pig.pen.util.ExampleTuple.getMemorySize(ExampleTuple.java:97)
>   at 
> org.apache.pig.data.DefaultAbstractBag.getMemorySize(DefaultAbstractBag.java:148)
>   at 
> org.apache.pig.data.DefaultAbstractBag.markSpillableIfNecessary(DefaultAbstractBag.java:100)
>   at 
> org.apache.pig.data.DefaultAbstractBag.add(DefaultAbstractBag.java:92)
>   at org.apache.pig.pen.Illustrator.addData(Illustrator.java:116)
>   at 
> org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POPackageLite.illustratorMarkup(POPackageLite.java:227)
>   at 
> org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POPackageLite.getNext(POPackageLite.java:182)
>   at 
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapReduce$Reduce.processOnePackageOutput(PigGenericMapReduce.java:422)
>   at 
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapReduce$Reduce.reduce(PigGenericMapReduce.java:413)
>   at 
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapReduce$Reduce.reduce(PigGenericMapReduce.java:257)
>   at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:176)
>   at 
> org.apache.pig.pen.LocalMapReduceSimulator.launchPig(LocalMapReduceSimulator.java:235)
>   at 
> org.apache.pig.pen.ExampleGenerator.getData(ExampleGenerator.java:257)
>   at 
> org.apache.pig.pen.ExampleGenerator.getData(ExampleGenerator.java:238)
>   at 
> org.apache.pig.pen.LineageTrimmingVisitor.init(LineageTrimmingVisitor.java:103)
>   at 
> org.apache.pig.pen.LineageTrimmingVisitor.(LineageTrimmingVisitor.java:98)
>   at 
> org.apache.pig.pen.ExampleGenerator.getExamples(ExampleGenerator.java:166)
>   at org.apache.pig.PigServer.getExamples(PigServer.java:1180)
>   at 
> org.apache.pig.tools.grunt.GruntParser.processIllustrate(GruntParser.java:738)
>   at 
> org.apache.pig.tools.pigscript.parser.PigScriptParser.Illustrate(PigScriptParser.java:626)
>   at 
> org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:323)
>   at 
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:193)
>   at 
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:169)
>   at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:69)
>   at org.apache.pig.Main.run(Main.java:538)
>   at org.apache.pig.Main.main(Main.java:154)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>   at java.lang.reflect.Method.invoke(Method.java:597)
>   at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
> ==
> At log file, the following:
> Pig Stack Trace
> ---
> ERROR 2997: Encountered IOException. Exception
> java.io.IOException: Exception
> at org.apache.pig.PigServer.getExamples(PigServer.java:1186)
> at 
> org.apache.pig.tools.grunt.GruntParser.processIllustrate(GruntParser.java:738)
> at 
> org.apache.pig.tools.pigscript.parser.PigScriptParser.Illustrate(PigScriptParser.java:626)
> at 
> org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:323)
> at 
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:193)
> at 
> org.

[jira] [Updated] (PIG-2963) Illustrate command and POPackageLite

[Jonathan Coveney (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PIG-2963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jonathan Coveney updated PIG-2963:
--

Resolution: Fixed
Status: Resolved  (was: Patch Available)

> Illustrate command and POPackageLite
> 
>
> Key: PIG-2963
> URL: https://issues.apache.org/jira/browse/PIG-2963
> Project: Pig
>  Issue Type: Bug
>Reporter: Allan Avendaño
>Assignee: Cheolsoo Park
>Priority: Critical
> Fix For: 0.11
>
> Attachments: PIG-2963.patch
>
>
> While trying to execute a simple script like:
> A = LOAD 'test01' AS (f1:chararray,f2:int,f3:chararray);
> B = order A by f1;
> illustrate B;
> or 
> C = foreach B generate f1, f2; 
> illustrate C;
> I got the following exception:
> java.lang.RuntimeException: ReadOnceBag does not support getMemorySize 
> operation
>   at org.apache.pig.data.ReadOnceBag.getMemorySize(ReadOnceBag.java:74)
>   at org.apache.pig.data.SizeUtil.getPigObjMemSize(SizeUtil.java:61)
>   at org.apache.pig.data.DefaultTuple.getMemorySize(DefaultTuple.java:180)
>   at 
> org.apache.pig.pen.util.ExampleTuple.getMemorySize(ExampleTuple.java:97)
>   at 
> org.apache.pig.data.DefaultAbstractBag.getMemorySize(DefaultAbstractBag.java:148)
>   at 
> org.apache.pig.data.DefaultAbstractBag.markSpillableIfNecessary(DefaultAbstractBag.java:100)
>   at 
> org.apache.pig.data.DefaultAbstractBag.add(DefaultAbstractBag.java:92)
>   at org.apache.pig.pen.Illustrator.addData(Illustrator.java:116)
>   at 
> org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POPackageLite.illustratorMarkup(POPackageLite.java:227)
>   at 
> org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POPackageLite.getNext(POPackageLite.java:182)
>   at 
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapReduce$Reduce.processOnePackageOutput(PigGenericMapReduce.java:422)
>   at 
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapReduce$Reduce.reduce(PigGenericMapReduce.java:413)
>   at 
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapReduce$Reduce.reduce(PigGenericMapReduce.java:257)
>   at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:176)
>   at 
> org.apache.pig.pen.LocalMapReduceSimulator.launchPig(LocalMapReduceSimulator.java:235)
>   at 
> org.apache.pig.pen.ExampleGenerator.getData(ExampleGenerator.java:257)
>   at 
> org.apache.pig.pen.ExampleGenerator.getData(ExampleGenerator.java:238)
>   at 
> org.apache.pig.pen.LineageTrimmingVisitor.init(LineageTrimmingVisitor.java:103)
>   at 
> org.apache.pig.pen.LineageTrimmingVisitor.(LineageTrimmingVisitor.java:98)
>   at 
> org.apache.pig.pen.ExampleGenerator.getExamples(ExampleGenerator.java:166)
>   at org.apache.pig.PigServer.getExamples(PigServer.java:1180)
>   at 
> org.apache.pig.tools.grunt.GruntParser.processIllustrate(GruntParser.java:738)
>   at 
> org.apache.pig.tools.pigscript.parser.PigScriptParser.Illustrate(PigScriptParser.java:626)
>   at 
> org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:323)
>   at 
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:193)
>   at 
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:169)
>   at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:69)
>   at org.apache.pig.Main.run(Main.java:538)
>   at org.apache.pig.Main.main(Main.java:154)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>   at java.lang.reflect.Method.invoke(Method.java:597)
>   at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
> ==
> At log file, the following:
> Pig Stack Trace
> ---
> ERROR 2997: Encountered IOException. Exception
> java.io.IOException: Exception
> at org.apache.pig.PigServer.getExamples(PigServer.java:1186)
> at 
> org.apache.pig.tools.grunt.GruntParser.processIllustrate(GruntParser.java:738)
> at 
> org.apache.pig.tools.pigscript.parser.PigScriptParser.Illustrate(PigScriptParser.java:626)
> at 
> org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:323)
> at 
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:193)
> at 
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:169)
> at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:69)
> at org.apache.p

[jira] [Created] (PIG-2974) StreamingLocal_11 e2e test hangs

[Rohini Palaniswamy (JIRA)]

Rohini Palaniswamy created PIG-2974:
---

 Summary: StreamingLocal_11 e2e test hangs
 Key: PIG-2974
 URL: https://issues.apache.org/jira/browse/PIG-2974
 Project: Pig
  Issue Type: Sub-task
Affects Versions: 0.11
Reporter: Rohini Palaniswamy




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (PIG-2973) TestStreaming test times out

[Rohini Palaniswamy (JIRA)]

Rohini Palaniswamy created PIG-2973:
---

 Summary: TestStreaming test times out
 Key: PIG-2973
 URL: https://issues.apache.org/jira/browse/PIG-2973
 Project: Pig
  Issue Type: Sub-task
Affects Versions: 0.11
Reporter: Rohini Palaniswamy




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Re: Unit test failures

[Cheolsoo Park]

Now we have the umbrella ticket for test failures (unit/e2e):
https://issues.apache.org/jira/browse/PIG-2972

Please add new jiras as sub-tasks to it.

Thanks,
Cheolsoo

On Mon, Oct 15, 2012 at 11:49 AM, Cheolsoo Park wrote:

> >> Thinking of creating a umbrella ticket to track all current unit
> test failures and get them fixed.
>
> Once we open a umbrella ticket for test failures, everyone should link
> their jiras that are for failing tests to it. I know that there are already
> many open jiras for failing tests. I have several jiras myself.
>
> Please let me know if you want me to open the umbrella ticket and link
> existing tickets to it.
>
> Thanks,
> Cheolsoo
>
>
> On Mon, Oct 15, 2012 at 11:36 AM, Cheolsoo Park wrote:
>
>> This will be great! Please feel free to assign me bug fixes.
>>
>>
>> On Mon, Oct 15, 2012 at 11:32 AM, Jonathan Coveney wrote:
>>
>>> I think this is an excellent idea, Rohini. We have resources to fix these
>>> failures, so pointing them out should get them fixed. The error we've run
>>> into is errors that are transient or related to a specific build on a
>>> specific machine (and thus hard to replicate) vs. more serious errors.
>>> Both
>>> should be fixed, but it's hard to do the former when you have no control
>>> over the CI. Throw in all the false fails from clover issues or
>>> what-have-you and we're in a bad state. Cleaning that up would be
>>> awesome.
>>>
>>> 2012/10/15 Julien Le Dem 
>>>
>>> > I think this is a great idea.
>>> > I don't think we have access to the build right now.
>>> > Daniel seemed to know more about that.
>>> > Julien
>>> >
>>> > On Mon, Oct 15, 2012 at 11:15 AM, Rohini Palaniswamy <
>>> > rohini.adi...@gmail.com> wrote:
>>> >
>>> > > Hi all,
>>> > > Quite a few unit tests are broken in trunk/0.11. Now that 0.11 is
>>> > > branched it is more important to have them fixed and pass for
>>> stability.
>>> > > Also we need to figure out some way to have tests not broken. But
>>> with
>>> > the
>>> > > time unit tests take it is difficult for everyone to run it. I think
>>> the
>>> > > first thing to do would be to have the Apache jenkins build fixed and
>>> > treat
>>> > > build failures there seriously. I can help out with that if someone
>>> can
>>> > > guide me with access, modifying jenkins scripts, etc. Any other
>>> ideas?
>>> > >Thinking of creating a umbrella ticket to track all current unit
>>> test
>>> > > failures and get them fixed. Or is there some other process that we
>>> > usually
>>> > > follow?
>>> > >
>>> > > Thanks,
>>> > > Rohini
>>> > >
>>> >
>>>
>>
>>
>

[jira] [Updated] (PIG-2908) Fix unit tests to work with jdk7

[Cheolsoo Park (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PIG-2908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Cheolsoo Park updated PIG-2908:
---

Issue Type: Sub-task  (was: Test)
Parent: PIG-2972

> Fix unit tests to work with jdk7
> 
>
> Key: PIG-2908
> URL: https://issues.apache.org/jira/browse/PIG-2908
> Project: Pig
>  Issue Type: Sub-task
>Affects Versions: 0.10.0
>Reporter: Rohini Palaniswamy
>Assignee: Rohini Palaniswamy
>Priority: Minor
> Fix For: 0.11, 0.10.1
>
> Attachments: PIG-2908-branch10-1.patch, PIG-2908-branch10.patch, 
> PIG-2908-trunk-1.patch, PIG-2908-trunk.patch
>
>
> Fix unit tests that fail when compiling with jdk7 as they depend on the order 
> of tests run - TestGFCross, TestJsonLoaderStorage, TestLogToPhyCompiler, 
> TestMRCompiler, TestMergeJoinOuter and TestNewPlanLogToPhyTranslationVisitor. 
> jdk7 returns methods in a different order during reflection.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (PIG-2972) Umbrella ticket for test failures in 0.11/trunk

[Cheolsoo Park (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PIG-2972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Cheolsoo Park updated PIG-2972:
---

Summary: Umbrella ticket for test failures in 0.11/trunk  (was: Umbrella 
ticket for unit test failures in 0.11/trunk)

> Umbrella ticket for test failures in 0.11/trunk
> ---
>
> Key: PIG-2972
> URL: https://issues.apache.org/jira/browse/PIG-2972
> Project: Pig
>  Issue Type: Bug
>Affects Versions: 0.11
>Reporter: Rohini Palaniswamy
> Fix For: 0.11
>
>


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (PIG-2928) Fix e2e test failures in trunk: FilterBoolean_23/24

[Cheolsoo Park (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PIG-2928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Cheolsoo Park updated PIG-2928:
---

Parent Issue: PIG-2972  (was: PIG-1314)

> Fix e2e test failures in trunk: FilterBoolean_23/24
> ---
>
> Key: PIG-2928
> URL: https://issues.apache.org/jira/browse/PIG-2928
> Project: Pig
>  Issue Type: Sub-task
>Affects Versions: 0.10.0
>Reporter: Cheolsoo Park
>Assignee: Cheolsoo Park
> Fix For: 0.11
>
> Attachments: PIG-2928.patch
>
>
> To reproduce the failure, please run
> {code}
> ant -Dhadoopversion=20 ... -Dtest-e2e-deploy-local
> ant -Dhadoopversion=20 ... -Dtest-e2e-local -Dtests.to.run="-t 
> FilterBoolean_23"
> {code}
> This is a regression from PIG-1314.
> PIG-1314 reverted the following change made by PIG-2593 that had made it 
> possible to filter by a boolean value.
> {code}
> --- src/org/apache/pig/parser/QueryParser.g
> +++ src/org/apache/pig/parser/QueryParser.g
> @@ -391,13 +391,16 @@ or_cond : and_cond  ( OR^ and_cond )*
>  and_cond : unary_cond ( AND^ unary_cond )*
>  ;
>  
> -unary_cond : LEFT_PAREN! cond RIGHT_PAREN!
> -   | not_cond
> -   | expr rel_op^ expr
> +unary_cond : expr rel_op^ expr
> +   | LEFT_PAREN! cond RIGHT_PAREN!
> +   | not_cond   
> | func_eval
> | null_check_cond
> +   | bool_cond   
>  ;
>  
> +bool_cond: expr -> ^(BOOL_COND expr);
> +
>  not_cond : NOT^ unary_cond
>  ;
> @@ -784,6 +787,7 @@ eid : rel_str_op
>  | TRUE
>  | FALSE
>  | REALIAS
> +| BOOL_COND
>  ;
> {code}
> This makes FilterBoolean_23/24 in e2e test fail.
> If this change is intended, we should disable FilterBoolean_23/24.
> If not, we should put the reverted change back to the grammar.
> Thanks!

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (PIG-2737) TestIndexedStorage is failing, should be refactored

[Cheolsoo Park (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PIG-2737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Cheolsoo Park updated PIG-2737:
---

Issue Type: Sub-task  (was: Bug)
Parent: PIG-2972

> TestIndexedStorage is failing, should be refactored
> ---
>
> Key: PIG-2737
> URL: https://issues.apache.org/jira/browse/PIG-2737
> Project: Pig
>  Issue Type: Sub-task
>Reporter: Jonathan Coveney
>Assignee: Jonathan Coveney
> Fix For: 0.11
>
> Attachments: PIG-2737-0.patch, PIG-2737-1.patch
>
>
> TestIndexedStorage is failing because it is not properly creating the output 
> directory. I've refactored it to use real temp directories instead of 
> something in the root.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (PIG-2913) org.apache.pig.test.TestPigServerWithMacros fails sometimes because it picks up previous minicluster configuration file

[Cheolsoo Park (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PIG-2913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Cheolsoo Park updated PIG-2913:
---

Issue Type: Sub-task  (was: Bug)
Parent: PIG-2972

> org.apache.pig.test.TestPigServerWithMacros fails sometimes because it picks 
> up previous minicluster configuration file
> ---
>
> Key: PIG-2913
> URL: https://issues.apache.org/jira/browse/PIG-2913
> Project: Pig
>  Issue Type: Sub-task
>Reporter: Julien Le Dem
>Assignee: Julien Le Dem
> Attachments: PIG-2913.patch
>
>


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (PIG-2926) TestPoissonSampleLoader failing on rhel environment

[Cheolsoo Park (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PIG-2926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Cheolsoo Park updated PIG-2926:
---

Issue Type: Sub-task  (was: Bug)
Parent: PIG-2972

> TestPoissonSampleLoader failing on rhel environment
> ---
>
> Key: PIG-2926
> URL: https://issues.apache.org/jira/browse/PIG-2926
> Project: Pig
>  Issue Type: Sub-task
>Reporter: Koji Noguchi
>Priority: Minor
>
> Testing on rhel environment, TestPoissonSampleLoader fails with 
> {noformat}
> Testcase: testNumSamples took 22.077 sec
> FAILED
> expected:<47> but was:<42>
> junit.framework.AssertionFailedError: expected:<47> but was:<42>
> at 
> org.apache.pig.test.TestPoissonSampleLoader.testNumSamples(TestPoissonSampleLoader.java:125)
> {noformat}
> From 
> {noformat}
> 124 count = testNumSamples("0.0001", "100");
> 125 assertEquals(count, 42);
> {noformat}
> This runs fine on my mac environment.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (PIG-2971) Add new parameter to specify the streaming environment

[Jonathan Coveney (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PIG-2971?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jonathan Coveney updated PIG-2971:
--

Attachment: PIG-2971-0.patch

Here is a patch with tests. Passes test-commit.

> Add new parameter to specify the streaming environment
> --
>
> Key: PIG-2971
> URL: https://issues.apache.org/jira/browse/PIG-2971
> Project: Pig
>  Issue Type: Bug
>Reporter: Jonathan Coveney
>Assignee: Jonathan Coveney
> Fix For: 0.11
>
> Attachments: PIG-2971-0.patch
>
>
> The current proposal in PIG-2925 is a bit arbitrary. How do we set what is 
> and is not too big? This instead let's the user set what they need.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (PIG-2966) Test failures on CentOS 6 because MALLOC_ARENA_MAX is not set

[Cheolsoo Park (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PIG-2966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Cheolsoo Park updated PIG-2966:
---

Issue Type: Sub-task  (was: Bug)
Parent: PIG-2972

> Test failures on CentOS 6 because MALLOC_ARENA_MAX is not set
> -
>
> Key: PIG-2966
> URL: https://issues.apache.org/jira/browse/PIG-2966
> Project: Pig
>  Issue Type: Sub-task
>  Components: build, piggybank
>Reporter: Cheolsoo Park
>Assignee: Cheolsoo Park
>  Labels: test
> Attachments: PIG-2966-2.patch, PIG-2966.patch
>
>
> To reproduce, please run on CentOS 6:
> {code}
> ant clean compile-test jar-withouthadoop -Dhadoopversion=23
> cd contrib/piggybank/java
> ant clean test -Dhadoopversion=23
> {code}
> Three tests are failing:
> {code}
> [junit] Test 
> org.apache.pig.piggybank.test.evaluation.string.TestLookupInFiles FAILED
> [junit] Test org.apache.pig.piggybank.test.storage.TestDBStorage FAILED
> [junit] Test org.apache.pig.piggybank.test.storage.TestMultiStorage FAILED
> {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (PIG-2966) Test failures on CentOS 6 because MALLOC_ARENA_MAX is not set

[Cheolsoo Park (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PIG-2966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Cheolsoo Park updated PIG-2966:
---

Component/s: build
Description: 
To reproduce, please run on CentOS 6:
{code}
ant clean compile-test jar-withouthadoop -Dhadoopversion=23
cd contrib/piggybank/java
ant clean test -Dhadoopversion=23
{code}

Three tests are failing:
{code}
[junit] Test org.apache.pig.piggybank.test.evaluation.string.TestLookupInFiles 
FAILED
[junit] Test org.apache.pig.piggybank.test.storage.TestDBStorage FAILED
[junit] Test org.apache.pig.piggybank.test.storage.TestMultiStorage FAILED
{code}

  was:
To reproduce, please run:
{code}
ant clean compile-test jar-withouthadoop -Dhadoopversion=23
cd contrib/piggybank/java
ant clean test -Dhadoopversion=23
{code}

Three tests are failing:
{code}
[junit] Test org.apache.pig.piggybank.test.evaluation.string.TestLookupInFiles 
FAILED
[junit] Test org.apache.pig.piggybank.test.storage.TestDBStorage FAILED
[junit] Test org.apache.pig.piggybank.test.storage.TestMultiStorage FAILED
{code}

Summary: Test failures on CentOS 6 because MALLOC_ARENA_MAX is not set  
(was: Piggybank test failures on CentOS 6: TestLookupInFiles, TestDBStorage, 
and TestMultiStorage)

> Test failures on CentOS 6 because MALLOC_ARENA_MAX is not set
> -
>
> Key: PIG-2966
> URL: https://issues.apache.org/jira/browse/PIG-2966
> Project: Pig
>  Issue Type: Bug
>  Components: build, piggybank
>Reporter: Cheolsoo Park
>Assignee: Cheolsoo Park
>  Labels: test
> Attachments: PIG-2966-2.patch, PIG-2966.patch
>
>
> To reproduce, please run on CentOS 6:
> {code}
> ant clean compile-test jar-withouthadoop -Dhadoopversion=23
> cd contrib/piggybank/java
> ant clean test -Dhadoopversion=23
> {code}
> Three tests are failing:
> {code}
> [junit] Test 
> org.apache.pig.piggybank.test.evaluation.string.TestLookupInFiles FAILED
> [junit] Test org.apache.pig.piggybank.test.storage.TestDBStorage FAILED
> [junit] Test org.apache.pig.piggybank.test.storage.TestMultiStorage FAILED
> {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Re: Pig 0.11

[Alan Gates]

At this point no one has taken on release documentation for 0.11.

Alan.

On Oct 15, 2012, at 11:49 AM, Olga Natkovich wrote:

> Thanks!
>  
> Are you talking about items 15 and 16 on the How To Release.Publish  page? 
>  
> Also, who is doing release documentation these days? I can help with that as 
> well. I would also be happy to roll the release if you guys need help with 
> that.
>  
> Olga
> 
> 
> 
> From: Dmitriy Ryaboy 
> To: "dev@pig.apache.org"  
> Cc: "dev@pig.apache.org"  
> Sent: Friday, October 12, 2012 5:59 PM
> Subject: Re: Pig 0.11
> 
> Thanks Olga and welcome back! 
> I know there's some process for linking jiras to releases, but I'm not sure 
> what that is. If you could explain and maybe cover a portion of that work, 
> that'd be super helpful. And reviews, of course. 
> 
> On Oct 12, 2012, at 2:06 PM, Olga Natkovich  wrote:
> 
>> Dmitry, I would be happy to help with the release process. Want to get back 
>> into this now that I am back at work. Let me know what you would like me to 
>> do.
>>   
>> Olga
>> 
>> 
>> 
>> 
>> From: Dmitriy Ryaboy 
>> To: dev@pig.apache.org 
>> Cc: billgra...@gmail.com 
>> Sent: Thursday, October 11, 2012 2:44 PM
>> Subject: Re: Pig 0.11
>> 
>> Ok I will branch 0.11 tomorrow morning unless someone objects.
>> From then on, committers should be careful to commit bug fixes to both
>> 0.11 branch and trunk; minor polish can go into the branch, but whole
>> new features should not (we can discuss on the list if something is in
>> the gray area).
>> 
>> D
>> 
>> On Thu, Oct 11, 2012 at 2:16 PM, Gianmarco De Francisci Morales
>>  wrote:
>>> I added it as a dependency as it has already its own Jira.
>>> I hope it is OK.
>>> 
>>> Cheers,
>>> --
>>> Gianmarco
>>> 
>>> 
>>> 
>>> On Wed, Oct 10, 2012 at 11:23 PM, Bill Graham  wrote:
>>> 
 +1 for me.
 
 There's https://issues.apache.org/jira/browse/PIG-2756 which tracks a few
 documentation issues that should block Pig 0.11, but they can also be done
 on the trunk and merged to the branch. Gianmarco, you can add a rank
 subtask there to serve as a reminder.
 
 
 On Wed, Oct 10, 2012 at 11:03 PM, Gianmarco De Francisci Morales <
 g...@apache.org> wrote:
 
> We are missing some documentation on the RANK but I guess we could add
 that
> to the branch and trunk in parallel.
> All the patches I was keeping an eye on are in.
> 
> So +1 for me.
> --
> Gianmarco
> 
> 
> 
> On Wed, Oct 10, 2012 at 5:31 PM, Jonathan Coveney > wrote:
> 
>> I think all of the major patches are in, no? Now it's just bug testing?
>> Just wanted to touch base on where we are at with this.
>> 
> 
 
 
 
 --
 *Note that I'm no longer using my Yahoo! email address. Please email me at
 billgra...@gmail.com going forward.*

Re: Pig 0.11

[Olga Natkovich]

Thanks - happy to be back :)




From: Jonathan Coveney 
To: dev@pig.apache.org 
Sent: Sunday, October 14, 2012 1:36 PM
Subject: Re: Pig 0.11

Glad to hear you're back on Pig, Olga :)

2012/10/12 Dmitriy Ryaboy 

> Thanks Olga and welcome back!
> I know there's some process for linking jiras to releases, but I'm not
> sure what that is. If you could explain and maybe cover a portion of that
> work, that'd be super helpful. And reviews, of course.
>
> On Oct 12, 2012, at 2:06 PM, Olga Natkovich  wrote:
>
> > Dmitry, I would be happy to help with the release process. Want to get
> back into this now that I am back at work. Let me know what you would like
> me to do.
> >
> > Olga
> >
> >
> >
> > 
> > From: Dmitriy Ryaboy 
> > To: dev@pig.apache.org
> > Cc: billgra...@gmail.com
> > Sent: Thursday, October 11, 2012 2:44 PM
> > Subject: Re: Pig 0.11
> >
> > Ok I will branch 0.11 tomorrow morning unless someone objects.
> > From then on, committers should be careful to commit bug fixes to both
> > 0.11 branch and trunk; minor polish can go into the branch, but whole
> > new features should not (we can discuss on the list if something is in
> > the gray area).
> >
> > D
> >
> > On Thu, Oct 11, 2012 at 2:16 PM, Gianmarco De Francisci Morales
> >  wrote:
> >> I added it as a dependency as it has already its own Jira.
> >> I hope it is OK.
> >>
> >> Cheers,
> >> --
> >> Gianmarco
> >>
> >>
> >>
> >> On Wed, Oct 10, 2012 at 11:23 PM, Bill Graham 
> wrote:
> >>
> >>> +1 for me.
> >>>
> >>> There's https://issues.apache.org/jira/browse/PIG-2756 which tracks a
> few
> >>> documentation issues that should block Pig 0.11, but they can also be
> done
> >>> on the trunk and merged to the branch. Gianmarco, you can add a rank
> >>> subtask there to serve as a reminder.
> >>>
> >>>
> >>> On Wed, Oct 10, 2012 at 11:03 PM, Gianmarco De Francisci Morales <
> >>> g...@apache.org> wrote:
> >>>
>  We are missing some documentation on the RANK but I guess we could add
> >>> that
>  to the branch and trunk in parallel.
>  All the patches I was keeping an eye on are in.
> 
>  So +1 for me.
>  --
>  Gianmarco
> 
> 
> 
>  On Wed, Oct 10, 2012 at 5:31 PM, Jonathan Coveney  > wrote:
> 
> > I think all of the major patches are in, no? Now it's just bug
> testing?
> > Just wanted to touch base on where we are at with this.
> >
> 
> >>>
> >>>
> >>>
> >>> --
> >>> *Note that I'm no longer using my Yahoo! email address. Please email
> me at
> >>> billgra...@gmail.com going forward.*
>

Re: Pig 0.11

[Olga Natkovich]

Thanks!
 
Are you talking about items 15 and 16 on the How To Release.Publish  page? 
 
Also, who is doing release documentation these days? I can help with that as 
well. I would also be happy to roll the release if you guys need help with that.
 
Olga



From: Dmitriy Ryaboy 
To: "dev@pig.apache.org"  
Cc: "dev@pig.apache.org"  
Sent: Friday, October 12, 2012 5:59 PM
Subject: Re: Pig 0.11

Thanks Olga and welcome back! 
I know there's some process for linking jiras to releases, but I'm not sure 
what that is. If you could explain and maybe cover a portion of that work, 
that'd be super helpful. And reviews, of course. 

On Oct 12, 2012, at 2:06 PM, Olga Natkovich  wrote:

> Dmitry, I would be happy to help with the release process. Want to get back 
> into this now that I am back at work. Let me know what you would like me to 
> do.
>  
> Olga
> 
> 
> 
> 
> From: Dmitriy Ryaboy 
> To: dev@pig.apache.org 
> Cc: billgra...@gmail.com 
> Sent: Thursday, October 11, 2012 2:44 PM
> Subject: Re: Pig 0.11
> 
> Ok I will branch 0.11 tomorrow morning unless someone objects.
> From then on, committers should be careful to commit bug fixes to both
> 0.11 branch and trunk; minor polish can go into the branch, but whole
> new features should not (we can discuss on the list if something is in
> the gray area).
> 
> D
> 
> On Thu, Oct 11, 2012 at 2:16 PM, Gianmarco De Francisci Morales
>  wrote:
>> I added it as a dependency as it has already its own Jira.
>> I hope it is OK.
>> 
>> Cheers,
>> --
>> Gianmarco
>> 
>> 
>> 
>> On Wed, Oct 10, 2012 at 11:23 PM, Bill Graham  wrote:
>> 
>>> +1 for me.
>>> 
>>> There's https://issues.apache.org/jira/browse/PIG-2756 which tracks a few
>>> documentation issues that should block Pig 0.11, but they can also be done
>>> on the trunk and merged to the branch. Gianmarco, you can add a rank
>>> subtask there to serve as a reminder.
>>> 
>>> 
>>> On Wed, Oct 10, 2012 at 11:03 PM, Gianmarco De Francisci Morales <
>>> g...@apache.org> wrote:
>>> 
 We are missing some documentation on the RANK but I guess we could add
>>> that
 to the branch and trunk in parallel.
 All the patches I was keeping an eye on are in.
 
 So +1 for me.
 --
 Gianmarco
 
 
 
 On Wed, Oct 10, 2012 at 5:31 PM, Jonathan Coveney  wrote:
 
> I think all of the major patches are in, no? Now it's just bug testing?
> Just wanted to touch base on where we are at with this.
> 
 
>>> 
>>> 
>>> 
>>> --
>>> *Note that I'm no longer using my Yahoo! email address. Please email me at
>>> billgra...@gmail.com going forward.*

[jira] [Created] (PIG-2972) Umbrella ticket for unit test failures in 0.11/trunk

[Rohini Palaniswamy (JIRA)]

Rohini Palaniswamy created PIG-2972:
---

 Summary: Umbrella ticket for unit test failures in 0.11/trunk
 Key: PIG-2972
 URL: https://issues.apache.org/jira/browse/PIG-2972
 Project: Pig
  Issue Type: Bug
Affects Versions: 0.11
Reporter: Rohini Palaniswamy
 Fix For: 0.11




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (PIG-2971) Add new parameter to specify the streaming environment

[Jonathan Coveney (JIRA)]

Jonathan Coveney created PIG-2971:
-

 Summary: Add new parameter to specify the streaming environment
 Key: PIG-2971
 URL: https://issues.apache.org/jira/browse/PIG-2971
 Project: Pig
  Issue Type: Bug
Reporter: Jonathan Coveney
Assignee: Jonathan Coveney
 Fix For: 0.11


The current proposal in PIG-2925 is a bit arbitrary. How do we set what is and 
is not too big? This instead let's the user set what they need.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Re: Unit test failures

[Cheolsoo Park]

>> Thinking of creating a umbrella ticket to track all current unit
test failures and get them fixed.

Once we open a umbrella ticket for test failures, everyone should link
their jiras that are for failing tests to it. I know that there are already
many open jiras for failing tests. I have several jiras myself.

Please let me know if you want me to open the umbrella ticket and link
existing tickets to it.

Thanks,
Cheolsoo

On Mon, Oct 15, 2012 at 11:36 AM, Cheolsoo Park wrote:

> This will be great! Please feel free to assign me bug fixes.
>
>
> On Mon, Oct 15, 2012 at 11:32 AM, Jonathan Coveney wrote:
>
>> I think this is an excellent idea, Rohini. We have resources to fix these
>> failures, so pointing them out should get them fixed. The error we've run
>> into is errors that are transient or related to a specific build on a
>> specific machine (and thus hard to replicate) vs. more serious errors.
>> Both
>> should be fixed, but it's hard to do the former when you have no control
>> over the CI. Throw in all the false fails from clover issues or
>> what-have-you and we're in a bad state. Cleaning that up would be awesome.
>>
>> 2012/10/15 Julien Le Dem 
>>
>> > I think this is a great idea.
>> > I don't think we have access to the build right now.
>> > Daniel seemed to know more about that.
>> > Julien
>> >
>> > On Mon, Oct 15, 2012 at 11:15 AM, Rohini Palaniswamy <
>> > rohini.adi...@gmail.com> wrote:
>> >
>> > > Hi all,
>> > > Quite a few unit tests are broken in trunk/0.11. Now that 0.11 is
>> > > branched it is more important to have them fixed and pass for
>> stability.
>> > > Also we need to figure out some way to have tests not broken. But with
>> > the
>> > > time unit tests take it is difficult for everyone to run it. I think
>> the
>> > > first thing to do would be to have the Apache jenkins build fixed and
>> > treat
>> > > build failures there seriously. I can help out with that if someone
>> can
>> > > guide me with access, modifying jenkins scripts, etc. Any other ideas?
>> > >Thinking of creating a umbrella ticket to track all current unit
>> test
>> > > failures and get them fixed. Or is there some other process that we
>> > usually
>> > > follow?
>> > >
>> > > Thanks,
>> > > Rohini
>> > >
>> >
>>
>
>

Re: Unit test failures

[Cheolsoo Park]

This will be great! Please feel free to assign me bug fixes.

On Mon, Oct 15, 2012 at 11:32 AM, Jonathan Coveney wrote:

> I think this is an excellent idea, Rohini. We have resources to fix these
> failures, so pointing them out should get them fixed. The error we've run
> into is errors that are transient or related to a specific build on a
> specific machine (and thus hard to replicate) vs. more serious errors. Both
> should be fixed, but it's hard to do the former when you have no control
> over the CI. Throw in all the false fails from clover issues or
> what-have-you and we're in a bad state. Cleaning that up would be awesome.
>
> 2012/10/15 Julien Le Dem 
>
> > I think this is a great idea.
> > I don't think we have access to the build right now.
> > Daniel seemed to know more about that.
> > Julien
> >
> > On Mon, Oct 15, 2012 at 11:15 AM, Rohini Palaniswamy <
> > rohini.adi...@gmail.com> wrote:
> >
> > > Hi all,
> > > Quite a few unit tests are broken in trunk/0.11. Now that 0.11 is
> > > branched it is more important to have them fixed and pass for
> stability.
> > > Also we need to figure out some way to have tests not broken. But with
> > the
> > > time unit tests take it is difficult for everyone to run it. I think
> the
> > > first thing to do would be to have the Apache jenkins build fixed and
> > treat
> > > build failures there seriously. I can help out with that if someone can
> > > guide me with access, modifying jenkins scripts, etc. Any other ideas?
> > >Thinking of creating a umbrella ticket to track all current unit
> test
> > > failures and get them fixed. Or is there some other process that we
> > usually
> > > follow?
> > >
> > > Thanks,
> > > Rohini
> > >
> >
>

Re: Unit test failures

[Jonathan Coveney]

I think this is an excellent idea, Rohini. We have resources to fix these
failures, so pointing them out should get them fixed. The error we've run
into is errors that are transient or related to a specific build on a
specific machine (and thus hard to replicate) vs. more serious errors. Both
should be fixed, but it's hard to do the former when you have no control
over the CI. Throw in all the false fails from clover issues or
what-have-you and we're in a bad state. Cleaning that up would be awesome.

2012/10/15 Julien Le Dem 

> I think this is a great idea.
> I don't think we have access to the build right now.
> Daniel seemed to know more about that.
> Julien
>
> On Mon, Oct 15, 2012 at 11:15 AM, Rohini Palaniswamy <
> rohini.adi...@gmail.com> wrote:
>
> > Hi all,
> > Quite a few unit tests are broken in trunk/0.11. Now that 0.11 is
> > branched it is more important to have them fixed and pass for stability.
> > Also we need to figure out some way to have tests not broken. But with
> the
> > time unit tests take it is difficult for everyone to run it. I think the
> > first thing to do would be to have the Apache jenkins build fixed and
> treat
> > build failures there seriously. I can help out with that if someone can
> > guide me with access, modifying jenkins scripts, etc. Any other ideas?
> >Thinking of creating a umbrella ticket to track all current unit test
> > failures and get them fixed. Or is there some other process that we
> usually
> > follow?
> >
> > Thanks,
> > Rohini
> >
>

Re: Unit test failures

[Julien Le Dem]

I think this is a great idea.
I don't think we have access to the build right now.
Daniel seemed to know more about that.
Julien

On Mon, Oct 15, 2012 at 11:15 AM, Rohini Palaniswamy <
rohini.adi...@gmail.com> wrote:

> Hi all,
> Quite a few unit tests are broken in trunk/0.11. Now that 0.11 is
> branched it is more important to have them fixed and pass for stability.
> Also we need to figure out some way to have tests not broken. But with the
> time unit tests take it is difficult for everyone to run it. I think the
> first thing to do would be to have the Apache jenkins build fixed and treat
> build failures there seriously. I can help out with that if someone can
> guide me with access, modifying jenkins scripts, etc. Any other ideas?
>Thinking of creating a umbrella ticket to track all current unit test
> failures and get them fixed. Or is there some other process that we usually
> follow?
>
> Thanks,
> Rohini
>

[jira] [Commented] (PIG-2926) TestPoissonSampleLoader failing on rhel environment

[Koji Noguchi (JIRA)]

[ 
https://issues.apache.org/jira/browse/PIG-2926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13476144#comment-13476144
 ] 

Koji Noguchi commented on PIG-2926:
---

Can I take this test away since the result changes depending on the jvm 
environment?

124 count = testNumSamples("0.0001", "100");
125 assertEquals(count, 42);


> TestPoissonSampleLoader failing on rhel environment
> ---
>
> Key: PIG-2926
> URL: https://issues.apache.org/jira/browse/PIG-2926
> Project: Pig
>  Issue Type: Bug
>Reporter: Koji Noguchi
>Priority: Minor
>
> Testing on rhel environment, TestPoissonSampleLoader fails with 
> {noformat}
> Testcase: testNumSamples took 22.077 sec
> FAILED
> expected:<47> but was:<42>
> junit.framework.AssertionFailedError: expected:<47> but was:<42>
> at 
> org.apache.pig.test.TestPoissonSampleLoader.testNumSamples(TestPoissonSampleLoader.java:125)
> {noformat}
> From 
> {noformat}
> 124 count = testNumSamples("0.0001", "100");
> 125 assertEquals(count, 42);
> {noformat}
> This runs fine on my mac environment.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (PIG-2968) ColumnMapKeyPrune fails to prune a subtree inside foreach

[Koji Noguchi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PIG-2968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Koji Noguchi updated PIG-2968:
--

Status: Patch Available  (was: Open)

> ColumnMapKeyPrune fails to prune a subtree inside foreach
> -
>
> Key: PIG-2968
> URL: https://issues.apache.org/jira/browse/PIG-2968
> Project: Pig
>  Issue Type: Bug
>  Components: parser
>Reporter: Koji Noguchi
>Assignee: Koji Noguchi
>Priority: Minor
> Attachments: pig-2968-trunk_v01.txt, pig-2968-trunk_v02.txt
>
>
> Sample code 
> {noformat}
> $ cat test/foreach.pig 
> daily = load 'nyse' as (exchange, symbol);
> grpd = group daily by exchange;
> uniquecnt = foreach grpd {
> sym = daily.symbol;
> uniq_sym = distinct sym;
> generate group, uniq_sym;
> };
> another = FOREACH uniquecnt GENERATE group;
> explain another;
> {noformat}
> This breaks when it tries to prune uniq_sym->sym->innerload_daily
> bq. 2012-10-12 14:54:11,031 [main] ERROR org.apache.pig.tools.grunt.Grunt - 
> ERROR 2000: Error processing rule ColumnMapKeyPrune. Try -t ColumnMapKeyPrune

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (PIG-2968) ColumnMapKeyPrune fails to prune a subtree inside foreach

[Koji Noguchi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PIG-2968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Koji Noguchi updated PIG-2968:
--

Attachment: pig-2968-trunk_v02.txt

bq. ll.toArray(new Operator[ll.size()]) can be done instead of ll.toArray(new 
Operator[0]) 

Uploading with the above change.

> ColumnMapKeyPrune fails to prune a subtree inside foreach
> -
>
> Key: PIG-2968
> URL: https://issues.apache.org/jira/browse/PIG-2968
> Project: Pig
>  Issue Type: Bug
>  Components: parser
>Reporter: Koji Noguchi
>Assignee: Koji Noguchi
>Priority: Minor
> Attachments: pig-2968-trunk_v01.txt, pig-2968-trunk_v02.txt
>
>
> Sample code 
> {noformat}
> $ cat test/foreach.pig 
> daily = load 'nyse' as (exchange, symbol);
> grpd = group daily by exchange;
> uniquecnt = foreach grpd {
> sym = daily.symbol;
> uniq_sym = distinct sym;
> generate group, uniq_sym;
> };
> another = FOREACH uniquecnt GENERATE group;
> explain another;
> {noformat}
> This breaks when it tries to prune uniq_sym->sym->innerload_daily
> bq. 2012-10-12 14:54:11,031 [main] ERROR org.apache.pig.tools.grunt.Grunt - 
> ERROR 2000: Error processing rule ColumnMapKeyPrune. Try -t ColumnMapKeyPrune

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (PIG-2963) Illustrate command and POPackageLite

[JIRA]

[ 
https://issues.apache.org/jira/browse/PIG-2963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13476096#comment-13476096
 ] 

Allan Avendaño commented on PIG-2963:
-

Thank Cheolsoo! It's working now. 
Now, I'm wondering which is the process to include it into the trunk. 

Thanks again!

> Illustrate command and POPackageLite
> 
>
> Key: PIG-2963
> URL: https://issues.apache.org/jira/browse/PIG-2963
> Project: Pig
>  Issue Type: Bug
>Reporter: Allan Avendaño
>Assignee: Cheolsoo Park
>Priority: Critical
> Fix For: 0.11
>
> Attachments: PIG-2963.patch
>
>
> While trying to execute a simple script like:
> A = LOAD 'test01' AS (f1:chararray,f2:int,f3:chararray);
> B = order A by f1;
> illustrate B;
> or 
> C = foreach B generate f1, f2; 
> illustrate C;
> I got the following exception:
> java.lang.RuntimeException: ReadOnceBag does not support getMemorySize 
> operation
>   at org.apache.pig.data.ReadOnceBag.getMemorySize(ReadOnceBag.java:74)
>   at org.apache.pig.data.SizeUtil.getPigObjMemSize(SizeUtil.java:61)
>   at org.apache.pig.data.DefaultTuple.getMemorySize(DefaultTuple.java:180)
>   at 
> org.apache.pig.pen.util.ExampleTuple.getMemorySize(ExampleTuple.java:97)
>   at 
> org.apache.pig.data.DefaultAbstractBag.getMemorySize(DefaultAbstractBag.java:148)
>   at 
> org.apache.pig.data.DefaultAbstractBag.markSpillableIfNecessary(DefaultAbstractBag.java:100)
>   at 
> org.apache.pig.data.DefaultAbstractBag.add(DefaultAbstractBag.java:92)
>   at org.apache.pig.pen.Illustrator.addData(Illustrator.java:116)
>   at 
> org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POPackageLite.illustratorMarkup(POPackageLite.java:227)
>   at 
> org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POPackageLite.getNext(POPackageLite.java:182)
>   at 
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapReduce$Reduce.processOnePackageOutput(PigGenericMapReduce.java:422)
>   at 
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapReduce$Reduce.reduce(PigGenericMapReduce.java:413)
>   at 
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapReduce$Reduce.reduce(PigGenericMapReduce.java:257)
>   at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:176)
>   at 
> org.apache.pig.pen.LocalMapReduceSimulator.launchPig(LocalMapReduceSimulator.java:235)
>   at 
> org.apache.pig.pen.ExampleGenerator.getData(ExampleGenerator.java:257)
>   at 
> org.apache.pig.pen.ExampleGenerator.getData(ExampleGenerator.java:238)
>   at 
> org.apache.pig.pen.LineageTrimmingVisitor.init(LineageTrimmingVisitor.java:103)
>   at 
> org.apache.pig.pen.LineageTrimmingVisitor.(LineageTrimmingVisitor.java:98)
>   at 
> org.apache.pig.pen.ExampleGenerator.getExamples(ExampleGenerator.java:166)
>   at org.apache.pig.PigServer.getExamples(PigServer.java:1180)
>   at 
> org.apache.pig.tools.grunt.GruntParser.processIllustrate(GruntParser.java:738)
>   at 
> org.apache.pig.tools.pigscript.parser.PigScriptParser.Illustrate(PigScriptParser.java:626)
>   at 
> org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:323)
>   at 
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:193)
>   at 
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:169)
>   at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:69)
>   at org.apache.pig.Main.run(Main.java:538)
>   at org.apache.pig.Main.main(Main.java:154)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>   at java.lang.reflect.Method.invoke(Method.java:597)
>   at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
> ==
> At log file, the following:
> Pig Stack Trace
> ---
> ERROR 2997: Encountered IOException. Exception
> java.io.IOException: Exception
> at org.apache.pig.PigServer.getExamples(PigServer.java:1186)
> at 
> org.apache.pig.tools.grunt.GruntParser.processIllustrate(GruntParser.java:738)
> at 
> org.apache.pig.tools.pigscript.parser.PigScriptParser.Illustrate(PigScriptParser.java:626)
> at 
> org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:323)
> at 
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:193)
> at 
> org.apache.pig.tools.grunt.GruntParser.parseStopOnErr