[jira] [Commented] (PIG-2676) Enable piggybank DBStorage to read from JDBC databases

2012-12-19 Thread Daniel Dai (JIRA) 

[ 
https://issues.apache.org/jira/browse/PIG-2676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13535957#comment-13535957
 ] 

Daniel Dai commented on PIG-2676:
-

Jeff, we are about to release 0.10.1, are you still working on it? Otherwise I 
will unlink it from 0.10.1

 Enable piggybank DBStorage to read from JDBC databases
 --

 Key: PIG-2676
 URL: https://issues.apache.org/jira/browse/PIG-2676
 Project: Pig
  Issue Type: Improvement
  Components: piggybank
Affects Versions: 0.9.2, 0.10.0, 0.11, 0.10.1
Reporter: Russell Jurney
Priority: Critical
  Labels: dance, data, fun, happy, io, jdbc, mysql, pants, pig, 
 silly, sloppy, sql
 Fix For: 0.10.1


 I would like to load data from a SQL DB using a SQL SELECT query. DBStorage 
 does not implement loading data, getNext, etc.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (PIG-2527) ILLUSTRATE fails for relations LOADed with the AvroStorage UDF

2012-12-19 Thread Daniel Dai (JIRA) 

 [ 
https://issues.apache.org/jira/browse/PIG-2527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Dai updated PIG-2527:


Fix Version/s: (was: 0.10.1)

 ILLUSTRATE fails for relations LOADed with the AvroStorage UDF
 --

 Key: PIG-2527
 URL: https://issues.apache.org/jira/browse/PIG-2527
 Project: Pig
  Issue Type: Bug
  Components: piggybank
Affects Versions: 0.9.2, 0.10.0, 0.11, 0.10.1
Reporter: Russell Jurney
Assignee: Jonathan Coveney
Priority: Blocker
  Labels: avro, avro_udf, avrostorage, happy, pig, storage, udf

 grunt describe emails
 emails: {message_id: chararray,from: {PIG_WRAPPER: (ARRAY_ELEM: 
 chararray)},to: {PIG_WRAPPER: (ARRAY_ELEM: chararray)},cc: {PIG_WRAPPER: 
 (ARRAY_ELEM: chararray)},bcc: {PIG_WRAPPER: (ARRAY_ELEM: 
 chararray)},reply_to: {PIG_WRAPPER: (ARRAY_ELEM: chararray)},in_reply_to: 
 {PIG_WRAPPER: (ARRAY_ELEM: chararray)},subject: chararray,body: 
 chararray,date: chararray}
 grunt illustrate emails 
 2012-02-10 18:15:01,591 [main] INFO  
 org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting 
 to hadoop file system at: file:///
 2012-02-10 18:15:01,592 [main] INFO  
 org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler - 
 File concatenation threshold: 100 optimistic? false
 2012-02-10 18:15:01,649 [main] INFO  
 org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer
  - MR plan size before optimization: 1
 2012-02-10 18:15:01,649 [main] INFO  
 org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer
  - MR plan size after optimization: 1
 2012-02-10 18:15:01,649 [main] INFO  
 org.apache.pig.tools.pigstats.ScriptState - Pig script settings are added to 
 the job
 2012-02-10 18:15:01,649 [main] INFO  
 org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
  - mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3
 2012-02-10 18:15:01,668 [main] INFO  
 org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total input paths to 
 process : 5
 2012-02-10 18:15:02,719 [main] INFO  
 org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler - 
 File concatenation threshold: 100 optimistic? false
 2012-02-10 18:15:02,719 [main] INFO  
 org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer
  - MR plan size before optimization: 1
 2012-02-10 18:15:02,719 [main] INFO  
 org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer
  - MR plan size after optimization: 1
 2012-02-10 18:15:02,720 [main] INFO  
 org.apache.pig.tools.pigstats.ScriptState - Pig script settings are added to 
 the job
 2012-02-10 18:15:02,720 [main] INFO  
 org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
  - mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3
 2012-02-10 18:15:02,733 [main] INFO  
 org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler - 
 File concatenation threshold: 100 optimistic? false
 2012-02-10 18:15:02,734 [main] INFO  
 org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer
  - MR plan size before optimization: 1
 2012-02-10 18:15:02,734 [main] INFO  
 org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer
  - MR plan size after optimization: 1
 2012-02-10 18:15:02,734 [main] INFO  
 org.apache.pig.tools.pigstats.ScriptState - Pig script settings are added to 
 the job
 2012-02-10 18:15:02,734 [main] INFO  
 org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
  - mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3
 2012-02-10 18:15:02,749 [main] ERROR 
 org.apache.pig.pen.AugmentBaseDataVisitor - No (valid) input data found!
 java.lang.RuntimeException: No (valid) input data found!
 at 
 org.apache.pig.pen.AugmentBaseDataVisitor.visit(AugmentBaseDataVisitor.java:579)
 at 
 org.apache.pig.newplan.logical.relational.LOLoad.accept(LOLoad.java:218)
 at 
 org.apache.pig.pen.util.PreOrderDepthFirstWalker.depthFirst(PreOrderDepthFirstWalker.java:82)
 at 
 org.apache.pig.pen.util.PreOrderDepthFirstWalker.walk(PreOrderDepthFirstWalker.java:66)
 at org.apache.pig.newplan.PlanVisitor.visit(PlanVisitor.java:50)
 at 
 org.apache.pig.pen.ExampleGenerator.getExamples(ExampleGenerator.java:180)
 at org.apache.pig.PigServer.getExamples(PigServer.java:1245)
 at 
 org.apache.pig.tools.grunt.GruntParser.processIllustrate(GruntParser.java:698)
 at 
 org.apache.pig.tools.pigscript.parser.PigScriptParser.Illustrate(PigScriptParser.java:591)
 at

[jira] [Updated] (PIG-2527) ILLUSTRATE fails for relations LOADed with the AvroStorage UDF

2012-12-19 Thread Daniel Dai (JIRA) 

 [ 
https://issues.apache.org/jira/browse/PIG-2527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Dai updated PIG-2527:


Fix Version/s: 0.10.1

 ILLUSTRATE fails for relations LOADed with the AvroStorage UDF
 --

 Key: PIG-2527
 URL: https://issues.apache.org/jira/browse/PIG-2527
 Project: Pig
  Issue Type: Bug
  Components: piggybank
Affects Versions: 0.9.2, 0.10.0, 0.11, 0.10.1
Reporter: Russell Jurney
Assignee: Jonathan Coveney
Priority: Blocker
  Labels: avro, avro_udf, avrostorage, happy, pig, storage, udf
 Fix For: 0.10.1


 grunt describe emails
 emails: {message_id: chararray,from: {PIG_WRAPPER: (ARRAY_ELEM: 
 chararray)},to: {PIG_WRAPPER: (ARRAY_ELEM: chararray)},cc: {PIG_WRAPPER: 
 (ARRAY_ELEM: chararray)},bcc: {PIG_WRAPPER: (ARRAY_ELEM: 
 chararray)},reply_to: {PIG_WRAPPER: (ARRAY_ELEM: chararray)},in_reply_to: 
 {PIG_WRAPPER: (ARRAY_ELEM: chararray)},subject: chararray,body: 
 chararray,date: chararray}
 grunt illustrate emails 
 2012-02-10 18:15:01,591 [main] INFO  
 org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting 
 to hadoop file system at: file:///
 2012-02-10 18:15:01,592 [main] INFO  
 org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler - 
 File concatenation threshold: 100 optimistic? false
 2012-02-10 18:15:01,649 [main] INFO  
 org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer
  - MR plan size before optimization: 1
 2012-02-10 18:15:01,649 [main] INFO  
 org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer
  - MR plan size after optimization: 1
 2012-02-10 18:15:01,649 [main] INFO  
 org.apache.pig.tools.pigstats.ScriptState - Pig script settings are added to 
 the job
 2012-02-10 18:15:01,649 [main] INFO  
 org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
  - mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3
 2012-02-10 18:15:01,668 [main] INFO  
 org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total input paths to 
 process : 5
 2012-02-10 18:15:02,719 [main] INFO  
 org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler - 
 File concatenation threshold: 100 optimistic? false
 2012-02-10 18:15:02,719 [main] INFO  
 org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer
  - MR plan size before optimization: 1
 2012-02-10 18:15:02,719 [main] INFO  
 org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer
  - MR plan size after optimization: 1
 2012-02-10 18:15:02,720 [main] INFO  
 org.apache.pig.tools.pigstats.ScriptState - Pig script settings are added to 
 the job
 2012-02-10 18:15:02,720 [main] INFO  
 org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
  - mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3
 2012-02-10 18:15:02,733 [main] INFO  
 org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler - 
 File concatenation threshold: 100 optimistic? false
 2012-02-10 18:15:02,734 [main] INFO  
 org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer
  - MR plan size before optimization: 1
 2012-02-10 18:15:02,734 [main] INFO  
 org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer
  - MR plan size after optimization: 1
 2012-02-10 18:15:02,734 [main] INFO  
 org.apache.pig.tools.pigstats.ScriptState - Pig script settings are added to 
 the job
 2012-02-10 18:15:02,734 [main] INFO  
 org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
  - mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3
 2012-02-10 18:15:02,749 [main] ERROR 
 org.apache.pig.pen.AugmentBaseDataVisitor - No (valid) input data found!
 java.lang.RuntimeException: No (valid) input data found!
 at 
 org.apache.pig.pen.AugmentBaseDataVisitor.visit(AugmentBaseDataVisitor.java:579)
 at 
 org.apache.pig.newplan.logical.relational.LOLoad.accept(LOLoad.java:218)
 at 
 org.apache.pig.pen.util.PreOrderDepthFirstWalker.depthFirst(PreOrderDepthFirstWalker.java:82)
 at 
 org.apache.pig.pen.util.PreOrderDepthFirstWalker.walk(PreOrderDepthFirstWalker.java:66)
 at org.apache.pig.newplan.PlanVisitor.visit(PlanVisitor.java:50)
 at 
 org.apache.pig.pen.ExampleGenerator.getExamples(ExampleGenerator.java:180)
 at org.apache.pig.PigServer.getExamples(PigServer.java:1245)
 at 
 org.apache.pig.tools.grunt.GruntParser.processIllustrate(GruntParser.java:698)
 at 
 org.apache.pig.tools.pigscript.parser.PigScriptParser.Illustrate(PigScriptParser.java:591)
 at

[jira] [Resolved] (PIG-2527) ILLUSTRATE fails for relations LOADed with the AvroStorage UDF

2012-12-19 Thread Daniel Dai (JIRA) 

 [ 
https://issues.apache.org/jira/browse/PIG-2527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Dai resolved PIG-2527.
-

Resolution: Fixed

Verified it is fixed in 0.10 branch. Mark it as fixed in 0.10.1.

 ILLUSTRATE fails for relations LOADed with the AvroStorage UDF
 --

 Key: PIG-2527
 URL: https://issues.apache.org/jira/browse/PIG-2527
 Project: Pig
  Issue Type: Bug
  Components: piggybank
Affects Versions: 0.9.2, 0.10.0, 0.11, 0.10.1
Reporter: Russell Jurney
Assignee: Jonathan Coveney
Priority: Blocker
  Labels: avro, avro_udf, avrostorage, happy, pig, storage, udf
 Fix For: 0.10.1


 grunt describe emails
 emails: {message_id: chararray,from: {PIG_WRAPPER: (ARRAY_ELEM: 
 chararray)},to: {PIG_WRAPPER: (ARRAY_ELEM: chararray)},cc: {PIG_WRAPPER: 
 (ARRAY_ELEM: chararray)},bcc: {PIG_WRAPPER: (ARRAY_ELEM: 
 chararray)},reply_to: {PIG_WRAPPER: (ARRAY_ELEM: chararray)},in_reply_to: 
 {PIG_WRAPPER: (ARRAY_ELEM: chararray)},subject: chararray,body: 
 chararray,date: chararray}
 grunt illustrate emails 
 2012-02-10 18:15:01,591 [main] INFO  
 org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting 
 to hadoop file system at: file:///
 2012-02-10 18:15:01,592 [main] INFO  
 org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler - 
 File concatenation threshold: 100 optimistic? false
 2012-02-10 18:15:01,649 [main] INFO  
 org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer
  - MR plan size before optimization: 1
 2012-02-10 18:15:01,649 [main] INFO  
 org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer
  - MR plan size after optimization: 1
 2012-02-10 18:15:01,649 [main] INFO  
 org.apache.pig.tools.pigstats.ScriptState - Pig script settings are added to 
 the job
 2012-02-10 18:15:01,649 [main] INFO  
 org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
  - mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3
 2012-02-10 18:15:01,668 [main] INFO  
 org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total input paths to 
 process : 5
 2012-02-10 18:15:02,719 [main] INFO  
 org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler - 
 File concatenation threshold: 100 optimistic? false
 2012-02-10 18:15:02,719 [main] INFO  
 org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer
  - MR plan size before optimization: 1
 2012-02-10 18:15:02,719 [main] INFO  
 org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer
  - MR plan size after optimization: 1
 2012-02-10 18:15:02,720 [main] INFO  
 org.apache.pig.tools.pigstats.ScriptState - Pig script settings are added to 
 the job
 2012-02-10 18:15:02,720 [main] INFO  
 org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
  - mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3
 2012-02-10 18:15:02,733 [main] INFO  
 org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler - 
 File concatenation threshold: 100 optimistic? false
 2012-02-10 18:15:02,734 [main] INFO  
 org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer
  - MR plan size before optimization: 1
 2012-02-10 18:15:02,734 [main] INFO  
 org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer
  - MR plan size after optimization: 1
 2012-02-10 18:15:02,734 [main] INFO  
 org.apache.pig.tools.pigstats.ScriptState - Pig script settings are added to 
 the job
 2012-02-10 18:15:02,734 [main] INFO  
 org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
  - mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3
 2012-02-10 18:15:02,749 [main] ERROR 
 org.apache.pig.pen.AugmentBaseDataVisitor - No (valid) input data found!
 java.lang.RuntimeException: No (valid) input data found!
 at 
 org.apache.pig.pen.AugmentBaseDataVisitor.visit(AugmentBaseDataVisitor.java:579)
 at 
 org.apache.pig.newplan.logical.relational.LOLoad.accept(LOLoad.java:218)
 at 
 org.apache.pig.pen.util.PreOrderDepthFirstWalker.depthFirst(PreOrderDepthFirstWalker.java:82)
 at 
 org.apache.pig.pen.util.PreOrderDepthFirstWalker.walk(PreOrderDepthFirstWalker.java:66)
 at org.apache.pig.newplan.PlanVisitor.visit(PlanVisitor.java:50)
 at 
 org.apache.pig.pen.ExampleGenerator.getExamples(ExampleGenerator.java:180)
 at org.apache.pig.PigServer.getExamples(PigServer.java:1245)
 at 
 org.apache.pig.tools.grunt.GruntParser.processIllustrate(GruntParser.java:698)
 at

[jira] [Updated] (PIG-2539) Cannot use s3 buckets with '_' in the name

2012-12-19 Thread Daniel Dai (JIRA) 

 [ 
https://issues.apache.org/jira/browse/PIG-2539?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Dai updated PIG-2539:


Fix Version/s: (was: 0.10.1)

Some more investigation needed, unlink from 0.10.1.

 Cannot use s3 buckets with '_' in the name
 --

 Key: PIG-2539
 URL: https://issues.apache.org/jira/browse/PIG-2539
 Project: Pig
  Issue Type: Bug
  Components: grunt, parser
Affects Versions: 0.9.1, 0.10.0, 0.10.1
 Environment: Amazon Elastic Map Reduce
Reporter: Russell Jurney
Assignee: Daniel Dai
Priority: Blocker
  Labels: amazon, fun, happy, me, no, pants, pig, sad, work
 Attachments: PIG-2539-0.patch


 grunt cd s3://agile_data
 2012-02-16 22:05:59,461 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 
 2999: Unexpected internal error. Invalid hostname in URI s3://agile_data
 Details at logfile: /home/hadoop/pig_1329429351155.log
 I think the next behavior is already documented/bug filed:
 grunt cd 's3://agile_data'
 2012-02-16 22:02:28,489 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 
 2999: Unexpected internal error. java.net.URISyntaxException: Illegal 
 character in scheme name at index 0: 's3://agile_data'
 Details at logfile: /home/hadoop/pig_1329429351155.log

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (PIG-2676) Enable piggybank DBStorage to read from JDBC databases

2012-12-19 Thread Daniel Dai (JIRA) 

 [ 
https://issues.apache.org/jira/browse/PIG-2676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Dai updated PIG-2676:


Fix Version/s: (was: 0.10.1)

Unlink from 0.10.1. Jeff, feel free to submit patch, we can include in the 
later release.

 Enable piggybank DBStorage to read from JDBC databases
 --

 Key: PIG-2676
 URL: https://issues.apache.org/jira/browse/PIG-2676
 Project: Pig
  Issue Type: Improvement
  Components: piggybank
Affects Versions: 0.9.2, 0.10.0, 0.11, 0.10.1
Reporter: Russell Jurney
Priority: Critical
  Labels: dance, data, fun, happy, io, jdbc, mysql, pants, pig, 
 silly, sloppy, sql

 I would like to load data from a SQL DB using a SQL SELECT query. DBStorage 
 does not implement loading data, getNext, etc.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (PIG-3070) PigTest - Patch for PIG-3027 - Pls review to see if trim required after replacing all \r\n with \n to avoid additional newline at end of expected string.

2012-12-19 Thread Daniel Dai (JIRA) 

 [ 
https://issues.apache.org/jira/browse/PIG-3070?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Dai updated PIG-3070:


Fix Version/s: (was: 0.10.1)
   0.11
 Assignee: Daniel Dai

The base Jira targets 0.11. Set target for this Jira to 0.11 as well.

 PigTest - Patch for PIG-3027 - Pls review to see if trim required after 
 replacing all \r\n with \n to avoid additional newline at end of expected 
 string. 
 --

 Key: PIG-3070
 URL: https://issues.apache.org/jira/browse/PIG-3070
 Project: Pig
  Issue Type: Bug
  Components: build
Affects Versions: 0.10.0
Reporter: Raghavendra Saravanamurthy
Assignee: Daniel Dai
Priority: Minor
 Fix For: 0.11


 Ref Patch: 
 https://issues.apache.org/jira/secure/attachment/12552008/PIG-3027.trunk.1.patch
 With the above patch fix:
 readFile(expected).replaceAll(\r\n, \n) gives the expected file contents 
 as (note: the exp file does not contain a \n after the last line of text):
 
 (yahoo,25)
 (facebook,15)
 (twitter,7)
 12/11/28 02:59:06 INFO pigunit.PigTest:..
 
 And the test case fails with:
 
 junit.framework.ComparisonFailure: expected:...book,15)
 (twitter,7)[
 ] but was:...book,15)
 (twitter,7)[]
 at junit.framework.Assert.assertEquals(Assert.java:85)
 at junit.framework.Assert.assertEquals(Assert.java:91)
 
 After adding trim to the above patch:
 readFile(expected).replaceAll(\r\n, \n).trim() gives the expected file 
 contents as:
 
 (yahoo,25)
 (facebook,15)
 (twitter,7)
 12/11/28 03:10:06 INFO pigunit.PigTest:..
 
 And the above test case passed.
 Please review if trim should be added to avoid the additional newline at the 
 end of expected string.
 Pls note: I was running the tests in rhel environment.
 Thank you.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (PIG-3100) If a .pig_schema file is present, can get an index out of bounds error

2012-12-19 Thread Jonathan Coveney (JIRA) 
Jonathan Coveney created PIG-3100:
-

 Summary: If a .pig_schema file is present, can get an index out of 
bounds error
 Key: PIG-3100
 URL: https://issues.apache.org/jira/browse/PIG-3100
 Project: Pig
  Issue Type: Bug
Reporter: Jonathan Coveney
Assignee: Jonathan Coveney
 Fix For: 0.12


In the case that a .pig_schema file is present, if you have a record with fewer 
than expected fields, pig errors out with an index out of bounds exception that 
is annoying, unnecessary, and unhelpful.

Instead of improving logging, I decided to just do what pig should do, which is 
fill in the records.

Patch will include a test and the fix.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (PIG-3100) If a .pig_schema file is present, can get an index out of bounds error

2012-12-19 Thread Jonathan Coveney (JIRA) 

 [ 
https://issues.apache.org/jira/browse/PIG-3100?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jonathan Coveney updated PIG-3100:
--

Attachment: PIG-3100-0.patch
PIG-3100-0_nows.patch

Here is the patch with test and fix.

 If a .pig_schema file is present, can get an index out of bounds error
 --

 Key: PIG-3100
 URL: https://issues.apache.org/jira/browse/PIG-3100
 Project: Pig
  Issue Type: Bug
Reporter: Jonathan Coveney
Assignee: Jonathan Coveney
 Fix For: 0.12

 Attachments: PIG-3100-0_nows.patch, PIG-3100-0.patch


 In the case that a .pig_schema file is present, if you have a record with 
 fewer than expected fields, pig errors out with an index out of bounds 
 exception that is annoying, unnecessary, and unhelpful.
 Instead of improving logging, I decided to just do what pig should do, which 
 is fill in the records.
 Patch will include a test and the fix.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Assigned] (PIG-3090) Introduce a syntax to be able to easily refer to the previously defined relation

2012-12-19 Thread Jonathan Coveney (JIRA) 

 [ 
https://issues.apache.org/jira/browse/PIG-3090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jonathan Coveney reassigned PIG-3090:
-

Assignee: Jonathan Coveney

 Introduce a syntax to be able to easily refer to the previously defined 
 relation
 

 Key: PIG-3090
 URL: https://issues.apache.org/jira/browse/PIG-3090
 Project: Pig
  Issue Type: New Feature
Reporter: Jonathan Coveney
Assignee: Jonathan Coveney
 Attachments: PIG-3090-0.patch


 Sometimes I feel like swimming with ANTLRs. This particular feature isn't too 
 hard to add... and supports syntax like this:
 {code}
 a = load 'thing' as (x:int);
 b = foreach @ generate x;
 c = foreach @ generate x;
 d = foreach @ generate x;
 {code}
 I have a patch, though I need to make sure it doesn't change anything (it 
 shouldn't) and I need to add tests.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Resolved] (PIG-3056) CLONE - PigStorageSchema need to deal with missing field

2012-12-19 Thread Rohini Palaniswamy (JIRA) 

 [ 
https://issues.apache.org/jira/browse/PIG-3056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rohini Palaniswamy resolved PIG-3056.
-

Resolution: Duplicate

 CLONE - PigStorageSchema need to deal with missing field
 

 Key: PIG-3056
 URL: https://issues.apache.org/jira/browse/PIG-3056
 Project: Pig
  Issue Type: Bug
  Components: impl
Affects Versions: 0.10.0
Reporter: Koji Noguchi

 Currently, if data contains fewer columns than the schema, PigStorageSchema 
 will throw IndexOutOfBound exception (PigStorageSchema:97). We should padding 
 null in this case as we did in PigStorage.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Re: Prepare for Pig 0.10.1 release

2012-12-19 Thread Julien Le Dem 
no objection to PIG-2907 in 0.10.1
However I do think new features should go in trunk.
Julien


On Wed, Dec 19, 2012 at 1:20 PM, Rohini Palaniswamy rohini.adi...@gmail.com
 wrote:

 Daniel,
Since 0.10.1 is going out sooner and has support for hadoop 23, if it is
 acceptable, I would like to pull in PIG-2907 into 0.10.1. Other projects
 are blocked (hcat, oozie) on running unit tests with hadoop 23 because we
 don't have a pig jar for 23 in maven.  Thoughts/objections?

 Regards,
 Rohini


 On Tue, Dec 18, 2012 at 6:54 PM, Russell Jurney russell.jur...@gmail.com
 wrote:

  High five! Think of all those new cores running Pig jobs!
  On Dec 18, 2012 6:24 PM, Daniel Dai da...@hortonworks.com wrote:
 
   I would like to commit Windows patches to 0.11.0. Allow me several
 days.
  
   Thanks,
   Daniel
  
   On Tue, Dec 18, 2012 at 5:47 PM, Julien Le Dem jul...@twitter.com
  wrote:
Sounds good to me.
Can we cut pig 0.11.0 at the same time ?
Julien
   
   
On Tue, Dec 18, 2012 at 7:54 AM, Daniel Dai da...@hortonworks.com
   wrote:
   
Hi, Pig developers,
   
We have fixed a bunch of bugs since
0.10.0(
   
 http://svn.apache.org/repos/asf/pig/branches/branch-0.10/CHANGES.txt
  ).
I would like to propose a 0.10.1 release from top of 0.10 branch
 after
clearing all pending issues
(
   
  
 
 https://issues.apache.org/jira/issues/?jql=project%20%3D%20PIG%20AND%20resolution%20%3D%20Unresolved%20AND%20fixVersion%20%3D%20%220.10.1%22
).
   
Any objections?
   
Thanks,
Daniel

[jira] [Commented] (PIG-3099) Pig unit test fixes for TestGrunt(1), TestStore(2), TestEmptyInputDir(3)

2012-12-19 Thread Vikram Dixit K (JIRA) 

[ 
https://issues.apache.org/jira/browse/PIG-3099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13536498#comment-13536498
 ] 

Vikram Dixit K commented on PIG-3099:
-

Removing extraneous parts of the patch.

 Pig unit test fixes for TestGrunt(1), TestStore(2), TestEmptyInputDir(3)
 

 Key: PIG-3099
 URL: https://issues.apache.org/jira/browse/PIG-3099
 Project: Pig
  Issue Type: Bug
Affects Versions: 0.10.0
Reporter: Vikram Dixit K
Assignee: Vikram Dixit K
 Attachments: PIG-3099_2.patch, PIG-3099.patch




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (PIG-3099) Pig unit test fixes for TestGrunt(1), TestStore(2), TestEmptyInputDir(3)

2012-12-19 Thread Vikram Dixit K (JIRA) 

 [ 
https://issues.apache.org/jira/browse/PIG-3099?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vikram Dixit K updated PIG-3099:


Fix Version/s: 0.10.1
   0.11
   Status: Patch Available  (was: Open)

 Pig unit test fixes for TestGrunt(1), TestStore(2), TestEmptyInputDir(3)
 

 Key: PIG-3099
 URL: https://issues.apache.org/jira/browse/PIG-3099
 Project: Pig
  Issue Type: Bug
Affects Versions: 0.10.0
Reporter: Vikram Dixit K
Assignee: Vikram Dixit K
 Fix For: 0.11, 0.10.1

 Attachments: PIG-3099_2.patch, PIG-3099.patch




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (PIG-3076) make TestScalarAliases more reliable

2012-12-19 Thread Julien Le Dem (JIRA) 

 [ 
https://issues.apache.org/jira/browse/PIG-3076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Julien Le Dem updated PIG-3076:
---

Attachment: PIG-3076_1.patch

PIG-3076_1.patch addresses comments

 make TestScalarAliases more reliable
 

 Key: PIG-3076
 URL: https://issues.apache.org/jira/browse/PIG-3076
 Project: Pig
  Issue Type: Test
Reporter: Julien Le Dem
Assignee: Julien Le Dem
 Fix For: 0.11, 0.12

 Attachments: PIG-3076_1.patch, PIG-3076.patch


 currently, this test writes in the root directory so its output is not 
 deleted by ant clean.
 Also it deletes its output in the end instead of the begining.
 The consequence is that if the test fail once then it will keep failing until 
 the directory is manually cleaned up (not good for CI)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (PIG-3076) make TestScalarAliases more reliable

2012-12-19 Thread Jonathan Coveney (JIRA) 

[ 
https://issues.apache.org/jira/browse/PIG-3076?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13536580#comment-13536580
 ] 

Jonathan Coveney commented on PIG-3076:
---

+1

 make TestScalarAliases more reliable
 

 Key: PIG-3076
 URL: https://issues.apache.org/jira/browse/PIG-3076
 Project: Pig
  Issue Type: Test
Reporter: Julien Le Dem
Assignee: Julien Le Dem
 Fix For: 0.11, 0.12

 Attachments: PIG-3076_1.patch, PIG-3076.patch


 currently, this test writes in the root directory so its output is not 
 deleted by ant clean.
 Also it deletes its output in the end instead of the begining.
 The consequence is that if the test fail once then it will keep failing until 
 the directory is manually cleaned up (not good for CI)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Re: Prepare for Pig 0.10.1 release

2012-12-19 Thread Daniel Dai 
Hi, Rohini,
If you want to commit, can you do it immediately? I'd like to get
0.10.1 out ASAP.

Thanks,
Daniel

On Wed, Dec 19, 2012 at 9:55 PM, Julien Le Dem jul...@twitter.com wrote:
 no objection to PIG-2907 in 0.10.1
 However I do think new features should go in trunk.
 Julien


 On Wed, Dec 19, 2012 at 1:20 PM, Rohini Palaniswamy rohini.adi...@gmail.com
 wrote:

 Daniel,
Since 0.10.1 is going out sooner and has support for hadoop 23, if it is
 acceptable, I would like to pull in PIG-2907 into 0.10.1. Other projects
 are blocked (hcat, oozie) on running unit tests with hadoop 23 because we
 don't have a pig jar for 23 in maven.  Thoughts/objections?

 Regards,
 Rohini


 On Tue, Dec 18, 2012 at 6:54 PM, Russell Jurney russell.jur...@gmail.com
 wrote:

  High five! Think of all those new cores running Pig jobs!
  On Dec 18, 2012 6:24 PM, Daniel Dai da...@hortonworks.com wrote:
 
   I would like to commit Windows patches to 0.11.0. Allow me several
 days.
  
   Thanks,
   Daniel
  
   On Tue, Dec 18, 2012 at 5:47 PM, Julien Le Dem jul...@twitter.com
  wrote:
Sounds good to me.
Can we cut pig 0.11.0 at the same time ?
Julien
   
   
On Tue, Dec 18, 2012 at 7:54 AM, Daniel Dai da...@hortonworks.com
   wrote:
   
Hi, Pig developers,
   
We have fixed a bunch of bugs since
0.10.0(
   
 http://svn.apache.org/repos/asf/pig/branches/branch-0.10/CHANGES.txt
  ).
I would like to propose a 0.10.1 release from top of 0.10 branch
 after
clearing all pending issues
(
   
  
 
 https://issues.apache.org/jira/issues/?jql=project%20%3D%20PIG%20AND%20resolution%20%3D%20Unresolved%20AND%20fixVersion%20%3D%20%220.10.1%22
).
   
Any objections?
   
Thanks,
Daniel

[jira] Subscription: PIG patch available

2012-12-19 Thread jira 
Issue Subscription
Filter: PIG patch available (40 issues)

Subscriber: pigdaily

Key Summary
PIG-3099Pig unit test fixes for TestGrunt(1), TestStore(2), 
TestEmptyInputDir(3)
https://issues.apache.org/jira/browse/PIG-3099
PIG-3098Add another test for the self join case
https://issues.apache.org/jira/browse/PIG-3098
PIG-3096Make PigUnit thread safe
https://issues.apache.org/jira/browse/PIG-3096
PIG-3088Add a builtin udf which removes prefixes
https://issues.apache.org/jira/browse/PIG-3088
PIG-3086Allow A Prefix To Be Added To URIs In PigUnit Tests 
https://issues.apache.org/jira/browse/PIG-3086
PIG-3078Make a UDF that, given a string, returns just the columns prefixed 
by that string
https://issues.apache.org/jira/browse/PIG-3078
PIG-3073POUserFunc creating log spam for large scripts
https://issues.apache.org/jira/browse/PIG-3073
PIG-3069Native Windows Compatibility for Pig E2E Tests and Harness
https://issues.apache.org/jira/browse/PIG-3069
PIG-3067HBaseStorage should be split up to become more managable
https://issues.apache.org/jira/browse/PIG-3067
PIG-3066Fix TestPigRunner in trunk
https://issues.apache.org/jira/browse/PIG-3066
PIG-3057make readField protected to be able to override it if we extend 
PigStorage
https://issues.apache.org/jira/browse/PIG-3057
PIG-3051java.lang.IndexOutOfBoundsException  failure with LimitOptimizer + 
ColumnPruning
https://issues.apache.org/jira/browse/PIG-3051
PIG-3050Fix FindBugs multithreading warnings
https://issues.apache.org/jira/browse/PIG-3050
PIG-3029TestTypeCheckingValidatorNewLP has some path reference issues for 
cross-platform execution
https://issues.apache.org/jira/browse/PIG-3029
PIG-3028testGrunt dev test needs some command filters to run correctly 
without cygwin
https://issues.apache.org/jira/browse/PIG-3028
PIG-3027pigTest unit test needs a newline filter for comparisons of golden 
multi-line
https://issues.apache.org/jira/browse/PIG-3027
PIG-3026Pig checked-in baseline comparisons need a pre-filter to address 
OS-specific newline differences
https://issues.apache.org/jira/browse/PIG-3026
PIG-3025TestPruneColumn unit test - SimpleEchoStreamingCommand perl inline 
script needs simplification
https://issues.apache.org/jira/browse/PIG-3025
PIG-3024TestEmptyInputDir unit test - hadoop version detection logic is 
brittle
https://issues.apache.org/jira/browse/PIG-3024
PIG-3015Rewrite of AvroStorage
https://issues.apache.org/jira/browse/PIG-3015
PIG-3010Allow UDF's to flatten themselves
https://issues.apache.org/jira/browse/PIG-3010
PIG-2959Add a pig.cmd for Pig to run under Windows
https://issues.apache.org/jira/browse/PIG-2959
PIG-2957TetsScriptUDF fail due to volume prefix in jar
https://issues.apache.org/jira/browse/PIG-2957
PIG-2956Invalid cache specification for some streaming statement
https://issues.apache.org/jira/browse/PIG-2956
PIG-2955 Fix bunch of Pig e2e tests on Windows 
https://issues.apache.org/jira/browse/PIG-2955
PIG-2878Pig current releases lack a UDF equalIgnoreCase.This function 
returns a Boolean value indicating whether string left is equal to string 
right. This check is case insensitive.
https://issues.apache.org/jira/browse/PIG-2878
PIG-2873Converting bin/pig shell script to python
https://issues.apache.org/jira/browse/PIG-2873
PIG-2834MultiStorage requires unused constructor argument
https://issues.apache.org/jira/browse/PIG-2834
PIG-2824Pushing checking number of fields into LoadFunc
https://issues.apache.org/jira/browse/PIG-2824
PIG-2788improved string interpolation of variables
https://issues.apache.org/jira/browse/PIG-2788
PIG-2661Pig uses an extra job for loading data in Pigmix L9
https://issues.apache.org/jira/browse/PIG-2661
PIG-2645PigSplit does not handle the case where SerializationFactory 
returns null
https://issues.apache.org/jira/browse/PIG-2645
PIG-2614AvroStorage crashes on LOADING a single bad error
https://issues.apache.org/jira/browse/PIG-2614
PIG-2507Semicolon in paramenters for UDF results in parsing error
https://issues.apache.org/jira/browse/PIG-2507
PIG-2433Jython import module not working if module path is in classpath
https://issues.apache.org/jira/browse/PIG-2433
PIG-2417Streaming UDFs -  allow users to easily write UDFs in scripting 
languages with no JVM implementation.
https://issues.apache.org/jira/browse/PIG-2417
PIG-2362Rework Ant build.xml to use macrodef instead of antcall

[jira] [Updated] (PIG-2764) Add a biginteger and bigdecimal type to pig

2012-12-19 Thread Jonathan Coveney (JIRA) 

 [ 
https://issues.apache.org/jira/browse/PIG-2764?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jonathan Coveney updated PIG-2764:
--

Attachment: PIG-2764-2.patch
PIG-2764-2_nows.patch

Updated! And I fixed a couple of errors. Merging in the DateTime patch took a 
little tlc, but it should be fully functional now. I don't know that I love 
that the DateTime patch let's you cast ints to DateTime and so on, but I 
followed that convention.

I would love some eyes on this. It really needs more tests...I can tackle that 
eventually, but wouldn't mind some guidance one what is critical to test.

 Add a biginteger and bigdecimal type to pig
 ---

 Key: PIG-2764
 URL: https://issues.apache.org/jira/browse/PIG-2764
 Project: Pig
  Issue Type: Improvement
Reporter: Jonathan Coveney
Assignee: Jonathan Coveney
 Attachments: fixedpoint.patch, PIG-2764-0.patch, PIG-2764-1.patch, 
 PIG-2764-2_nows.patch, PIG-2764-2.patch


 I think it would be useful for applications where precision is more important 
 than speed to have the option of using java's bigdecimal and biginteger types 
 natively.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira