Re: Welcome to our new Pig PMC member Xuefu Zhang

[Jarek Jarcec Cecho]

Congratulations Xuefu!

Jarcec

> On Feb 24, 2016, at 1:29 PM, Rohini Palaniswamy  
> wrote:
> 
> It is my pleasure to announce that Xuefu Zhang is our newest addition to
> the Pig PMC. Xuefu is a long time committer of Pig and has been actively
> involved in driving the Pig on Spark effort for the past year.
> 
> Please join me in congratulating Xuefu !!!
> 
> Regards,
> Rohini

[jira] [Updated] (PIG-2995) Refactor unit test temporary file allocation patterns to use FileLocalizer.getTemporaryPath

[Jarek Jarcec Cecho (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PIG-2995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jarek Jarcec Cecho updated PIG-2995:

Assignee: (was: Jarek Jarcec Cecho)

> Refactor unit test temporary file allocation patterns to use 
> FileLocalizer.getTemporaryPath
> ---
>
> Key: PIG-2995
> URL: https://issues.apache.org/jira/browse/PIG-2995
> Project: Pig
>  Issue Type: Bug
>  Components: build
>Affects Versions: 0.10.0
>Reporter: John Gordon
>Priority: Minor
>
> Pig unit tests have a lot of diverse patterns for temporary file allocation.  
> Not all of them are best practices.  There is an abstraction that could house 
> best practices for test temporary file allocation -- 
> FileLocalizer.getTemporaryPath.  With this, we should be able to have 
> all/most of the temporary file usage fall under just a few methods that can 
> handle arbitrary pig contexts and provide more flexibility around testing pig 
> with different fs implementations.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (PIG-3187) Add -tagFile and -tagPath options to AvroStorage

[Jarek Jarcec Cecho (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PIG-3187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jarek Jarcec Cecho updated PIG-3187:

Assignee: (was: Jarek Jarcec Cecho)

> Add -tagFile and -tagPath options to AvroStorage
> 
>
> Key: PIG-3187
> URL: https://issues.apache.org/jira/browse/PIG-3187
> Project: Pig
>  Issue Type: Improvement
>Affects Versions: 0.11
>Reporter: Cheolsoo Park
>Priority: Minor
>
> PigStorage already supports these options, and it can be useful in 
> AvroStorage as well in particular when we want to filter records by filenames.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (PIG-2591) Unit tests should not write to /tmp but respect java.io.tmpdir

[Jarek Jarcec Cecho (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PIG-2591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jarek Jarcec Cecho updated PIG-2591:

Assignee: (was: Jarek Jarcec Cecho)

> Unit tests should not write to /tmp but respect java.io.tmpdir
> --
>
> Key: PIG-2591
> URL: https://issues.apache.org/jira/browse/PIG-2591
> Project: Pig
>  Issue Type: Bug
>  Components: tools
>Reporter: Thomas Weise
> Fix For: 0.16.0
>
> Attachments: PIG-2495.patch, bugPIG-2591.patch
>
>
> Several tests use /tmp but should derive temporary file location from 
> java.io.tmpdir to avoid side effects (java.io.tmpdir is already set to a test 
> run specific location in build.xml)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (PIG-3830) HiveColumnarLoader throwing FileNotFoundException on Hadoop 2

[Jarek Jarcec Cecho (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PIG-3830?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jarek Jarcec Cecho updated PIG-3830:

Assignee: (was: Jarek Jarcec Cecho)

> HiveColumnarLoader throwing FileNotFoundException on Hadoop 2
> -
>
> Key: PIG-3830
> URL: https://issues.apache.org/jira/browse/PIG-3830
> Project: Pig
>  Issue Type: Bug
>Affects Versions: 0.12.0
>    Reporter: Jarek Jarcec Cecho
> Fix For: 0.16.0
>
> Attachments: PIG-3830.patch
>
>
> I've noticed that {{HiveColumnarLoader}} will thrown 
> {{java.io.FileNotFoundException}} when used with glob path on Hadoop 2.0. It 
> will run just fine on Hadoop 1.0:
> {code}
> Failed to parse: java.io.FileNotFoundException: File 
> /home/jarcec/cloudera/repos/pig/contrib/piggybank/java/simpleDataDir1395623312698/*.txt
>  does not exist
>   at 
> org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:198)
>   at org.apache.pig.PigServer$Graph.parseQuery(PigServer.java:1676)
>   at org.apache.pig.PigServer$Graph.registerQuery(PigServer.java:1623)
>   at org.apache.pig.PigServer.registerQuery(PigServer.java:575)
>   at org.apache.pig.PigServer.registerQuery(PigServer.java:588)
>   at 
> org.apache.pig.piggybank.test.storage.TestHiveColumnarLoader.testHdfdsGlobbing(TestHiveColumnarLoader.java:220)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:601)
>   at junit.framework.TestCase.runTest(TestCase.java:176)
>   at junit.framework.TestCase.runBare(TestCase.java:141)
>   at junit.framework.TestResult$1.protect(TestResult.java:122)
>   at junit.framework.TestResult.runProtected(TestResult.java:142)
>   at junit.framework.TestResult.run(TestResult.java:125)
>   at junit.framework.TestCase.run(TestCase.java:129)
>   at junit.framework.TestSuite.runTest(TestSuite.java:255)
>   at junit.framework.TestSuite.run(TestSuite.java:250)
>   at 
> org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:518)
>   at 
> org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1052)
>   at 
> org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:906)
> Caused by: java.lang.RuntimeException: java.io.FileNotFoundException: File 
> /home/jarcec/cloudera/repos/pig/contrib/piggybank/java/simpleDataDir1395623312698/*.txt
>  does not exist
>   at 
> org.apache.hadoop.fs.RawLocalFileSystem.listStatus(RawLocalFileSystem.java:362)
>   at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1484)
>   at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1524)
>   at 
> org.apache.hadoop.fs.ChecksumFileSystem.listStatus(ChecksumFileSystem.java:564)
>   at 
> org.apache.pig.piggybank.storage.partition.PathPartitioner.getPartitionKeys(PathPartitioner.java:105)
>   at 
> org.apache.pig.piggybank.storage.partition.PathPartitionHelper.getPartitionKeys(PathPartitionHelper.java:101)
>   at 
> org.apache.pig.piggybank.storage.HiveColumnarLoader.getPartitionColumns(HiveColumnarLoader.java:576)
>   at 
> org.apache.pig.piggybank.storage.HiveColumnarLoader.getSchema(HiveColumnarLoader.java:646)
>   at 
> org.apache.pig.newplan.logical.relational.LOLoad.getSchemaFromMetaData(LOLoad.java:175)
>   at 
> org.apache.pig.newplan.logical.relational.LOLoad.(LOLoad.java:89)
>   at 
> org.apache.pig.parser.LogicalPlanBuilder.buildLoadOp(LogicalPlanBuilder.java:853)
>   at 
> org.apache.pig.parser.LogicalPlanGenerator.load_clause(LogicalPlanGenerator.java:3479)
>   at 
> org.apache.pig.parser.LogicalPlanGenerator.op_clause(LogicalPlanGenerator.java:1536)
>   at 
> org.apache.pig.parser.LogicalPlanGenerator.general_statement(LogicalPlanGenerator.java:1013)
>   at 
> org.apache.pig.parser.LogicalPlanGenerator.statement(LogicalPlanGenerator.java:553)
>   at 
> org.apache.pig.parser.LogicalPlanGenerator.query(LogicalPlanGenerator.java:421)
>   at 
> org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:188)
>   ... 20 more
> Caused by: java.io.FileNotFoundException: File 
> /home/jarcec/cloudera/repos/pig/contrib/piggybank/java/simpleDataDir1395623312698/*.txt
>  does not exist
>   ..

Re: Welcome our new Pig PMC chair Rohini Palaniswamy

[Jarek Jarcec Cecho]

Congratulations Rohini!

The king is dead, long live the King :)

Jarcec

> On Mar 18, 2015, at 7:28 PM, Xuefu Zhang  wrote:
> 
> Congratulations, Rohini!
> 
> --Xuefu
> 
> On Wed, Mar 18, 2015 at 6:48 PM, Cheolsoo Park  wrote:
> 
>> Hi all,
>> 
>> Now it's official that Rohini Palaniswamy is our new Pig PMC chair. Please
>> join me in congratulating Rohini for her new role. Congrats!
>> 
>> Thanks!
>> Cheolsoo
>> 

Re: [VOTE] Drop support for Hadoop 0.20 from Pig 0.14

[Jarek Jarcec Cecho]

+1 (non-binding)

Jarcec

On Sep 17, 2014, at 6:38 AM, Rohini Palaniswamy  wrote:

> Hi,
>   Hadoop has matured far from Hadoop 0.20 and has had two major releases
> after that and there has been no development on branch-0.20 (
> http://svn.apache.org/viewvc/hadoop/common/branches/branch-0.20/) for 3
> years now. It is high time we drop support for Hadoop 0.20 and only support
> Hadoop 1.x and 2.x lines going forward. This will reduce the maintenance
> effort and also enable us to right more efficient code and cut down on
> reflections.
> 
> Vote closes on Tuesday, Sep 23 2014.
> 
> Thanks,
> Rohini

Re: [VOTE] Drop support for JDK 6 from Pig 0.14

[Jarek Jarcec Cecho]

+1 (non-binding)

Jarcec

On Sep 17, 2014, at 7:09 AM, Julien Le Dem  wrote:

> +1
> 
> On Tuesday, September 16, 2014, Rohini Palaniswamy 
> wrote:
> 
>> Hi,
>>   Hadoop is dropping support for JDK6 from hadoop-2.7 this year as
>> mentioned in the mail below. Pig should also move to JDK7 to be able to
>> compile against future hadoop 2.x releases and start making releases with
>> jars (binaries, maven repo) compiled in JDK 7. This would also open it up
>> for developers to code with JDK7 specific APIs.
>> 
>> Vote closes on Tuesday, Sep 23 2014.
>> 
>> Thanks,
>> Rohini
>> 
>> 
>> 
>> 
>> -- Forwarded message --
>> From: Arun C Murthy >
>> Date: Tue, Aug 19, 2014 at 10:52 AM
>> Subject: Dropping support for JDK6 in Apache Hadoop
>> To: "d...@hbase.apache.org " > >, d...@hive.apache.org ,
>> dev@pig.apache.org , d...@oozie.apache.org 
>> Cc: "common-...@hadoop.apache.org " <
>> common-...@hadoop.apache.org >
>> 
>> 
>> [Apologies for the wide distribution.]
>> 
>> Dear HBase/Hive/Pig/Oozie communities,
>> 
>> We, over at Hadoop are considering dropping support for JDK6 this year.
>> 
>> As you maybe aware we just released hadoop-2.5.0 and are now considering
>> making the next release i.e. hadoop-2.6.0 the *last* release of Apache
>> Hadoop which supports JDK6. This means, from hadoop-2.7.0 onwards we will
>> not support JDK6 anymore and we *may* start relying on JDK7-specific apis.
>> 
>> Now, the above releases a proposal and we do not want to pull the trigger
>> without talking to projects downstream - hence the request for you
>> feedback.
>> 
>> Please feel free to forward this to other communities you might deem to be
>> at risk from this too.
>> 
>> thanks,
>> Arun
>> 

Re: Creating a branch for Pig on Spark (PIG-4059)

[Jarek Jarcec Cecho]

I would second Mayur - as he and Praveen are most familiar with all the 
changes, I would let them be the only additional branch committers. I’m 
expecting that they will review other contributor's patches similarly as normal 
committers and hence the contributors will start building Karma to get the 
“branch committership” later as well (if the branch will be around for that 
long).

Jarcec

On Aug 26, 2014, at 9:04 AM, Mayur Rustagi  wrote:

> Praveen & me should be good enough for us. We will have 1-2 more folks
> contributing from Sigmoid but we can manage by merging patches from them.
> 
>Initially it will be good to structure patches through us as we
> would like to avoid breaking architecture & flow.
> 
> Regards
> Mayur
> 
> 
> On Tue, Aug 26, 2014 at 8:27 AM, Cheolsoo Park  wrote:
> 
>> Hi guys,
>> 
>> I asked about branch committership to the infra mailing list, and here is
>> the reply-
>> 
>> Many projects have what they consider 'partial committers' that is
>> folks who have access to specific parts of a projects svn tree. Some
>> projects do this for GSoC participants, others as a mechanism for
>> moving to 'full committership' within the project.
>> 
>> Do note though that in the eyes of the ASF someone with an ICLA and an
>> account with any permissions to commit code anywhere in the public svn
>> tree is a committer. IOW, you would vote, have ICLAs filed, and
>> request account creation as per normal, and then merely adjust the
>> karma in asf-authorization-template (and or LDAP)
>> 
>> 
>> Looks like we need to vote and follow the normal process just like any
>> other new committer.
>> 
>> @Praveen, Jacec,
>> I think Mayur and Praveen from Sigmoid Analytics need branch committership.
>> Will anyone else work on Pig-on-Spark? Please reply.
>> 
>> Once I have a full list of people, I will open a vote for Pig PMCs.
>> 
>> Thanks,
>> Cheolsoo
>> 
>> 
>> On Mon, Aug 25, 2014 at 11:51 AM, Cheolsoo Park 
>> wrote:
>> 
>>> Additionally, I will give "branch-specific" commit permission to people
>>> who will work on Pig on Spark (assuming it is possible).
>>> 
>>> Please let me know if you have any objection on this too.
>>> 
>>> 
>>> On Mon, Aug 25, 2014 at 10:25 AM, Jarek Jarcec Cecho 
>>> wrote:
>>> 
>>>> No objections from my side, thank you for creating the branch Cheolsoo
>>>> and kudos to the Sigmoid Analytics team for the great work!
>>>> 
>>>> Jarcec
>>>> 
>>>> On Aug 25, 2014, at 7:14 PM, Cheolsoo Park 
>> wrote:
>>>> 
>>>>> Hi devs,
>>>>> 
>>>>> Sigmoid Analytics has been working on Pig-on-Spark (PIG-4059), and
>> they
>>>> want to merge their work into Apache.
>>>>> 
>>>>> I am going to create a "Spark" branch for them. Please let me know if
>>>> you have any concerns.
>>>>> 
>>>>> Thanks,
>>>>> Cheolsoo
>>>> 
>>>> 
>>> 
>> 
> 
> 
> 
> -- 
> Regards,
> Mayur Rustagi
> Ph: +1 (760) 203 3257
> http://www.sigmoidanalytics.com
> @mayur_rustagi <http://www.twitter.com/mayur_rustagi>

Re: Creating a branch for Pig on Spark (PIG-4059)

[Jarek Jarcec Cecho]

No objections from my side, thank you for creating the branch Cheolsoo and 
kudos to the Sigmoid Analytics team for the great work!

Jarcec

On Aug 25, 2014, at 7:14 PM, Cheolsoo Park  wrote:

> Hi devs,
> 
> Sigmoid Analytics has been working on Pig-on-Spark (PIG-4059), and they want 
> to merge their work into Apache.
> 
> I am going to create a "Spark" branch for them. Please let me know if you 
> have any concerns.
> 
> Thanks,
> Cheolsoo

[jira] [Commented] (PIG-4059) Pig on Spark

[Jarek Jarcec Cecho (JIRA)]

[ 
https://issues.apache.org/jira/browse/PIG-4059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14065084#comment-14065084
 ] 

Jarek Jarcec Cecho commented on PIG-4059:
-

I would love to see ability to run Pig on top of Spark, hence I've voted for 
this JIRA :-)

> Pig on Spark
> 
>
> Key: PIG-4059
> URL: https://issues.apache.org/jira/browse/PIG-4059
> Project: Pig
>  Issue Type: New Feature
>Reporter: Rohini Palaniswamy
>
>There is lot of interest in adding Spark as a backend execution engine for 
> Pig. 



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Re: [ANNOUNCE] Welcome new Pig Committer - Lorand Bendig

[Jarek Jarcec Cecho]

Congratulations Lorand, well deserved!

Jarcec

On Sat, Jun 21, 2014 at 10:30:01PM -0700, Cheolsoo Park wrote:
> It is my pleasure to announce that Lorand Bendig became the newest addition
> to the Pig Committers! Lorand has been actively contributing to Pig for a
> year now.
> 
> Please join me in congratulating Lorand!


signature.asc
Description: Digital signature

[jira] [Commented] (PIG-4005) depend on hbase-hadoop2-compat rather than hbase-hadoop1-compat when hbaseversion is 95

[Jarek Jarcec Cecho (JIRA)]

[ 
https://issues.apache.org/jira/browse/PIG-4005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14026575#comment-14026575
 ] 

Jarek Jarcec Cecho commented on PIG-4005:
-

That depends on Hadoop version that you're using and not on HBase version. If 
you're using Hadoop 1.x, then you should use {{hbase-hadoop1-compat}}, if 
you're using Hadoop 2.x then you should use {{hbase-hadoop2-compat}}.

> depend on hbase-hadoop2-compat rather than hbase-hadoop1-compat when 
> hbaseversion is 95
> ---
>
> Key: PIG-4005
> URL: https://issues.apache.org/jira/browse/PIG-4005
> Project: Pig
>  Issue Type: Wish
>Affects Versions: 0.12.0
>Reporter: John
>
> If users set hbaseversion to 95, shall we use hbase-hadoop2-compat instead of 
> hbase-hadoop1-compat?



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (PIG-3830) HiveColumnarLoader throwing FileNotFoundException on Hadoop 2

[Jarek Jarcec Cecho (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PIG-3830?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jarek Jarcec Cecho updated PIG-3830:


Status: Patch Available  (was: Open)

I've forgot to switch the status to "Patch available". Thank you for catching 
it, [~cheolsoo]!

> HiveColumnarLoader throwing FileNotFoundException on Hadoop 2
> -
>
> Key: PIG-3830
> URL: https://issues.apache.org/jira/browse/PIG-3830
> Project: Pig
>  Issue Type: Bug
>Affects Versions: 0.12.0
>Reporter: Jarek Jarcec Cecho
>Assignee: Jarek Jarcec Cecho
> Fix For: 0.13.0
>
> Attachments: PIG-3830.patch
>
>
> I've noticed that {{HiveColumnarLoader}} will thrown 
> {{java.io.FileNotFoundException}} when used with glob path on Hadoop 2.0. It 
> will run just fine on Hadoop 1.0:
> {code}
> Failed to parse: java.io.FileNotFoundException: File 
> /home/jarcec/cloudera/repos/pig/contrib/piggybank/java/simpleDataDir1395623312698/*.txt
>  does not exist
>   at 
> org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:198)
>   at org.apache.pig.PigServer$Graph.parseQuery(PigServer.java:1676)
>   at org.apache.pig.PigServer$Graph.registerQuery(PigServer.java:1623)
>   at org.apache.pig.PigServer.registerQuery(PigServer.java:575)
>   at org.apache.pig.PigServer.registerQuery(PigServer.java:588)
>   at 
> org.apache.pig.piggybank.test.storage.TestHiveColumnarLoader.testHdfdsGlobbing(TestHiveColumnarLoader.java:220)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:601)
>   at junit.framework.TestCase.runTest(TestCase.java:176)
>   at junit.framework.TestCase.runBare(TestCase.java:141)
>   at junit.framework.TestResult$1.protect(TestResult.java:122)
>   at junit.framework.TestResult.runProtected(TestResult.java:142)
>   at junit.framework.TestResult.run(TestResult.java:125)
>   at junit.framework.TestCase.run(TestCase.java:129)
>   at junit.framework.TestSuite.runTest(TestSuite.java:255)
>   at junit.framework.TestSuite.run(TestSuite.java:250)
>   at 
> org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:518)
>   at 
> org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1052)
>   at 
> org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:906)
> Caused by: java.lang.RuntimeException: java.io.FileNotFoundException: File 
> /home/jarcec/cloudera/repos/pig/contrib/piggybank/java/simpleDataDir1395623312698/*.txt
>  does not exist
>   at 
> org.apache.hadoop.fs.RawLocalFileSystem.listStatus(RawLocalFileSystem.java:362)
>   at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1484)
>   at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1524)
>   at 
> org.apache.hadoop.fs.ChecksumFileSystem.listStatus(ChecksumFileSystem.java:564)
>   at 
> org.apache.pig.piggybank.storage.partition.PathPartitioner.getPartitionKeys(PathPartitioner.java:105)
>   at 
> org.apache.pig.piggybank.storage.partition.PathPartitionHelper.getPartitionKeys(PathPartitionHelper.java:101)
>   at 
> org.apache.pig.piggybank.storage.HiveColumnarLoader.getPartitionColumns(HiveColumnarLoader.java:576)
>   at 
> org.apache.pig.piggybank.storage.HiveColumnarLoader.getSchema(HiveColumnarLoader.java:646)
>   at 
> org.apache.pig.newplan.logical.relational.LOLoad.getSchemaFromMetaData(LOLoad.java:175)
>   at 
> org.apache.pig.newplan.logical.relational.LOLoad.(LOLoad.java:89)
>   at 
> org.apache.pig.parser.LogicalPlanBuilder.buildLoadOp(LogicalPlanBuilder.java:853)
>   at 
> org.apache.pig.parser.LogicalPlanGenerator.load_clause(LogicalPlanGenerator.java:3479)
>   at 
> org.apache.pig.parser.LogicalPlanGenerator.op_clause(LogicalPlanGenerator.java:1536)
>   at 
> org.apache.pig.parser.LogicalPlanGenerator.general_statement(LogicalPlanGenerator.java:1013)
>   at 
> org.apache.pig.parser.LogicalPlanGenerator.statement(LogicalPlanGenerator.java:553)
>   at 
> org.apache.pig.parser.LogicalPlanGenerator.query(LogicalPlanGenerator.java:421)
>   at 
> org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:188)
>   ... 20 more
> Caused by: java.io.Fil

[jira] [Updated] (PIG-3830) HiveColumnarLoader throwing FileNotFoundException on Hadoop 2

[Jarek Jarcec Cecho (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PIG-3830?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jarek Jarcec Cecho updated PIG-3830:


Attachment: PIG-3830.patch

Attaching patch that:

* Add catch block for the {{FileNotFound}} exception to gracefully handle the 
situation on Hadoop 2.0.
* Fix formatting of the affected method {{getPartitionKeys}}. Entire file is 
not using the usual formatting guidelines that we are using in Pig, so I've 
fixed at least the method I was changing.
* Added {{test.output}} variable to piggybank's {{build.xml}} file as it was 
missing

> HiveColumnarLoader throwing FileNotFoundException on Hadoop 2
> -
>
> Key: PIG-3830
> URL: https://issues.apache.org/jira/browse/PIG-3830
> Project: Pig
>  Issue Type: Bug
>Affects Versions: 0.12.0
>    Reporter: Jarek Jarcec Cecho
>Assignee: Jarek Jarcec Cecho
> Fix For: 0.13.0
>
> Attachments: PIG-3830.patch
>
>
> I've noticed that {{HiveColumnarLoader}} will thrown 
> {{java.io.FileNotFoundException}} when used with glob path on Hadoop 2.0. It 
> will run just fine on Hadoop 1.0:
> {code}
> Failed to parse: java.io.FileNotFoundException: File 
> /home/jarcec/cloudera/repos/pig/contrib/piggybank/java/simpleDataDir1395623312698/*.txt
>  does not exist
>   at 
> org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:198)
>   at org.apache.pig.PigServer$Graph.parseQuery(PigServer.java:1676)
>   at org.apache.pig.PigServer$Graph.registerQuery(PigServer.java:1623)
>   at org.apache.pig.PigServer.registerQuery(PigServer.java:575)
>   at org.apache.pig.PigServer.registerQuery(PigServer.java:588)
>   at 
> org.apache.pig.piggybank.test.storage.TestHiveColumnarLoader.testHdfdsGlobbing(TestHiveColumnarLoader.java:220)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:601)
>   at junit.framework.TestCase.runTest(TestCase.java:176)
>   at junit.framework.TestCase.runBare(TestCase.java:141)
>   at junit.framework.TestResult$1.protect(TestResult.java:122)
>   at junit.framework.TestResult.runProtected(TestResult.java:142)
>   at junit.framework.TestResult.run(TestResult.java:125)
>   at junit.framework.TestCase.run(TestCase.java:129)
>   at junit.framework.TestSuite.runTest(TestSuite.java:255)
>   at junit.framework.TestSuite.run(TestSuite.java:250)
>   at 
> org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:518)
>   at 
> org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1052)
>   at 
> org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:906)
> Caused by: java.lang.RuntimeException: java.io.FileNotFoundException: File 
> /home/jarcec/cloudera/repos/pig/contrib/piggybank/java/simpleDataDir1395623312698/*.txt
>  does not exist
>   at 
> org.apache.hadoop.fs.RawLocalFileSystem.listStatus(RawLocalFileSystem.java:362)
>   at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1484)
>   at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1524)
>   at 
> org.apache.hadoop.fs.ChecksumFileSystem.listStatus(ChecksumFileSystem.java:564)
>   at 
> org.apache.pig.piggybank.storage.partition.PathPartitioner.getPartitionKeys(PathPartitioner.java:105)
>   at 
> org.apache.pig.piggybank.storage.partition.PathPartitionHelper.getPartitionKeys(PathPartitionHelper.java:101)
>   at 
> org.apache.pig.piggybank.storage.HiveColumnarLoader.getPartitionColumns(HiveColumnarLoader.java:576)
>   at 
> org.apache.pig.piggybank.storage.HiveColumnarLoader.getSchema(HiveColumnarLoader.java:646)
>   at 
> org.apache.pig.newplan.logical.relational.LOLoad.getSchemaFromMetaData(LOLoad.java:175)
>   at 
> org.apache.pig.newplan.logical.relational.LOLoad.(LOLoad.java:89)
>   at 
> org.apache.pig.parser.LogicalPlanBuilder.buildLoadOp(LogicalPlanBuilder.java:853)
>   at 
> org.apache.pig.parser.LogicalPlanGenerator.load_clause(LogicalPlanGenerator.java:3479)
>   at 
> org.apache.pig.parser.LogicalPlanGenerator.op_clause(LogicalPlanGenerator.java:1536)
>   at 
> org.apache.pig.parser.LogicalPlanGenerator.general_statement(LogicalPlanGenerator.java:1013)
>   at 
> org.apache.pig.parser.Logic

[jira] [Created] (PIG-3830) HiveColumnarLoader throwing FileNotFoundException on Hadoop 2

[Jarek Jarcec Cecho (JIRA)]

Jarek Jarcec Cecho created PIG-3830:
---

 Summary: HiveColumnarLoader throwing FileNotFoundException on 
Hadoop 2
 Key: PIG-3830
 URL: https://issues.apache.org/jira/browse/PIG-3830
 Project: Pig
  Issue Type: Bug
Affects Versions: 0.12.0
Reporter: Jarek Jarcec Cecho
Assignee: Jarek Jarcec Cecho
 Fix For: 0.13.0


I've noticed that {{HiveColumnarLoader}} will thrown 
{{java.io.FileNotFoundException}} when used with glob path on Hadoop 2.0. It 
will run just fine on Hadoop 1.0:

{code}
Failed to parse: java.io.FileNotFoundException: File 
/home/jarcec/cloudera/repos/pig/contrib/piggybank/java/simpleDataDir1395623312698/*.txt
 does not exist
at 
org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:198)
at org.apache.pig.PigServer$Graph.parseQuery(PigServer.java:1676)
at org.apache.pig.PigServer$Graph.registerQuery(PigServer.java:1623)
at org.apache.pig.PigServer.registerQuery(PigServer.java:575)
at org.apache.pig.PigServer.registerQuery(PigServer.java:588)
at 
org.apache.pig.piggybank.test.storage.TestHiveColumnarLoader.testHdfdsGlobbing(TestHiveColumnarLoader.java:220)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at junit.framework.TestCase.runTest(TestCase.java:176)
at junit.framework.TestCase.runBare(TestCase.java:141)
at junit.framework.TestResult$1.protect(TestResult.java:122)
at junit.framework.TestResult.runProtected(TestResult.java:142)
at junit.framework.TestResult.run(TestResult.java:125)
at junit.framework.TestCase.run(TestCase.java:129)
at junit.framework.TestSuite.runTest(TestSuite.java:255)
at junit.framework.TestSuite.run(TestSuite.java:250)
at 
org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:518)
at 
org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1052)
at 
org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:906)
Caused by: java.lang.RuntimeException: java.io.FileNotFoundException: File 
/home/jarcec/cloudera/repos/pig/contrib/piggybank/java/simpleDataDir1395623312698/*.txt
 does not exist
at 
org.apache.hadoop.fs.RawLocalFileSystem.listStatus(RawLocalFileSystem.java:362)
at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1484)
at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1524)
at 
org.apache.hadoop.fs.ChecksumFileSystem.listStatus(ChecksumFileSystem.java:564)
at 
org.apache.pig.piggybank.storage.partition.PathPartitioner.getPartitionKeys(PathPartitioner.java:105)
at 
org.apache.pig.piggybank.storage.partition.PathPartitionHelper.getPartitionKeys(PathPartitionHelper.java:101)
at 
org.apache.pig.piggybank.storage.HiveColumnarLoader.getPartitionColumns(HiveColumnarLoader.java:576)
at 
org.apache.pig.piggybank.storage.HiveColumnarLoader.getSchema(HiveColumnarLoader.java:646)
at 
org.apache.pig.newplan.logical.relational.LOLoad.getSchemaFromMetaData(LOLoad.java:175)
at 
org.apache.pig.newplan.logical.relational.LOLoad.(LOLoad.java:89)
at 
org.apache.pig.parser.LogicalPlanBuilder.buildLoadOp(LogicalPlanBuilder.java:853)
at 
org.apache.pig.parser.LogicalPlanGenerator.load_clause(LogicalPlanGenerator.java:3479)
at 
org.apache.pig.parser.LogicalPlanGenerator.op_clause(LogicalPlanGenerator.java:1536)
at 
org.apache.pig.parser.LogicalPlanGenerator.general_statement(LogicalPlanGenerator.java:1013)
at 
org.apache.pig.parser.LogicalPlanGenerator.statement(LogicalPlanGenerator.java:553)
at 
org.apache.pig.parser.LogicalPlanGenerator.query(LogicalPlanGenerator.java:421)
at 
org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:188)
... 20 more
Caused by: java.io.FileNotFoundException: File 
/home/jarcec/cloudera/repos/pig/contrib/piggybank/java/simpleDataDir1395623312698/*.txt
 does not exist
... 37 more
{code}

I've dived into the problem and found a difference in Hadoop implementation of 
{{DistributedFileSystem}}. For non existing directory method {{listStatus}} 
will return {{null}} in [Hadoop 
1|https://github.com/apache/hadoop-common/blob/branch-1/src/hdfs/org/apache/hadoop/hdfs/DistributedFileSystem.java#L316]:

{code}
if (thisListing == null) { // the directory does not exist
  return null;
}
{code}

But will thrown an exception in [Hadoop 
2|https://github.com/apache/hadoop-common/blob/branch-2/hadoop-hdfs-project/h

Re: Welcome to the newest Pig Committer - Mark Wagner

[Jarek Jarcec Cecho]

Congratulations Mark, good job!

Jarcec

On Fri, Jan 31, 2014 at 05:20:26PM -0800, Julien Le Dem wrote:
> It is my pleasure to announce that Mark Wagner became the newest addition to 
> the Pig Committers!
> Mark has been actively contributing to Pig and in particular to the 
> Pig-on-Tez effort.
> Please, join me in congratulating Mark!


signature.asc
Description: Digital signature

[jira] [Commented] (PIG-3726) Ranking empty records leads to NullPointerException

[Jarek Jarcec Cecho (JIRA)]

[ 
https://issues.apache.org/jira/browse/PIG-3726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13888008#comment-13888008
 ] 

Jarek Jarcec Cecho commented on PIG-3726:
-

Thank you [~daijy] and [~mtelizhyn]!

> Ranking empty records leads to NullPointerException
> ---
>
> Key: PIG-3726
> URL: https://issues.apache.org/jira/browse/PIG-3726
> Project: Pig
>  Issue Type: Bug
>Affects Versions: 0.12.0
>Reporter: Mykhaylo Telizhyn
>    Assignee: Jarek Jarcec Cecho
> Fix For: 0.13.0
>
> Attachments: PIG-3726.patch, PIG-3726.patch, emptyRecordsNPE.patch
>
>
> steps to reproduce:
> 1) create an empty file on hdfs (e.g. /user/root/test/empty.txt)
> 2) execute the following pig script:
> records = LOAD '/user/root/test/empty.txt' AS (Amount: DOUBLE);
> ranked = RANK records BY Amount DESC;
> DUMP ranked;
> 3) on console you will get " ERROR org.apache.pig.tools.grunt.Grunt - ERROR 
> 2043: Unexpected error during execution." and in pig's logs you will see:
> Pig Stack Trace
> {code}
> ---
> ERROR 2043: Unexpected error during execution.
> org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066: Unable to 
> open iterator for alias ranked
> at org.apache.pig.PigServer.openIterator(PigServer.java:880)
> at org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:774)
> at 
> org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:372)
> at 
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:198)
> at 
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:173)
> at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:84)
> at org.apache.pig.Main.run(Main.java:607)
> at org.apache.pig.Main.main(Main.java:156)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
> Caused by: org.apache.pig.PigException: ERROR 1002: Unable to store alias 
> ranked
> at org.apache.pig.PigServer.storeEx(PigServer.java:982)
> at org.apache.pig.PigServer.store(PigServer.java:942)
> at org.apache.pig.PigServer.openIterator(PigServer.java:855)
> ... 12 more
> Caused by: org.apache.pig.backend.executionengine.ExecException: ERROR 2043: 
> Unexpected error during execution.
> at org.apache.pig.PigServer.launchPlan(PigServer.java:1333)
> at org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1307)
> at org.apache.pig.PigServer.storeEx(PigServer.java:978)
> ... 14 more
> Caused by: java.lang.RuntimeException: Error to read counters into Rank 
> operation counterSize 0
> at 
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.saveCounters(JobControlCompiler.java:386)
> at 
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.updateMROpPlan(JobControlCompiler.java:332)
> at 
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:387)
> at org.apache.pig.PigServer.launchPlan(PigServer.java:1322)
> ... 16 more
> Caused by: java.lang.NullPointerException
> at org.apache.hadoop.mapreduce.counters.Limits.filterName(Limits.java:44)
> at org.apache.hadoop.mapreduce.counters.Limits.filterGroupName(Limits.java:52)
> at 
> org.apache.hadoop.mapreduce.counters.AbstractCounters.getGroup(AbstractCounters.java:220)
> at org.apache.hadoop.mapred.Counters.getGroup(Counters.java:113)
> at 
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.saveCounters(JobControlCompiler.java:360)
> ... 19 more
> 
> {code}



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Updated] (PIG-3726) Ranking empty records leads to NullPointerException

[Jarek Jarcec Cecho (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PIG-3726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jarek Jarcec Cecho updated PIG-3726:


Fix Version/s: 0.13.0
Affects Version/s: 0.12.0
   Status: Patch Available  (was: Open)

> Ranking empty records leads to NullPointerException
> ---
>
> Key: PIG-3726
> URL: https://issues.apache.org/jira/browse/PIG-3726
> Project: Pig
>  Issue Type: Bug
>Affects Versions: 0.12.0
>Reporter: Mykhaylo Telizhyn
> Fix For: 0.13.0
>
> Attachments: PIG-3726.patch, PIG-3726.patch, emptyRecordsNPE.patch
>
>
> steps to reproduce:
> 1) create an empty file on hdfs (e.g. /user/root/test/empty.txt)
> 2) execute the following pig script:
> records = LOAD '/user/root/test/empty.txt' AS (Amount: DOUBLE);
> ranked = RANK records BY Amount DESC;
> DUMP ranked;
> 3) on console you will get " ERROR org.apache.pig.tools.grunt.Grunt - ERROR 
> 2043: Unexpected error during execution." and in pig's logs you will see:
> Pig Stack Trace
> {code}
> ---
> ERROR 2043: Unexpected error during execution.
> org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066: Unable to 
> open iterator for alias ranked
> at org.apache.pig.PigServer.openIterator(PigServer.java:880)
> at org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:774)
> at 
> org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:372)
> at 
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:198)
> at 
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:173)
> at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:84)
> at org.apache.pig.Main.run(Main.java:607)
> at org.apache.pig.Main.main(Main.java:156)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
> Caused by: org.apache.pig.PigException: ERROR 1002: Unable to store alias 
> ranked
> at org.apache.pig.PigServer.storeEx(PigServer.java:982)
> at org.apache.pig.PigServer.store(PigServer.java:942)
> at org.apache.pig.PigServer.openIterator(PigServer.java:855)
> ... 12 more
> Caused by: org.apache.pig.backend.executionengine.ExecException: ERROR 2043: 
> Unexpected error during execution.
> at org.apache.pig.PigServer.launchPlan(PigServer.java:1333)
> at org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1307)
> at org.apache.pig.PigServer.storeEx(PigServer.java:978)
> ... 14 more
> Caused by: java.lang.RuntimeException: Error to read counters into Rank 
> operation counterSize 0
> at 
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.saveCounters(JobControlCompiler.java:386)
> at 
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.updateMROpPlan(JobControlCompiler.java:332)
> at 
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:387)
> at org.apache.pig.PigServer.launchPlan(PigServer.java:1322)
> ... 16 more
> Caused by: java.lang.NullPointerException
> at org.apache.hadoop.mapreduce.counters.Limits.filterName(Limits.java:44)
> at org.apache.hadoop.mapreduce.counters.Limits.filterGroupName(Limits.java:52)
> at 
> org.apache.hadoop.mapreduce.counters.AbstractCounters.getGroup(AbstractCounters.java:220)
> at org.apache.hadoop.mapred.Counters.getGroup(Counters.java:113)
> at 
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.saveCounters(JobControlCompiler.java:360)
> ... 19 more
> 
> {code}



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Assigned] (PIG-3726) Ranking empty records leads to NullPointerException

[Jarek Jarcec Cecho (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PIG-3726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jarek Jarcec Cecho reassigned PIG-3726:
---

Assignee: Jarek Jarcec Cecho

> Ranking empty records leads to NullPointerException
> ---
>
> Key: PIG-3726
> URL: https://issues.apache.org/jira/browse/PIG-3726
> Project: Pig
>  Issue Type: Bug
>Affects Versions: 0.12.0
>Reporter: Mykhaylo Telizhyn
>    Assignee: Jarek Jarcec Cecho
> Fix For: 0.13.0
>
> Attachments: PIG-3726.patch, PIG-3726.patch, emptyRecordsNPE.patch
>
>
> steps to reproduce:
> 1) create an empty file on hdfs (e.g. /user/root/test/empty.txt)
> 2) execute the following pig script:
> records = LOAD '/user/root/test/empty.txt' AS (Amount: DOUBLE);
> ranked = RANK records BY Amount DESC;
> DUMP ranked;
> 3) on console you will get " ERROR org.apache.pig.tools.grunt.Grunt - ERROR 
> 2043: Unexpected error during execution." and in pig's logs you will see:
> Pig Stack Trace
> {code}
> ---
> ERROR 2043: Unexpected error during execution.
> org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066: Unable to 
> open iterator for alias ranked
> at org.apache.pig.PigServer.openIterator(PigServer.java:880)
> at org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:774)
> at 
> org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:372)
> at 
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:198)
> at 
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:173)
> at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:84)
> at org.apache.pig.Main.run(Main.java:607)
> at org.apache.pig.Main.main(Main.java:156)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
> Caused by: org.apache.pig.PigException: ERROR 1002: Unable to store alias 
> ranked
> at org.apache.pig.PigServer.storeEx(PigServer.java:982)
> at org.apache.pig.PigServer.store(PigServer.java:942)
> at org.apache.pig.PigServer.openIterator(PigServer.java:855)
> ... 12 more
> Caused by: org.apache.pig.backend.executionengine.ExecException: ERROR 2043: 
> Unexpected error during execution.
> at org.apache.pig.PigServer.launchPlan(PigServer.java:1333)
> at org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1307)
> at org.apache.pig.PigServer.storeEx(PigServer.java:978)
> ... 14 more
> Caused by: java.lang.RuntimeException: Error to read counters into Rank 
> operation counterSize 0
> at 
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.saveCounters(JobControlCompiler.java:386)
> at 
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.updateMROpPlan(JobControlCompiler.java:332)
> at 
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:387)
> at org.apache.pig.PigServer.launchPlan(PigServer.java:1322)
> ... 16 more
> Caused by: java.lang.NullPointerException
> at org.apache.hadoop.mapreduce.counters.Limits.filterName(Limits.java:44)
> at org.apache.hadoop.mapreduce.counters.Limits.filterGroupName(Limits.java:52)
> at 
> org.apache.hadoop.mapreduce.counters.AbstractCounters.getGroup(AbstractCounters.java:220)
> at org.apache.hadoop.mapred.Counters.getGroup(Counters.java:113)
> at 
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.saveCounters(JobControlCompiler.java:360)
> ... 19 more
> 
> {code}



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Updated] (PIG-3726) Ranking empty records leads to NullPointerException

[Jarek Jarcec Cecho (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PIG-3726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jarek Jarcec Cecho updated PIG-3726:


Attachment: PIG-3726.patch

Thank you for your feedback [~mtelizhyn]. I went ahead and added the check to 
ensure that the only acceptable option for missing RANK counters is when the 
relation is empty.

> Ranking empty records leads to NullPointerException
> ---
>
> Key: PIG-3726
> URL: https://issues.apache.org/jira/browse/PIG-3726
> Project: Pig
>  Issue Type: Bug
>Affects Versions: 0.12.0
>Reporter: Mykhaylo Telizhyn
> Fix For: 0.13.0
>
> Attachments: PIG-3726.patch, PIG-3726.patch, emptyRecordsNPE.patch
>
>
> steps to reproduce:
> 1) create an empty file on hdfs (e.g. /user/root/test/empty.txt)
> 2) execute the following pig script:
> records = LOAD '/user/root/test/empty.txt' AS (Amount: DOUBLE);
> ranked = RANK records BY Amount DESC;
> DUMP ranked;
> 3) on console you will get " ERROR org.apache.pig.tools.grunt.Grunt - ERROR 
> 2043: Unexpected error during execution." and in pig's logs you will see:
> Pig Stack Trace
> {code}
> ---
> ERROR 2043: Unexpected error during execution.
> org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066: Unable to 
> open iterator for alias ranked
> at org.apache.pig.PigServer.openIterator(PigServer.java:880)
> at org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:774)
> at 
> org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:372)
> at 
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:198)
> at 
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:173)
> at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:84)
> at org.apache.pig.Main.run(Main.java:607)
> at org.apache.pig.Main.main(Main.java:156)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
> Caused by: org.apache.pig.PigException: ERROR 1002: Unable to store alias 
> ranked
> at org.apache.pig.PigServer.storeEx(PigServer.java:982)
> at org.apache.pig.PigServer.store(PigServer.java:942)
> at org.apache.pig.PigServer.openIterator(PigServer.java:855)
> ... 12 more
> Caused by: org.apache.pig.backend.executionengine.ExecException: ERROR 2043: 
> Unexpected error during execution.
> at org.apache.pig.PigServer.launchPlan(PigServer.java:1333)
> at org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1307)
> at org.apache.pig.PigServer.storeEx(PigServer.java:978)
> ... 14 more
> Caused by: java.lang.RuntimeException: Error to read counters into Rank 
> operation counterSize 0
> at 
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.saveCounters(JobControlCompiler.java:386)
> at 
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.updateMROpPlan(JobControlCompiler.java:332)
> at 
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:387)
> at org.apache.pig.PigServer.launchPlan(PigServer.java:1322)
> ... 16 more
> Caused by: java.lang.NullPointerException
> at org.apache.hadoop.mapreduce.counters.Limits.filterName(Limits.java:44)
> at org.apache.hadoop.mapreduce.counters.Limits.filterGroupName(Limits.java:52)
> at 
> org.apache.hadoop.mapreduce.counters.AbstractCounters.getGroup(AbstractCounters.java:220)
> at org.apache.hadoop.mapred.Counters.getGroup(Counters.java:113)
> at 
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.saveCounters(JobControlCompiler.java:360)
> ... 19 more
> 
> {code}



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Updated] (PIG-3726) Ranking empty records leads to NullPointerException

[Jarek Jarcec Cecho (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PIG-3726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jarek Jarcec Cecho updated PIG-3726:


Attachment: PIG-3726.patch

Sure [~mtelizhyn], here is patch that I was working on just yesterday!  As you 
can see, it's very similar to yours :-)

The reason why I did not submitted it yet is that I wasn't sure whether my 
(our) approach is entirely correct. Method {{saveCounters}} is written in a way 
to throw an exception if the RANK counters are not present. I'm assuming that 
the counters might be missing from many reasons and the only valid case that we 
want to cover is when the input relation is empty. I wanted poke around a bit 
to see if that is indeed the case. Will be happy to hear others thoughts.

> Ranking empty records leads to NullPointerException
> ---
>
> Key: PIG-3726
> URL: https://issues.apache.org/jira/browse/PIG-3726
> Project: Pig
>  Issue Type: Bug
>Reporter: Mykhaylo Telizhyn
> Attachments: PIG-3726.patch, emptyRecordsNPE.patch
>
>
> steps to reproduce:
> 1) create an empty file on hdfs (e.g. /user/root/test/empty.txt)
> 2) execute the following pig script:
> records = LOAD '/user/root/test/empty.txt' AS (Amount: DOUBLE);
> ranked = RANK records BY Amount DESC;
> DUMP ranked;
> 3) on console you will get " ERROR org.apache.pig.tools.grunt.Grunt - ERROR 
> 2043: Unexpected error during execution." and in pig's logs you will see:
> Pig Stack Trace
> {code}
> ---
> ERROR 2043: Unexpected error during execution.
> org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066: Unable to 
> open iterator for alias ranked
> at org.apache.pig.PigServer.openIterator(PigServer.java:880)
> at org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:774)
> at 
> org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:372)
> at 
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:198)
> at 
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:173)
> at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:84)
> at org.apache.pig.Main.run(Main.java:607)
> at org.apache.pig.Main.main(Main.java:156)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
> Caused by: org.apache.pig.PigException: ERROR 1002: Unable to store alias 
> ranked
> at org.apache.pig.PigServer.storeEx(PigServer.java:982)
> at org.apache.pig.PigServer.store(PigServer.java:942)
> at org.apache.pig.PigServer.openIterator(PigServer.java:855)
> ... 12 more
> Caused by: org.apache.pig.backend.executionengine.ExecException: ERROR 2043: 
> Unexpected error during execution.
> at org.apache.pig.PigServer.launchPlan(PigServer.java:1333)
> at org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1307)
> at org.apache.pig.PigServer.storeEx(PigServer.java:978)
> ... 14 more
> Caused by: java.lang.RuntimeException: Error to read counters into Rank 
> operation counterSize 0
> at 
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.saveCounters(JobControlCompiler.java:386)
> at 
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.updateMROpPlan(JobControlCompiler.java:332)
> at 
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:387)
> at org.apache.pig.PigServer.launchPlan(PigServer.java:1322)
> ... 16 more
> Caused by: java.lang.NullPointerException
> at org.apache.hadoop.mapreduce.counters.Limits.filterName(Limits.java:44)
> at org.apache.hadoop.mapreduce.counters.Limits.filterGroupName(Limits.java:52)
> at 
> org.apache.hadoop.mapreduce.counters.AbstractCounters.getGroup(AbstractCounters.java:220)
> at org.apache.hadoop.mapred.Counters.getGroup(Counters.java:113)
> at 
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.saveCounters(JobControlCompiler.java:360)
> ... 19 more
> 
> {code}



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (PIG-3726) Ranking empty records leads to NullPointerException

[Jarek Jarcec Cecho (JIRA)]

[ 
https://issues.apache.org/jira/browse/PIG-3726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13882896#comment-13882896
 ] 

Jarek Jarcec Cecho commented on PIG-3726:
-

I've noticed the same problem as well and have very similar patch to this one 
:-)

> Ranking empty records leads to NullPointerException
> ---
>
> Key: PIG-3726
> URL: https://issues.apache.org/jira/browse/PIG-3726
> Project: Pig
>  Issue Type: Bug
>Reporter: Mykhaylo Telizhyn
> Attachments: emptyRecordsNPE.patch
>
>
> steps to reproduce:
> 1) create an empty file on hdfs (e.g. /user/root/test/empty.txt)
> 2) execute the following pig script:
> records = LOAD '/user/root/test/empty.txt' AS (Amount: DOUBLE);
> ranked = RANK records BY Amount DESC;
> DUMP ranked;
> 3) on console you will get " ERROR org.apache.pig.tools.grunt.Grunt - ERROR 
> 2043: Unexpected error during execution." and in pig's logs you will see:
> Pig Stack Trace
> ---
> ERROR 2043: Unexpected error during execution.
> org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066: Unable to 
> open iterator for alias ranked
> at org.apache.pig.PigServer.openIterator(PigServer.java:880)
> at org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:774)
> at 
> org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:372)
> at 
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:198)
> at 
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:173)
> at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:84)
> at org.apache.pig.Main.run(Main.java:607)
> at org.apache.pig.Main.main(Main.java:156)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
> Caused by: org.apache.pig.PigException: ERROR 1002: Unable to store alias 
> ranked
> at org.apache.pig.PigServer.storeEx(PigServer.java:982)
> at org.apache.pig.PigServer.store(PigServer.java:942)
> at org.apache.pig.PigServer.openIterator(PigServer.java:855)
> ... 12 more
> Caused by: org.apache.pig.backend.executionengine.ExecException: ERROR 2043: 
> Unexpected error during execution.
> at org.apache.pig.PigServer.launchPlan(PigServer.java:1333)
> at org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1307)
> at org.apache.pig.PigServer.storeEx(PigServer.java:978)
> ... 14 more
> Caused by: java.lang.RuntimeException: Error to read counters into Rank 
> operation counterSize 0
> at 
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.saveCounters(JobControlCompiler.java:386)
> at 
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.updateMROpPlan(JobControlCompiler.java:332)
> at 
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:387)
> at org.apache.pig.PigServer.launchPlan(PigServer.java:1322)
> ... 16 more
> Caused by: java.lang.NullPointerException
> at org.apache.hadoop.mapreduce.counters.Limits.filterName(Limits.java:44)
> at org.apache.hadoop.mapreduce.counters.Limits.filterGroupName(Limits.java:52)
> at 
> org.apache.hadoop.mapreduce.counters.AbstractCounters.getGroup(AbstractCounters.java:220)
> at org.apache.hadoop.mapred.Counters.getGroup(Counters.java:113)
> at 
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.saveCounters(JobControlCompiler.java:360)
> ... 19 more
> 



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Updated] (PIG-3726) Ranking empty records leads to NullPointerException

[Jarek Jarcec Cecho (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PIG-3726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jarek Jarcec Cecho updated PIG-3726:


Description: 
steps to reproduce:

1) create an empty file on hdfs (e.g. /user/root/test/empty.txt)
2) execute the following pig script:

records = LOAD '/user/root/test/empty.txt' AS (Amount: DOUBLE);
ranked = RANK records BY Amount DESC;
DUMP ranked;

3) on console you will get " ERROR org.apache.pig.tools.grunt.Grunt - ERROR 
2043: Unexpected error during execution." and in pig's logs you will see:

Pig Stack Trace
{code}
---
ERROR 2043: Unexpected error during execution.

org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066: Unable to open 
iterator for alias ranked
at org.apache.pig.PigServer.openIterator(PigServer.java:880)
at org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:774)
at 
org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:372)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:198)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:173)
at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:84)
at org.apache.pig.Main.run(Main.java:607)
at org.apache.pig.Main.main(Main.java:156)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
Caused by: org.apache.pig.PigException: ERROR 1002: Unable to store alias ranked
at org.apache.pig.PigServer.storeEx(PigServer.java:982)
at org.apache.pig.PigServer.store(PigServer.java:942)
at org.apache.pig.PigServer.openIterator(PigServer.java:855)
... 12 more
Caused by: org.apache.pig.backend.executionengine.ExecException: ERROR 2043: 
Unexpected error during execution.
at org.apache.pig.PigServer.launchPlan(PigServer.java:1333)
at org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1307)
at org.apache.pig.PigServer.storeEx(PigServer.java:978)
... 14 more
Caused by: java.lang.RuntimeException: Error to read counters into Rank 
operation counterSize 0
at 
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.saveCounters(JobControlCompiler.java:386)
at 
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.updateMROpPlan(JobControlCompiler.java:332)
at 
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:387)
at org.apache.pig.PigServer.launchPlan(PigServer.java:1322)
... 16 more
Caused by: java.lang.NullPointerException
at org.apache.hadoop.mapreduce.counters.Limits.filterName(Limits.java:44)
at org.apache.hadoop.mapreduce.counters.Limits.filterGroupName(Limits.java:52)
at 
org.apache.hadoop.mapreduce.counters.AbstractCounters.getGroup(AbstractCounters.java:220)
at org.apache.hadoop.mapred.Counters.getGroup(Counters.java:113)
at 
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.saveCounters(JobControlCompiler.java:360)
... 19 more

{code}

  was:
steps to reproduce:

1) create an empty file on hdfs (e.g. /user/root/test/empty.txt)
2) execute the following pig script:

records = LOAD '/user/root/test/empty.txt' AS (Amount: DOUBLE);
ranked = RANK records BY Amount DESC;
DUMP ranked;

3) on console you will get " ERROR org.apache.pig.tools.grunt.Grunt - ERROR 
2043: Unexpected error during execution." and in pig's logs you will see:

Pig Stack Trace
---
ERROR 2043: Unexpected error during execution.

org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066: Unable to open 
iterator for alias ranked
at org.apache.pig.PigServer.openIterator(PigServer.java:880)
at org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:774)
at 
org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:372)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:198)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:173)
at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:84)
at org.apache.pig.Main.run(Main.java:607)
at org.apache.pig.Main.main(Main.java:156)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
Caused by: org.apache.pig.PigException: ERROR 1002: Unable to store alias ranked
at org.a

Re: Jira notification spam from me

[Jarek Jarcec Cecho]

Thank you Daniel!

Jarcec

On Thu, Jan 23, 2014 at 01:01:36PM -0800, Daniel Dai wrote:
> Removed.
> 
> On Thu, Jan 23, 2014 at 10:48 AM, Jarek Jarcec Cecho  
> wrote:
> > I've also created couple of  JIRAs while the JIRA was in some sort of weird 
> > state:
> >
> > * PIG-3685
> > * PIG-3684
> > * PIG-3683
> >
> > I would appreciate if someone with sufficient privileges could remove those.
> >
> > Jarcec
> >
> > On Thu, Jan 23, 2014 at 08:15:39AM -0800, Cheolsoo Park wrote:
> >> Hi devs,
> >>
> >> I am sorry about the jira notification spams that I sent to dev mailing
> >> list yesterday and today.
> >>
> >> I accidentally created/deleted the same jiras several times. I was trying
> >> to open a sub-task for pig on tez, but apparently, someone else was doing
> >> at the same time. That causes a strange concurrency issue in jira resulting
> >> in duplicated jiras... I deleted them all today.
> >>
> >> Thanks,
> >> Cheolsoo
> 
> -- 
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to 
> which it is addressed and may contain information that is confidential, 
> privileged and exempt from disclosure under applicable law. If the reader 
> of this message is not the intended recipient, you are hereby notified that 
> any printing, copying, dissemination, distribution, disclosure or 
> forwarding of this communication is strictly prohibited. If you have 
> received this communication in error, please contact the sender immediately 
> and delete it from your system. Thank You.


signature.asc
Description: Digital signature

[jira] [Commented] (PIG-3717) AvroStorage is not filtering hidden files on directory recursion

[Jarek Jarcec Cecho (JIRA)]

[ 
https://issues.apache.org/jira/browse/PIG-3717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13880387#comment-13880387
 ] 

Jarek Jarcec Cecho commented on PIG-3717:
-

Thank you for the feedback [~rohini]!

> AvroStorage is not filtering hidden files on directory recursion
> 
>
> Key: PIG-3717
> URL: https://issues.apache.org/jira/browse/PIG-3717
> Project: Pig
>  Issue Type: Bug
>Affects Versions: 0.12.0
>        Reporter: Jarek Jarcec Cecho
>    Assignee: Jarek Jarcec Cecho
> Fix For: 0.13.0
>
> Attachments: PIG-3717.patch, PIG-3717.patch
>
>
> We've noticed that when we generate couple of sub-directories with Oozie and 
> then use AvroStorage (the older version from PiggyBank) to load those 
> directories, hidden files such as {{_SUCCESS}} won't get ignored as they 
> should be and in turn will cause failures in the Pig job. This seems to be 
> regression caused by PIG-3223.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (PIG-3717) AvroStorage is not filtering hidden files on directory recursion

[Jarek Jarcec Cecho (JIRA)]

[ 
https://issues.apache.org/jira/browse/PIG-3717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13880347#comment-13880347
 ] 

Jarek Jarcec Cecho commented on PIG-3717:
-

Do you think that we can go ahead and commit this one separately or would you 
prefer to cover it by your patch [~rohini]?

> AvroStorage is not filtering hidden files on directory recursion
> 
>
> Key: PIG-3717
> URL: https://issues.apache.org/jira/browse/PIG-3717
> Project: Pig
>  Issue Type: Bug
>Affects Versions: 0.12.0
>        Reporter: Jarek Jarcec Cecho
>    Assignee: Jarek Jarcec Cecho
> Fix For: 0.13.0
>
> Attachments: PIG-3717.patch, PIG-3717.patch
>
>
> We've noticed that when we generate couple of sub-directories with Oozie and 
> then use AvroStorage (the older version from PiggyBank) to load those 
> directories, hidden files such as {{_SUCCESS}} won't get ignored as they 
> should be and in turn will cause failures in the Pig job. This seems to be 
> regression caused by PIG-3223.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

Re: Jira notification spam from me

[Jarek Jarcec Cecho]

I've also created couple of  JIRAs while the JIRA was in some sort of weird 
state:

* PIG-3685
* PIG-3684
* PIG-3683
 
I would appreciate if someone with sufficient privileges could remove those.

Jarcec

On Thu, Jan 23, 2014 at 08:15:39AM -0800, Cheolsoo Park wrote:
> Hi devs,
> 
> I am sorry about the jira notification spams that I sent to dev mailing
> list yesterday and today.
> 
> I accidentally created/deleted the same jiras several times. I was trying
> to open a sub-task for pig on tez, but apparently, someone else was doing
> at the same time. That causes a strange concurrency issue in jira resulting
> in duplicated jiras... I deleted them all today.
> 
> Thanks,
> Cheolsoo


signature.asc
Description: Digital signature

[jira] [Updated] (PIG-3717) AvroStorage is not filtering hidden files on directory recursion

[Jarek Jarcec Cecho (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PIG-3717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jarek Jarcec Cecho updated PIG-3717:


Attachment: PIG-3717.patch

> AvroStorage is not filtering hidden files on directory recursion
> 
>
> Key: PIG-3717
> URL: https://issues.apache.org/jira/browse/PIG-3717
> Project: Pig
>  Issue Type: Bug
>Affects Versions: 0.12.0
>    Reporter: Jarek Jarcec Cecho
>    Assignee: Jarek Jarcec Cecho
> Fix For: 0.13.0
>
> Attachments: PIG-3717.patch, PIG-3717.patch
>
>
> We've noticed that when we generate couple of sub-directories with Oozie and 
> then use AvroStorage (the older version from PiggyBank) to load those 
> directories, hidden files such as {{_SUCCESS}} won't get ignored as they 
> should be and in turn will cause failures in the Pig job. This seems to be 
> regression caused by PIG-3223.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (PIG-3683) AvroStorage is not filtering hidden files on directory recursion

[Jarek Jarcec Cecho (JIRA)]

[ 
https://issues.apache.org/jira/browse/PIG-3683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13880018#comment-13880018
 ] 

Jarek Jarcec Cecho commented on PIG-3683:
-

This has been created when the JIRA was in weird state, so it can be removed.

> AvroStorage is not filtering hidden files on directory recursion
> 
>
> Key: PIG-3683
> URL: https://issues.apache.org/jira/browse/PIG-3683
> Project: Pig
>  Issue Type: Bug
>Affects Versions: 0.12.0
>        Reporter: Jarek Jarcec Cecho
>    Assignee: Jarek Jarcec Cecho
>Priority: Minor
> Fix For: 0.13.0
>
>
> We've noticed that when we generate couple of sub-directories with Oozie and 
> then use {{AvroStorage}} (the older version from PiggyBank) to load those 
> directories up,  hidden files such as {{_SUCCESS} won't get ignored as they 
> should be and in turn entire Pig execution fill fail.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (PIG-3685) AvroStorage is not filtering hidden files on directory recursion

[Jarek Jarcec Cecho (JIRA)]

[ 
https://issues.apache.org/jira/browse/PIG-3685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13880021#comment-13880021
 ] 

Jarek Jarcec Cecho commented on PIG-3685:
-

This has been created when the JIRA was in weird state, so it can be removed.

> AvroStorage is not filtering hidden files on directory recursion
> 
>
> Key: PIG-3685
> URL: https://issues.apache.org/jira/browse/PIG-3685
> Project: Pig
>  Issue Type: Bug
>Affects Versions: 0.12.0
>        Reporter: Jarek Jarcec Cecho
>    Assignee: Jarek Jarcec Cecho
>Priority: Minor
> Fix For: 0.13.0
>
>
> We've noticed that when we generate couple of sub-directories with Oozie and 
> then use {{AvroStorage}} (the older version from PiggyBank) to load those 
> directories up,  hidden files such as {{_SUCCESS} won't get ignored as they 
> should be and in turn entire Pig execution fill fail.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (PIG-3684) AvroStorage is not filtering hidden files on directory recursion

[Jarek Jarcec Cecho (JIRA)]

[ 
https://issues.apache.org/jira/browse/PIG-3684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13880019#comment-13880019
 ] 

Jarek Jarcec Cecho commented on PIG-3684:
-

This has been created when the JIRA was in weird state, so it can be removed.

> AvroStorage is not filtering hidden files on directory recursion
> 
>
> Key: PIG-3684
> URL: https://issues.apache.org/jira/browse/PIG-3684
> Project: Pig
>  Issue Type: Bug
>Affects Versions: 0.12.0
>        Reporter: Jarek Jarcec Cecho
>    Assignee: Jarek Jarcec Cecho
>Priority: Minor
> Fix For: 0.13.0
>
>
> We've noticed that when we generate couple of sub-directories with Oozie and 
> then use {{AvroStorage}} (the older version from PiggyBank) to load those 
> directories up,  hidden files such as {{_SUCCESS} won't get ignored as they 
> should be and in turn entire Pig execution fill fail.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Updated] (PIG-3717) AvroStorage is not filtering hidden files on directory recursion

[Jarek Jarcec Cecho (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PIG-3717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jarek Jarcec Cecho updated PIG-3717:


Fix Version/s: 0.13.0
Affects Version/s: 0.12.0
   Status: Patch Available  (was: Open)

> AvroStorage is not filtering hidden files on directory recursion
> 
>
> Key: PIG-3717
> URL: https://issues.apache.org/jira/browse/PIG-3717
> Project: Pig
>  Issue Type: Bug
>Affects Versions: 0.12.0
>    Reporter: Jarek Jarcec Cecho
>    Assignee: Jarek Jarcec Cecho
> Fix For: 0.13.0
>
> Attachments: PIG-3717.patch
>
>
> We've noticed that when we generate couple of sub-directories with Oozie and 
> then use AvroStorage (the older version from PiggyBank) to load those 
> directories, hidden files such as {{_SUCCESS}} won't get ignored as they 
> should be and in turn will cause failures in the Pig job.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Updated] (PIG-3717) AvroStorage is not filtering hidden files on directory recursion

[Jarek Jarcec Cecho (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PIG-3717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jarek Jarcec Cecho updated PIG-3717:


Description: We've noticed that when we generate couple of sub-directories 
with Oozie and then use AvroStorage (the older version from PiggyBank) to load 
those directories, hidden files such as {{_SUCCESS}} won't get ignored as they 
should be and in turn will cause failures in the Pig job. This seems to be 
regression caused by PIG-3223.  (was: We've noticed that when we generate 
couple of sub-directories with Oozie and then use AvroStorage (the older 
version from PiggyBank) to load those directories, hidden files such as 
{{_SUCCESS}} won't get ignored as they should be and in turn will cause 
failures in the Pig job.)

> AvroStorage is not filtering hidden files on directory recursion
> 
>
> Key: PIG-3717
> URL: https://issues.apache.org/jira/browse/PIG-3717
> Project: Pig
>  Issue Type: Bug
>Affects Versions: 0.12.0
>Reporter: Jarek Jarcec Cecho
>Assignee: Jarek Jarcec Cecho
> Fix For: 0.13.0
>
> Attachments: PIG-3717.patch
>
>
> We've noticed that when we generate couple of sub-directories with Oozie and 
> then use AvroStorage (the older version from PiggyBank) to load those 
> directories, hidden files such as {{_SUCCESS}} won't get ignored as they 
> should be and in turn will cause failures in the Pig job. This seems to be 
> regression caused by PIG-3223.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Updated] (PIG-3717) AvroStorage is not filtering hidden files on directory recursion

[Jarek Jarcec Cecho (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PIG-3717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jarek Jarcec Cecho updated PIG-3717:


Attachment: PIG-3717.patch

> AvroStorage is not filtering hidden files on directory recursion
> 
>
> Key: PIG-3717
> URL: https://issues.apache.org/jira/browse/PIG-3717
> Project: Pig
>  Issue Type: Bug
>Affects Versions: 0.12.0
>    Reporter: Jarek Jarcec Cecho
>    Assignee: Jarek Jarcec Cecho
> Fix For: 0.13.0
>
> Attachments: PIG-3717.patch
>
>
> We've noticed that when we generate couple of sub-directories with Oozie and 
> then use AvroStorage (the older version from PiggyBank) to load those 
> directories, hidden files such as {{_SUCCESS}} won't get ignored as they 
> should be and in turn will cause failures in the Pig job.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Created] (PIG-3717) AvroStorage is not filtering hidden files on directory recursion

[Jarek Jarcec Cecho (JIRA)]

Jarek Jarcec Cecho created PIG-3717:
---

 Summary: AvroStorage is not filtering hidden files on directory 
recursion
 Key: PIG-3717
 URL: https://issues.apache.org/jira/browse/PIG-3717
 Project: Pig
  Issue Type: Bug
Reporter: Jarek Jarcec Cecho
Assignee: Jarek Jarcec Cecho


We've noticed that when we generate couple of sub-directories with Oozie and 
then use AvroStorage (the older version from PiggyBank) to load those 
directories, hidden files such as {{_SUCCESS}} won't get ignored as they should 
be and in turn will cause failures in the Pig job.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

Re: Welcome to the new Pig PMC member Aniket Mokashi

[Jarek Jarcec Cecho]

Congratulations Aniket, good work!

Jarcec

On Tue, Jan 14, 2014 at 06:52:10PM -0800, JULIEN LE DEM wrote:
> It's my pleasure to announce that Aniket Mokashi became the newest addition 
> to the Pig PMC.
> Aniket has been actively contributing to Pig for years.
> Please join me in congratulating Aniket!
> 
> Julien
> 


signature.asc
Description: Digital signature

[jira] [Commented] (PIG-3639) TestRegisteredJarVisibility is broken in trunk

[Jarek Jarcec Cecho (JIRA)]

[ 
https://issues.apache.org/jira/browse/PIG-3639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13855477#comment-13855477
 ] 

Jarek Jarcec Cecho commented on PIG-3639:
-

Thank you for filling the JIRA and providing patch [~cheolsoo]! I was 
experiencing the same failure in our environment when upgraded Avro to version 
1.7.5. Sadly I did not had time to investigate it myself. 

+1 (non-binding)

> TestRegisteredJarVisibility is broken in trunk
> --
>
> Key: PIG-3639
> URL: https://issues.apache.org/jira/browse/PIG-3639
> Project: Pig
>  Issue Type: Bug
>Reporter: Cheolsoo Park
>Assignee: Cheolsoo Park
> Fix For: 0.13.0
>
> Attachments: PIG-3639-1.patch
>
>
> {code}
> [junit] Running org.apache.pig.test.TestRegisteredJarVisibility
> [junit] Tests run: 3, Failures: 1, Errors: 0, Time elapsed: 44.127 sec
> [junit] Test org.apache.pig.test.TestRegisteredJarVisibility FAILED
> {code}
> This is a side-effect of PIG-3584 that bumped avro version to 1.7.5.
> The problem is that avro 1.7.5 pulls down jackson 1.9.9 jars as dependencies, 
> and that makes 
> TestRegisteredJarVisibility.testRegisterJarOverridePigJarPackages fail 
> because the test case assumes that jackson 1.9.9 jars are not present in 
> classpath.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (PIG-3633) AvroStorage tests are failing when running against Avro 1.7.5

[Jarek Jarcec Cecho (JIRA)]

[ 
https://issues.apache.org/jira/browse/PIG-3633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13852715#comment-13852715
 ] 

Jarek Jarcec Cecho commented on PIG-3633:
-

Thank you for the review [~cheolsoo], appreciated!

> AvroStorage tests are failing when running against Avro 1.7.5
> -
>
> Key: PIG-3633
> URL: https://issues.apache.org/jira/browse/PIG-3633
> Project: Pig
>  Issue Type: Bug
>Affects Versions: 0.12.0
>        Reporter: Jarek Jarcec Cecho
>    Assignee: Jarek Jarcec Cecho
>Priority: Minor
> Fix For: 0.13.0
>
> Attachments: PIG-3633.patch
>
>
> {{AvroStorage}} tests executed against latest Avro release 1.7.5 are failing:
> {code}
> ant clean test -Dtestcase=TestAvroStorage -Davro.version=1.7.5
> ...
>[junit] Running org.apache.pig.builtin.TestAvroStorage
>[junit] Tests run: 33, Failures: 15, Errors: 0, Time elapsed: 14.211 sec
> {code}
> With following exception:
> {code}
> Failed to parse: Pig script failed to parse: 
>  pig script failed to validate: java.lang.RuntimeException: 
> could not instantiate 'AvroStorage' with arguments '[, -r]'
>   at 
> org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:196)
>   at org.apache.pig.PigServer$Graph.parseQuery(PigServer.java:1676)
>   at org.apache.pig.PigServer$Graph.registerQuery(PigServer.java:1623)
>   at org.apache.pig.PigServer.registerQuery(PigServer.java:575)
>   at 
> org.apache.pig.tools.grunt.GruntParser.processPig(GruntParser.java:1093)
>   at 
> org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:501)
>   at 
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:198)
>   at org.apache.pig.PigServer.registerScript(PigServer.java:649)
>   at org.apache.pig.PigServer.registerScript(PigServer.java:726)
>   at org.apache.pig.PigServer.registerScript(PigServer.java:699)
>   at 
> org.apache.pig.builtin.TestAvroStorage.testAvroStorage(TestAvroStorage.java:775)
>   at 
> org.apache.pig.builtin.TestAvroStorage.testLoadRecursiveRecordsOptionOn(TestAvroStorage.java:588)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:601)
>   at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
>   at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>   at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
>   at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
>   at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
>   at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
>   at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
>   at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
>   at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
>   at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
>   at 
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
>   at 
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
>   at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
>   at junit.framework.JUnit4TestAdapter.run(JUnit4TestAdapter.java:38)
>   at 
> org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:518)
>   at 
> org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1052)
>   at 
> org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:906)
> Caused by: 
>  pig script failed to validate: java.lang.RuntimeException: 
> could not instantiate 'AvroStorage' with arguments '[, -r]'
>   at 
> org.apache.pig.parser.LogicalPlanBuilder.buildLoadOp(LogicalPlanBuilder.java:849)
>   at 
> org.apache.pig.parser.LogicalPlanGenerator.load_clause(LogicalPlanGenerator.java:3479)
>

[jira] [Updated] (PIG-3633) AvroStorage tests are failing when running against Avro 1.7.5

[Jarek Jarcec Cecho (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PIG-3633?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jarek Jarcec Cecho updated PIG-3633:


Attachment: PIG-3633.patch

Attaching patch that will skip schema parsing if the given string is empty. 
This seems to resolve the failing tests.

> AvroStorage tests are failing when running against Avro 1.7.5
> -
>
> Key: PIG-3633
> URL: https://issues.apache.org/jira/browse/PIG-3633
> Project: Pig
>  Issue Type: Bug
>Affects Versions: 0.12.0
>    Reporter: Jarek Jarcec Cecho
>    Assignee: Jarek Jarcec Cecho
>Priority: Minor
> Fix For: 0.13.0
>
> Attachments: PIG-3633.patch
>
>
> {{AvroStorage}} tests executed against latest Avro release 1.7.5 are failing:
> {code}
> ant clean test -Dtestcase=TestAvroStorage -Davro.version=1.7.5
> ...
>[junit] Running org.apache.pig.builtin.TestAvroStorage
>[junit] Tests run: 33, Failures: 15, Errors: 0, Time elapsed: 14.211 sec
> {code}
> With following exception:
> {code}
> Failed to parse: Pig script failed to parse: 
>  pig script failed to validate: java.lang.RuntimeException: 
> could not instantiate 'AvroStorage' with arguments '[, -r]'
>   at 
> org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:196)
>   at org.apache.pig.PigServer$Graph.parseQuery(PigServer.java:1676)
>   at org.apache.pig.PigServer$Graph.registerQuery(PigServer.java:1623)
>   at org.apache.pig.PigServer.registerQuery(PigServer.java:575)
>   at 
> org.apache.pig.tools.grunt.GruntParser.processPig(GruntParser.java:1093)
>   at 
> org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:501)
>   at 
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:198)
>   at org.apache.pig.PigServer.registerScript(PigServer.java:649)
>   at org.apache.pig.PigServer.registerScript(PigServer.java:726)
>   at org.apache.pig.PigServer.registerScript(PigServer.java:699)
>   at 
> org.apache.pig.builtin.TestAvroStorage.testAvroStorage(TestAvroStorage.java:775)
>   at 
> org.apache.pig.builtin.TestAvroStorage.testLoadRecursiveRecordsOptionOn(TestAvroStorage.java:588)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:601)
>   at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
>   at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>   at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
>   at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
>   at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
>   at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
>   at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
>   at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
>   at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
>   at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
>   at 
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
>   at 
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
>   at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
>   at junit.framework.JUnit4TestAdapter.run(JUnit4TestAdapter.java:38)
>   at 
> org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:518)
>   at 
> org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1052)
>   at 
> org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:906)
> Caused by: 
>  pig script failed to validate: java.lang.RuntimeException: 
> could not instantiate 'AvroStorage' with arguments '[, -r]'
>   at 
> org.apache.pig.parser.LogicalPlanBuilder.buildLoadOp(LogicalPlanBuilder.java:849)
>   at 
> org.apache.pig.parser.LogicalPlanGenerator.load_clau

[jira] [Updated] (PIG-3633) AvroStorage tests are failing when running against Avro 1.7.5

[Jarek Jarcec Cecho (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PIG-3633?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jarek Jarcec Cecho updated PIG-3633:


Status: Patch Available  (was: Open)

> AvroStorage tests are failing when running against Avro 1.7.5
> -
>
> Key: PIG-3633
> URL: https://issues.apache.org/jira/browse/PIG-3633
> Project: Pig
>  Issue Type: Bug
>Affects Versions: 0.12.0
>    Reporter: Jarek Jarcec Cecho
>    Assignee: Jarek Jarcec Cecho
>Priority: Minor
> Fix For: 0.13.0
>
> Attachments: PIG-3633.patch
>
>
> {{AvroStorage}} tests executed against latest Avro release 1.7.5 are failing:
> {code}
> ant clean test -Dtestcase=TestAvroStorage -Davro.version=1.7.5
> ...
>[junit] Running org.apache.pig.builtin.TestAvroStorage
>[junit] Tests run: 33, Failures: 15, Errors: 0, Time elapsed: 14.211 sec
> {code}
> With following exception:
> {code}
> Failed to parse: Pig script failed to parse: 
>  pig script failed to validate: java.lang.RuntimeException: 
> could not instantiate 'AvroStorage' with arguments '[, -r]'
>   at 
> org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:196)
>   at org.apache.pig.PigServer$Graph.parseQuery(PigServer.java:1676)
>   at org.apache.pig.PigServer$Graph.registerQuery(PigServer.java:1623)
>   at org.apache.pig.PigServer.registerQuery(PigServer.java:575)
>   at 
> org.apache.pig.tools.grunt.GruntParser.processPig(GruntParser.java:1093)
>   at 
> org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:501)
>   at 
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:198)
>   at org.apache.pig.PigServer.registerScript(PigServer.java:649)
>   at org.apache.pig.PigServer.registerScript(PigServer.java:726)
>   at org.apache.pig.PigServer.registerScript(PigServer.java:699)
>   at 
> org.apache.pig.builtin.TestAvroStorage.testAvroStorage(TestAvroStorage.java:775)
>   at 
> org.apache.pig.builtin.TestAvroStorage.testLoadRecursiveRecordsOptionOn(TestAvroStorage.java:588)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:601)
>   at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
>   at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>   at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
>   at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
>   at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
>   at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
>   at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
>   at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
>   at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
>   at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
>   at 
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
>   at 
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
>   at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
>   at junit.framework.JUnit4TestAdapter.run(JUnit4TestAdapter.java:38)
>   at 
> org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:518)
>   at 
> org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1052)
>   at 
> org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:906)
> Caused by: 
>  pig script failed to validate: java.lang.RuntimeException: 
> could not instantiate 'AvroStorage' with arguments '[, -r]'
>   at 
> org.apache.pig.parser.LogicalPlanBuilder.buildLoadOp(LogicalPlanBuilder.java:849)
>   at 
> org.apache.pig.parser.LogicalPlanGenerator.load_clause(LogicalPlanGenerator.java:3479)
>   at 
> org.apache.pig.parser.LogicalPlanGenerator.op_clause(LogicalPlanG

[jira] [Created] (PIG-3633) AvroStorage tests are failing when running against Avro 1.7.5

[Jarek Jarcec Cecho (JIRA)]

Jarek Jarcec Cecho created PIG-3633:
---

 Summary: AvroStorage tests are failing when running against Avro 
1.7.5
 Key: PIG-3633
 URL: https://issues.apache.org/jira/browse/PIG-3633
 Project: Pig
  Issue Type: Bug
Affects Versions: 0.12.0
Reporter: Jarek Jarcec Cecho
Assignee: Jarek Jarcec Cecho
Priority: Minor
 Fix For: 0.13.0


{{AvroStorage}} tests executed against latest Avro release 1.7.5 are failing:

{code}
ant clean test -Dtestcase=TestAvroStorage
...
   [junit] Running org.apache.pig.builtin.TestAvroStorage
   [junit] Tests run: 33, Failures: 15, Errors: 0, Time elapsed: 14.211 sec
{code}

With following exception:

{code}
Failed to parse: Pig script failed to parse: 
 pig script failed to validate: java.lang.RuntimeException: 
could not instantiate 'AvroStorage' with arguments '[, -r]'
at 
org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:196)
at org.apache.pig.PigServer$Graph.parseQuery(PigServer.java:1676)
at org.apache.pig.PigServer$Graph.registerQuery(PigServer.java:1623)
at org.apache.pig.PigServer.registerQuery(PigServer.java:575)
at 
org.apache.pig.tools.grunt.GruntParser.processPig(GruntParser.java:1093)
at 
org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:501)
at 
org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:198)
at org.apache.pig.PigServer.registerScript(PigServer.java:649)
at org.apache.pig.PigServer.registerScript(PigServer.java:726)
at org.apache.pig.PigServer.registerScript(PigServer.java:699)
at 
org.apache.pig.builtin.TestAvroStorage.testAvroStorage(TestAvroStorage.java:775)
at 
org.apache.pig.builtin.TestAvroStorage.testLoadRecursiveRecordsOptionOn(TestAvroStorage.java:588)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at 
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
at 
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at 
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
at junit.framework.JUnit4TestAdapter.run(JUnit4TestAdapter.java:38)
at 
org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:518)
at 
org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1052)
at 
org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:906)
Caused by: 
 pig script failed to validate: java.lang.RuntimeException: 
could not instantiate 'AvroStorage' with arguments '[, -r]'
at 
org.apache.pig.parser.LogicalPlanBuilder.buildLoadOp(LogicalPlanBuilder.java:849)
at 
org.apache.pig.parser.LogicalPlanGenerator.load_clause(LogicalPlanGenerator.java:3479)
at 
org.apache.pig.parser.LogicalPlanGenerator.op_clause(LogicalPlanGenerator.java:1536)
at 
org.apache.pig.parser.LogicalPlanGenerator.general_statement(LogicalPlanGenerator.java:1013)
at 
org.apache.pig.parser.LogicalPlanGenerator.statement(LogicalPlanGenerator.java:553)
at 
org.apache.pig.parser.LogicalPlanGenerator.query(LogicalPlanGenerator.java:421)
at 
org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:188)
... 34 more
Caused by: java.lang.RuntimeException: could not instantiate 'AvroStorage' with 
arguments '[, -r]'
at 
org.apache.pig.impl.PigCon

[jira] [Updated] (PIG-3633) AvroStorage tests are failing when running against Avro 1.7.5

[Jarek Jarcec Cecho (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PIG-3633?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jarek Jarcec Cecho updated PIG-3633:


Description: 
{{AvroStorage}} tests executed against latest Avro release 1.7.5 are failing:

{code}
ant clean test -Dtestcase=TestAvroStorage -Davro.version=1.7.5
...
   [junit] Running org.apache.pig.builtin.TestAvroStorage
   [junit] Tests run: 33, Failures: 15, Errors: 0, Time elapsed: 14.211 sec
{code}

With following exception:

{code}
Failed to parse: Pig script failed to parse: 
 pig script failed to validate: java.lang.RuntimeException: 
could not instantiate 'AvroStorage' with arguments '[, -r]'
at 
org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:196)
at org.apache.pig.PigServer$Graph.parseQuery(PigServer.java:1676)
at org.apache.pig.PigServer$Graph.registerQuery(PigServer.java:1623)
at org.apache.pig.PigServer.registerQuery(PigServer.java:575)
at 
org.apache.pig.tools.grunt.GruntParser.processPig(GruntParser.java:1093)
at 
org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:501)
at 
org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:198)
at org.apache.pig.PigServer.registerScript(PigServer.java:649)
at org.apache.pig.PigServer.registerScript(PigServer.java:726)
at org.apache.pig.PigServer.registerScript(PigServer.java:699)
at 
org.apache.pig.builtin.TestAvroStorage.testAvroStorage(TestAvroStorage.java:775)
at 
org.apache.pig.builtin.TestAvroStorage.testLoadRecursiveRecordsOptionOn(TestAvroStorage.java:588)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at 
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
at 
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at 
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
at junit.framework.JUnit4TestAdapter.run(JUnit4TestAdapter.java:38)
at 
org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:518)
at 
org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1052)
at 
org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:906)
Caused by: 
 pig script failed to validate: java.lang.RuntimeException: 
could not instantiate 'AvroStorage' with arguments '[, -r]'
at 
org.apache.pig.parser.LogicalPlanBuilder.buildLoadOp(LogicalPlanBuilder.java:849)
at 
org.apache.pig.parser.LogicalPlanGenerator.load_clause(LogicalPlanGenerator.java:3479)
at 
org.apache.pig.parser.LogicalPlanGenerator.op_clause(LogicalPlanGenerator.java:1536)
at 
org.apache.pig.parser.LogicalPlanGenerator.general_statement(LogicalPlanGenerator.java:1013)
at 
org.apache.pig.parser.LogicalPlanGenerator.statement(LogicalPlanGenerator.java:553)
at 
org.apache.pig.parser.LogicalPlanGenerator.query(LogicalPlanGenerator.java:421)
at 
org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:188)
... 34 more
Caused by: java.lang.RuntimeException: could not instantiate 'AvroStorage' with 
arguments '[, -r]'
at 
org.apache.pig.impl.PigContext.instantiateFuncFromSpec(PigContext.java:748)
at 
org.apache.pig.parser.LogicalPlanBuilder.buildLoadOp(LogicalPlanBuilder.java:837)
... 40 more
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeCo

[jira] [Commented] (PIG-3285) Jobs using HBaseStorage fail to ship dependency jars

[Jarek Jarcec Cecho (JIRA)]

[ 
https://issues.apache.org/jira/browse/PIG-3285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13831611#comment-13831611
 ] 

Jarek Jarcec Cecho commented on PIG-3285:
-

Thank you [~cheolsoo]!

> Jobs using HBaseStorage fail to ship dependency jars
> 
>
> Key: PIG-3285
> URL: https://issues.apache.org/jira/browse/PIG-3285
> Project: Pig
>  Issue Type: Bug
>Reporter: Nick Dimiduk
>Assignee: Nick Dimiduk
> Fix For: 0.13.0
>
> Attachments: 0001-PIG-3285-Add-HBase-dependency-jars.patch, 
> 0001-PIG-3285-Add-HBase-dependency-jars.patch, 
> 0001-PIG-3285-Add-HBase-dependency-jars.patch, 
> 0001-PIG-3285-Add-HBase-dependency-jars.patch, 
> 0001-PIG-3285-Add-HBase-dependency-jars.patch, 
> 0001-PIG-3285-Add-HBase-dependency-jars.patch, 1.pig, 1.txt, 2.pig
>
>
> Launching a job consuming {{HBaseStorage}} fails out of the box. The user 
> must specify {{-Dpig.additional.jars}} for HBase and all of its dependencies. 
> Exceptions look something like this:
> {noformat}
> 2013-04-19 18:58:39,360 FATAL org.apache.hadoop.mapred.Child: Error running 
> child : java.lang.NoClassDefFoundError: com/google/protobuf/Message
>   at 
> org.apache.hadoop.hbase.io.HbaseObjectWritable.(HbaseObjectWritable.java:266)
>   at org.apache.hadoop.hbase.ipc.Invocation.write(Invocation.java:139)
>   at 
> org.apache.hadoop.hbase.ipc.HBaseClient$Connection.sendParam(HBaseClient.java:612)
>   at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:975)
>   at 
> org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:84)
>   at $Proxy7.getProtocolVersion(Unknown Source)
>   at 
> org.apache.hadoop.hbase.ipc.WritableRpcEngine.getProxy(WritableRpcEngine.java:136)
>   at org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:208)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (PIG-3285) Jobs using HBaseStorage fail to ship dependency jars

[Jarek Jarcec Cecho (JIRA)]

[ 
https://issues.apache.org/jira/browse/PIG-3285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13830715#comment-13830715
 ] 

Jarek Jarcec Cecho commented on PIG-3285:
-

My apologies for the delay [~ndimiduk]. I've tried the latest patch against 
linked HBase RC and everything seems to be working.

+1 (non-binding)

Feel free to take a look [~cheolsoo]!

> Jobs using HBaseStorage fail to ship dependency jars
> 
>
> Key: PIG-3285
> URL: https://issues.apache.org/jira/browse/PIG-3285
> Project: Pig
>  Issue Type: Bug
>Reporter: Nick Dimiduk
>Assignee: Nick Dimiduk
> Fix For: 0.13.0
>
> Attachments: 0001-PIG-3285-Add-HBase-dependency-jars.patch, 
> 0001-PIG-3285-Add-HBase-dependency-jars.patch, 
> 0001-PIG-3285-Add-HBase-dependency-jars.patch, 
> 0001-PIG-3285-Add-HBase-dependency-jars.patch, 
> 0001-PIG-3285-Add-HBase-dependency-jars.patch, 
> 0001-PIG-3285-Add-HBase-dependency-jars.patch, 1.pig, 1.txt, 2.pig
>
>
> Launching a job consuming {{HBaseStorage}} fails out of the box. The user 
> must specify {{-Dpig.additional.jars}} for HBase and all of its dependencies. 
> Exceptions look something like this:
> {noformat}
> 2013-04-19 18:58:39,360 FATAL org.apache.hadoop.mapred.Child: Error running 
> child : java.lang.NoClassDefFoundError: com/google/protobuf/Message
>   at 
> org.apache.hadoop.hbase.io.HbaseObjectWritable.(HbaseObjectWritable.java:266)
>   at org.apache.hadoop.hbase.ipc.Invocation.write(Invocation.java:139)
>   at 
> org.apache.hadoop.hbase.ipc.HBaseClient$Connection.sendParam(HBaseClient.java:612)
>   at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:975)
>   at 
> org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:84)
>   at $Proxy7.getProtocolVersion(Unknown Source)
>   at 
> org.apache.hadoop.hbase.ipc.WritableRpcEngine.getProxy(WritableRpcEngine.java:136)
>   at org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:208)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (PIG-3285) Jobs using HBaseStorage fail to ship dependency jars

[Jarek Jarcec Cecho (JIRA)]

[ 
https://issues.apache.org/jira/browse/PIG-3285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13828234#comment-13828234
 ] 

Jarek Jarcec Cecho commented on PIG-3285:
-

I've tested the change on a real clusters running HBase 0.94 with and without 
the HBASE-9165 and everything seems to be working just fine. It would be great 
to also test 0.96 with and without the change, but I do not feel that it's 
entirely necessary. I'm +1 on the patch, provided that we will remove the JDK7 
only class {{ReflectiveOperationException}}.

> Jobs using HBaseStorage fail to ship dependency jars
> 
>
> Key: PIG-3285
> URL: https://issues.apache.org/jira/browse/PIG-3285
> Project: Pig
>  Issue Type: Bug
>Reporter: Nick Dimiduk
>Assignee: Nick Dimiduk
> Fix For: 0.13.0
>
> Attachments: 0001-PIG-3285-Add-HBase-dependency-jars.patch, 
> 0001-PIG-3285-Add-HBase-dependency-jars.patch, 
> 0001-PIG-3285-Add-HBase-dependency-jars.patch, 
> 0001-PIG-3285-Add-HBase-dependency-jars.patch, 
> 0001-PIG-3285-Add-HBase-dependency-jars.patch, 1.pig, 1.txt, 2.pig
>
>
> Launching a job consuming {{HBaseStorage}} fails out of the box. The user 
> must specify {{-Dpig.additional.jars}} for HBase and all of its dependencies. 
> Exceptions look something like this:
> {noformat}
> 2013-04-19 18:58:39,360 FATAL org.apache.hadoop.mapred.Child: Error running 
> child : java.lang.NoClassDefFoundError: com/google/protobuf/Message
>   at 
> org.apache.hadoop.hbase.io.HbaseObjectWritable.(HbaseObjectWritable.java:266)
>   at org.apache.hadoop.hbase.ipc.Invocation.write(Invocation.java:139)
>   at 
> org.apache.hadoop.hbase.ipc.HBaseClient$Connection.sendParam(HBaseClient.java:612)
>   at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:975)
>   at 
> org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:84)
>   at $Proxy7.getProtocolVersion(Unknown Source)
>   at 
> org.apache.hadoop.hbase.ipc.WritableRpcEngine.getProxy(WritableRpcEngine.java:136)
>   at org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:208)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (PIG-3285) Jobs using HBaseStorage fail to ship dependency jars

[Jarek Jarcec Cecho (JIRA)]

[ 
https://issues.apache.org/jira/browse/PIG-3285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13827083#comment-13827083
 ] 

Jarek Jarcec Cecho commented on PIG-3285:
-

Thank you for the quick turnaround [~ndimiduk]!

I was able to run {{TestHBaseStorage}} unit tests for all the combinations 
using following commands:

{code}
# Old path:
ant clean test -Dhbaseversion=94 -Dtestcase=TestHBaseStorage
ant clean test -Dhbaseversion=95 -Dtestcase=TestHBaseStorage  
-Dprotobuf-java.version=2.5.0

# New path:

# Compile HBase 0.94 branch and install it to local maven cache (mvn clean 
install -DskipTests)
ant clean test -Dhbaseversion=94 -Dtestcase=TestHBaseStorage 
-Dhbase94.version=0.94.14
# Compile HBase 0.96 branch and install it to local maven cache (mvn clean 
install -DskipTests)
ant clean test -Dhbaseversion=95 -Dtestcase=TestHBaseStorage  
-Dprotobuf-java.version=2.5.0 -Dhbase95.version=0.96.1-SNAPSHOT
{code}

The only problem that I've encounter so far is the exception 
{{ReflectiveOperationException}} 
([javadoc|http://docs.oracle.com/javase/7/docs/api/java/lang/ReflectiveOperationException.html])
 that has been added in JDK7, while Pig still requires JDK6 compatibility. I've 
substituted the exception with {{Exception}}  for the purpose of above tests.

I'll try to give it a spin on a real clusters to see if everything will work as 
expected.


> Jobs using HBaseStorage fail to ship dependency jars
> 
>
> Key: PIG-3285
> URL: https://issues.apache.org/jira/browse/PIG-3285
> Project: Pig
>  Issue Type: Bug
>Reporter: Nick Dimiduk
>Assignee: Nick Dimiduk
> Fix For: 0.13.0
>
> Attachments: 0001-PIG-3285-Add-HBase-dependency-jars.patch, 
> 0001-PIG-3285-Add-HBase-dependency-jars.patch, 
> 0001-PIG-3285-Add-HBase-dependency-jars.patch, 
> 0001-PIG-3285-Add-HBase-dependency-jars.patch, 
> 0001-PIG-3285-Add-HBase-dependency-jars.patch, 1.pig, 1.txt, 2.pig
>
>
> Launching a job consuming {{HBaseStorage}} fails out of the box. The user 
> must specify {{-Dpig.additional.jars}} for HBase and all of its dependencies. 
> Exceptions look something like this:
> {noformat}
> 2013-04-19 18:58:39,360 FATAL org.apache.hadoop.mapred.Child: Error running 
> child : java.lang.NoClassDefFoundError: com/google/protobuf/Message
>   at 
> org.apache.hadoop.hbase.io.HbaseObjectWritable.(HbaseObjectWritable.java:266)
>   at org.apache.hadoop.hbase.ipc.Invocation.write(Invocation.java:139)
>   at 
> org.apache.hadoop.hbase.ipc.HBaseClient$Connection.sendParam(HBaseClient.java:612)
>   at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:975)
>   at 
> org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:84)
>   at $Proxy7.getProtocolVersion(Unknown Source)
>   at 
> org.apache.hadoop.hbase.ipc.WritableRpcEngine.getProxy(WritableRpcEngine.java:136)
>   at org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:208)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (PIG-3285) Jobs using HBaseStorage fail to ship dependency jars

[Jarek Jarcec Cecho (JIRA)]

[ 
https://issues.apache.org/jira/browse/PIG-3285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13826981#comment-13826981
 ] 

Jarek Jarcec Cecho commented on PIG-3285:
-

Thank you [~ndimiduk]! Couple of high level notes:

{code}
-  TableMapReduceUtil.addDependencyJars(job.getConfiguration(), klass);
{code}

It seems that we've accidentally removed the call 
{{TableMapReduceUtil.addDependencyJars()}}  from method 
{{addClassToJobIfExists()}}. As a result not all required jars are propagated 
to the {{DistributedCache}}  when running on HBase 0.95+.

{code}
+// method doesn't exist or invocation failed. moving on.
{code}

Can we at least do a debug log message stating that 
{{TableMapReduceUtil.addHBaseDependencyJars}} is not available and thus we will 
revert to our own logic for propagating jars? It think that it will help to 
know what logic we used for propagating jars while investigating failing pig 
scripts.

> Jobs using HBaseStorage fail to ship dependency jars
> 
>
> Key: PIG-3285
> URL: https://issues.apache.org/jira/browse/PIG-3285
> Project: Pig
>  Issue Type: Bug
>Reporter: Nick Dimiduk
>Assignee: Nick Dimiduk
> Fix For: 0.13.0
>
> Attachments: 0001-PIG-3285-Add-HBase-dependency-jars.patch, 
> 0001-PIG-3285-Add-HBase-dependency-jars.patch, 
> 0001-PIG-3285-Add-HBase-dependency-jars.patch, 
> 0001-PIG-3285-Add-HBase-dependency-jars.patch, 1.pig, 1.txt, 2.pig
>
>
> Launching a job consuming {{HBaseStorage}} fails out of the box. The user 
> must specify {{-Dpig.additional.jars}} for HBase and all of its dependencies. 
> Exceptions look something like this:
> {noformat}
> 2013-04-19 18:58:39,360 FATAL org.apache.hadoop.mapred.Child: Error running 
> child : java.lang.NoClassDefFoundError: com/google/protobuf/Message
>   at 
> org.apache.hadoop.hbase.io.HbaseObjectWritable.(HbaseObjectWritable.java:266)
>   at org.apache.hadoop.hbase.ipc.Invocation.write(Invocation.java:139)
>   at 
> org.apache.hadoop.hbase.ipc.HBaseClient$Connection.sendParam(HBaseClient.java:612)
>   at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:975)
>   at 
> org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:84)
>   at $Proxy7.getProtocolVersion(Unknown Source)
>   at 
> org.apache.hadoop.hbase.ipc.WritableRpcEngine.getProxy(WritableRpcEngine.java:136)
>   at org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:208)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (PIG-3285) Jobs using HBaseStorage fail to ship dependency jars

[Jarek Jarcec Cecho (JIRA)]

[ 
https://issues.apache.org/jira/browse/PIG-3285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13825844#comment-13825844
 ] 

Jarek Jarcec Cecho commented on PIG-3285:
-

Thank you for your support [~cheolsoo]!

 Do you feel comfortable with making the suggested changes to the patch 
[~ndimiduk] or do you want me to jump in and do them?

> Jobs using HBaseStorage fail to ship dependency jars
> 
>
> Key: PIG-3285
> URL: https://issues.apache.org/jira/browse/PIG-3285
> Project: Pig
>  Issue Type: Bug
>Reporter: Nick Dimiduk
>Assignee: Nick Dimiduk
> Fix For: 0.13.0
>
> Attachments: 0001-PIG-3285-Add-HBase-dependency-jars.patch, 
> 0001-PIG-3285-Add-HBase-dependency-jars.patch, 
> 0001-PIG-3285-Add-HBase-dependency-jars.patch, 1.pig, 1.txt, 2.pig
>
>
> Launching a job consuming {{HBaseStorage}} fails out of the box. The user 
> must specify {{-Dpig.additional.jars}} for HBase and all of its dependencies. 
> Exceptions look something like this:
> {noformat}
> 2013-04-19 18:58:39,360 FATAL org.apache.hadoop.mapred.Child: Error running 
> child : java.lang.NoClassDefFoundError: com/google/protobuf/Message
>   at 
> org.apache.hadoop.hbase.io.HbaseObjectWritable.(HbaseObjectWritable.java:266)
>   at org.apache.hadoop.hbase.ipc.Invocation.write(Invocation.java:139)
>   at 
> org.apache.hadoop.hbase.ipc.HBaseClient$Connection.sendParam(HBaseClient.java:612)
>   at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:975)
>   at 
> org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:84)
>   at $Proxy7.getProtocolVersion(Unknown Source)
>   at 
> org.apache.hadoop.hbase.ipc.WritableRpcEngine.getProxy(WritableRpcEngine.java:136)
>   at org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:208)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (PIG-3285) Jobs using HBaseStorage fail to ship dependency jars

[Jarek Jarcec Cecho (JIRA)]

[ 
https://issues.apache.org/jira/browse/PIG-3285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13825029#comment-13825029
 ] 

Jarek Jarcec Cecho commented on PIG-3285:
-

Yeah, I hear your concern [~ndimiduk]. Nevertheless I'll be more than happy to 
work with you to ensure that all the required combinations will work. I have to 
say though that I'm not a Pig committer and thus we will need someone else to 
do the commit.

> Jobs using HBaseStorage fail to ship dependency jars
> 
>
> Key: PIG-3285
> URL: https://issues.apache.org/jira/browse/PIG-3285
> Project: Pig
>  Issue Type: Bug
>Reporter: Nick Dimiduk
>Assignee: Nick Dimiduk
> Fix For: 0.13.0
>
> Attachments: 0001-PIG-3285-Add-HBase-dependency-jars.patch, 
> 0001-PIG-3285-Add-HBase-dependency-jars.patch, 
> 0001-PIG-3285-Add-HBase-dependency-jars.patch, 1.pig, 1.txt, 2.pig
>
>
> Launching a job consuming {{HBaseStorage}} fails out of the box. The user 
> must specify {{-Dpig.additional.jars}} for HBase and all of its dependencies. 
> Exceptions look something like this:
> {noformat}
> 2013-04-19 18:58:39,360 FATAL org.apache.hadoop.mapred.Child: Error running 
> child : java.lang.NoClassDefFoundError: com/google/protobuf/Message
>   at 
> org.apache.hadoop.hbase.io.HbaseObjectWritable.(HbaseObjectWritable.java:266)
>   at org.apache.hadoop.hbase.ipc.Invocation.write(Invocation.java:139)
>   at 
> org.apache.hadoop.hbase.ipc.HBaseClient$Connection.sendParam(HBaseClient.java:612)
>   at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:975)
>   at 
> org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:84)
>   at $Proxy7.getProtocolVersion(Unknown Source)
>   at 
> org.apache.hadoop.hbase.ipc.WritableRpcEngine.getProxy(WritableRpcEngine.java:136)
>   at org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:208)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (PIG-3285) Jobs using HBaseStorage fail to ship dependency jars

[Jarek Jarcec Cecho (JIRA)]

[ 
https://issues.apache.org/jira/browse/PIG-3285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13824186#comment-13824186
 ] 

Jarek Jarcec Cecho commented on PIG-3285:
-

I really do like idea behind this patch, thank you for driving the effort on 
both sides  [~ndimiduk]! Do you think that we can leave the older code laying 
around for couple of Pig releases and switch between those two conditionally 
based on the HBase version? I'm concerned a bit about backward compatibility 
here as this patch will make pig not working on a HBase releases where it is 
currently working.

> Jobs using HBaseStorage fail to ship dependency jars
> 
>
> Key: PIG-3285
> URL: https://issues.apache.org/jira/browse/PIG-3285
> Project: Pig
>  Issue Type: Bug
>Reporter: Nick Dimiduk
>Assignee: Nick Dimiduk
> Fix For: 0.13.0
>
> Attachments: 0001-PIG-3285-Add-HBase-dependency-jars.patch, 
> 0001-PIG-3285-Add-HBase-dependency-jars.patch, 
> 0001-PIG-3285-Add-HBase-dependency-jars.patch, 1.pig, 1.txt, 2.pig
>
>
> Launching a job consuming {{HBaseStorage}} fails out of the box. The user 
> must specify {{-Dpig.additional.jars}} for HBase and all of its dependencies. 
> Exceptions look something like this:
> {noformat}
> 2013-04-19 18:58:39,360 FATAL org.apache.hadoop.mapred.Child: Error running 
> child : java.lang.NoClassDefFoundError: com/google/protobuf/Message
>   at 
> org.apache.hadoop.hbase.io.HbaseObjectWritable.(HbaseObjectWritable.java:266)
>   at org.apache.hadoop.hbase.ipc.Invocation.write(Invocation.java:139)
>   at 
> org.apache.hadoop.hbase.ipc.HBaseClient$Connection.sendParam(HBaseClient.java:612)
>   at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:975)
>   at 
> org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:84)
>   at $Proxy7.getProtocolVersion(Unknown Source)
>   at 
> org.apache.hadoop.hbase.ipc.WritableRpcEngine.getProxy(WritableRpcEngine.java:136)
>   at org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:208)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (PIG-3285) Jobs using HBaseStorage fail to ship dependency jars

[Jarek Jarcec Cecho (JIRA)]

[ 
https://issues.apache.org/jira/browse/PIG-3285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13824211#comment-13824211
 ] 

Jarek Jarcec Cecho commented on PIG-3285:
-

Yeah I agree, more compatibility is definitely better. Considering that the 
class {{TableMapReduceUtil}} have "always" been there, it might become fairly 
straightforward. Do you think that following snippet might work?

{code}
Method m = TableMapReduceUtil.class.getMethod("addHBaseDependencyJars", 
Configuration.class);
if(m != null) {
  m.invoke(null, job.getConfiguration());
} else {
  // Old code
}
{code}

And then the basic sanity testing would be with changing the HBase version on 
the command line, for example:

{code}
ant clean test -Dtestcase=TestHBaseStorage -Dhbase.version=0.94.1
ant clean test -Dtestcase=TestHBaseStorage -Dhbase.version=0.94.14
{code}

> Jobs using HBaseStorage fail to ship dependency jars
> 
>
> Key: PIG-3285
> URL: https://issues.apache.org/jira/browse/PIG-3285
> Project: Pig
>  Issue Type: Bug
>Reporter: Nick Dimiduk
>Assignee: Nick Dimiduk
> Fix For: 0.13.0
>
> Attachments: 0001-PIG-3285-Add-HBase-dependency-jars.patch, 
> 0001-PIG-3285-Add-HBase-dependency-jars.patch, 
> 0001-PIG-3285-Add-HBase-dependency-jars.patch, 1.pig, 1.txt, 2.pig
>
>
> Launching a job consuming {{HBaseStorage}} fails out of the box. The user 
> must specify {{-Dpig.additional.jars}} for HBase and all of its dependencies. 
> Exceptions look something like this:
> {noformat}
> 2013-04-19 18:58:39,360 FATAL org.apache.hadoop.mapred.Child: Error running 
> child : java.lang.NoClassDefFoundError: com/google/protobuf/Message
>   at 
> org.apache.hadoop.hbase.io.HbaseObjectWritable.(HbaseObjectWritable.java:266)
>   at org.apache.hadoop.hbase.ipc.Invocation.write(Invocation.java:139)
>   at 
> org.apache.hadoop.hbase.ipc.HBaseClient$Connection.sendParam(HBaseClient.java:612)
>   at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:975)
>   at 
> org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:84)
>   at $Proxy7.getProtocolVersion(Unknown Source)
>   at 
> org.apache.hadoop.hbase.ipc.WritableRpcEngine.getProxy(WritableRpcEngine.java:136)
>   at org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:208)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (PIG-3559) Trunk is broken by PIG-3522

[Jarek Jarcec Cecho (JIRA)]

[ 
https://issues.apache.org/jira/browse/PIG-3559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13812062#comment-13812062
 ] 

Jarek Jarcec Cecho commented on PIG-3559:
-

+1 (non-binding)

> Trunk is broken by PIG-3522
> ---
>
> Key: PIG-3559
> URL: https://issues.apache.org/jira/browse/PIG-3559
> Project: Pig
>  Issue Type: Bug
>  Components: build
>Reporter: Cheolsoo Park
>Assignee: Cheolsoo Park
>Priority: Blocker
> Fix For: 0.13.0
>
> Attachments: PIG-3559-1.patch
>
>
> Shock was not deleted from the ant script.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (PIG-3529) Upgrade HBase dependency from 0.95-SNAPSHOT to 0.96

[Jarek Jarcec Cecho (JIRA)]

[ 
https://issues.apache.org/jira/browse/PIG-3529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13803369#comment-13803369
 ] 

Jarek Jarcec Cecho commented on PIG-3529:
-

Thank you [~daijy]!

> Upgrade HBase dependency from 0.95-SNAPSHOT to 0.96
> ---
>
> Key: PIG-3529
> URL: https://issues.apache.org/jira/browse/PIG-3529
> Project: Pig
>  Issue Type: Task
>Affects Versions: 0.12.0
>        Reporter: Jarek Jarcec Cecho
>    Assignee: Jarek Jarcec Cecho
> Fix For: 0.13.0
>
> Attachments: PIG-3529.patch
>
>
> HBase has recently released version 0.96, so we should upgrade from the 
> 0.95.3-SNAPSHOT that has been introduced via PIG-3390.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (PIG-3529) Upgrade HBase dependency from 0.95-SNAPSHOT to 0.96

[Jarek Jarcec Cecho (JIRA)]

[ 
https://issues.apache.org/jira/browse/PIG-3529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13803313#comment-13803313
 ] 

Jarek Jarcec Cecho commented on PIG-3529:
-

Thank you for taking a look on this patch [~daijy]! I've opened follow up JIRA 
PIG-3537 for exactly this :-)

> Upgrade HBase dependency from 0.95-SNAPSHOT to 0.96
> ---
>
> Key: PIG-3529
> URL: https://issues.apache.org/jira/browse/PIG-3529
> Project: Pig
>  Issue Type: Task
>Affects Versions: 0.12.0
>Reporter: Jarek Jarcec Cecho
>    Assignee: Jarek Jarcec Cecho
> Fix For: 0.13.0
>
> Attachments: PIG-3529.patch
>
>
> HBase has recently released version 0.96, so we should upgrade from the 
> 0.95.3-SNAPSHOT that has been introduced via PIG-3390.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Created] (PIG-3537) Tweak build to support HBase with Hadoop 23 profile

[Jarek Jarcec Cecho (JIRA)]

Jarek Jarcec Cecho created PIG-3537:
---

 Summary: Tweak build to support HBase with Hadoop 23 profile
 Key: PIG-3537
 URL: https://issues.apache.org/jira/browse/PIG-3537
 Project: Pig
  Issue Type: Bug
Affects Versions: 0.12.0
Reporter: Jarek Jarcec Cecho
Assignee: Jarek Jarcec Cecho
 Fix For: 0.13.0


We should tweak the ant build to allow building (and running tests) 
HBaseStorage on Hadoop profile 23 (Hadoop 2).



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Updated] (PIG-3529) Upgrade HBase dependency from 0.95-SNAPSHOT to 0.96

[Jarek Jarcec Cecho (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PIG-3529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jarek Jarcec Cecho updated PIG-3529:


Attachment: PIG-3529.patch

The issue was in following exception: 

{code}
Caused by: java.lang.NullPointerException
at 
org.apache.hadoop.hbase.replication.regionserver.Replication.decorateMasterConfiguration(Replication.java:295)
at org.apache.hadoop.hbase.master.HMaster.(HMaster.java:461)
{code}

The problem is that property HBASE_MASTER_LOGCLEANER_PLUGINS that is being 
access in Replication.java:295 is not defined when it's expected to be. I've 
investigated the issue and found out that {{hbase-default.xml}} won't get it 
into the configuration. So, I've added call for {{HBaseConfiguration.create}} 
to propagate all HBase resources to the used configuration object. This wasn't 
an issue with 0.94 and 0.95 because the code in question will be executed only 
if replication is enabled which is not the case on those versions. The 
replication got enabled by default in 0.96 via HBASE-9505. I don't think that 
NPE is correct way how to handle this, so I'll follow up on HBase suggesting to 
be more careful in the code.

Verification steps for HBase 0.95 profile:

{code}
ant clean test -Dtestcase=TestHBaseStorage -Dhbaseversion=95 
-Dprotobuf-java.version=2.5.0
{code}

Verification steps for HBase 0.94 profile:
{code}
ant clean test -Dtestcase=TestHBaseStorage
{code}

I'll file a follow up JIRA to tweak the ant build for fetching proper 
dependencies in order to get the HBase tests working on Hadoop 2 (e.g. that is 
not included in this patch).

> Upgrade HBase dependency from 0.95-SNAPSHOT to 0.96
> ---
>
> Key: PIG-3529
> URL: https://issues.apache.org/jira/browse/PIG-3529
> Project: Pig
>  Issue Type: Task
>    Affects Versions: 0.12.0
>        Reporter: Jarek Jarcec Cecho
>Assignee: Jarek Jarcec Cecho
> Fix For: 0.13.0
>
> Attachments: PIG-3529.patch
>
>
> HBase has recently released version 0.96, so we should upgrade from the 
> 0.95.3-SNAPSHOT that has been introduced via PIG-3390.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Updated] (PIG-3529) Upgrade HBase dependency from 0.95-SNAPSHOT to 0.96

[Jarek Jarcec Cecho (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PIG-3529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jarek Jarcec Cecho updated PIG-3529:


Status: Patch Available  (was: Open)

> Upgrade HBase dependency from 0.95-SNAPSHOT to 0.96
> ---
>
> Key: PIG-3529
> URL: https://issues.apache.org/jira/browse/PIG-3529
> Project: Pig
>  Issue Type: Task
>Affects Versions: 0.12.0
>    Reporter: Jarek Jarcec Cecho
>    Assignee: Jarek Jarcec Cecho
> Fix For: 0.13.0
>
> Attachments: PIG-3529.patch
>
>
> HBase has recently released version 0.96, so we should upgrade from the 
> 0.95.3-SNAPSHOT that has been introduced via PIG-3390.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (PIG-3529) Upgrade HBase dependency from 0.95-SNAPSHOT to 0.96

[Jarek Jarcec Cecho (JIRA)]

[ 
https://issues.apache.org/jira/browse/PIG-3529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13802066#comment-13802066
 ] 

Jarek Jarcec Cecho commented on PIG-3529:
-

Just upgrading the version is not working for me, I'll investigate the failures 
that I'm seeing.

> Upgrade HBase dependency from 0.95-SNAPSHOT to 0.96
> ---
>
> Key: PIG-3529
> URL: https://issues.apache.org/jira/browse/PIG-3529
> Project: Pig
>  Issue Type: Task
>Affects Versions: 0.12.0
>Reporter: Jarek Jarcec Cecho
>Assignee: Jarek Jarcec Cecho
> Fix For: 0.13.0
>
>
> HBase has recently released version 0.96, so we should upgrade from the 
> 0.95.3-SNAPSHOT that has been introduced via PIG-3390.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Created] (PIG-3529) Upgrade HBase dependency from 0.95-SNAPSHOT to 0.96

[Jarek Jarcec Cecho (JIRA)]

Jarek Jarcec Cecho created PIG-3529:
---

 Summary: Upgrade HBase dependency from 0.95-SNAPSHOT to 0.96
 Key: PIG-3529
 URL: https://issues.apache.org/jira/browse/PIG-3529
 Project: Pig
  Issue Type: Task
Affects Versions: 0.12.0
Reporter: Jarek Jarcec Cecho
Assignee: Jarek Jarcec Cecho
 Fix For: 0.13.0


HBase has recently released version 0.96, so we should upgrade from the 
0.95.3-SNAPSHOT that has been introduced via PIG-3390.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (PIG-3519) Remove dependency on uber avro-tools jar

[Jarek Jarcec Cecho (JIRA)]

[ 
https://issues.apache.org/jira/browse/PIG-3519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13797106#comment-13797106
 ] 

Jarek Jarcec Cecho commented on PIG-3519:
-

Thank you [~cheolsoo]!

> Remove dependency on uber avro-tools jar
> 
>
> Key: PIG-3519
> URL: https://issues.apache.org/jira/browse/PIG-3519
> Project: Pig
>  Issue Type: Task
>Affects Versions: 0.12.0
>        Reporter: Jarek Jarcec Cecho
>    Assignee: Jarek Jarcec Cecho
> Fix For: 0.13.0
>
> Attachments: PIG-3519.patch
>
>
> We are currently depending (only for tests though) on uber jar {{avro-tools}} 
> that have 12 MB. It seems that the original intention was to use 
> {{avro-tools}} artifact without all dependencies, however the classifier was 
> incorrectly specified in the {{ivy.xml}}. Considering that we have most of 
> the {{avro-tools}} dependencies on classpath already, I think that it would 
> be cleaner to follow the intention.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Updated] (PIG-3519) Remove dependency on uber avro-tools jar

[Jarek Jarcec Cecho (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PIG-3519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jarek Jarcec Cecho updated PIG-3519:


Status: Patch Available  (was: Open)

> Remove dependency on uber avro-tools jar
> 
>
> Key: PIG-3519
> URL: https://issues.apache.org/jira/browse/PIG-3519
> Project: Pig
>  Issue Type: Task
>Affects Versions: 0.12.0
>    Reporter: Jarek Jarcec Cecho
>    Assignee: Jarek Jarcec Cecho
> Fix For: 0.13.0
>
> Attachments: PIG-3519.patch
>
>
> We are currently depending (only for tests though) on uber jar {{avro-tools}} 
> that have 12 MB. It seems that the original intention was to use 
> {{avro-tools}} artifact without all dependencies, however the classifier was 
> incorrectly specified in the {{ivy.xml}}. Considering that we have most of 
> the {{avro-tools}} dependencies on classpath already, I think that it would 
> be cleaner to follow the intention.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Updated] (PIG-3519) Remove dependency on uber avro-tools jar

[Jarek Jarcec Cecho (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PIG-3519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jarek Jarcec Cecho updated PIG-3519:


Attachment: PIG-3519.patch

> Remove dependency on uber avro-tools jar
> 
>
> Key: PIG-3519
> URL: https://issues.apache.org/jira/browse/PIG-3519
> Project: Pig
>  Issue Type: Task
>Affects Versions: 0.12.0
>    Reporter: Jarek Jarcec Cecho
>    Assignee: Jarek Jarcec Cecho
> Fix For: 0.13.0
>
> Attachments: PIG-3519.patch
>
>
> We are currently depending (only for tests though) on uber jar {{avro-tools}} 
> that have 12 MB. It seems that the original intention was to use 
> {{avro-tools}} artifact without all dependencies, however the classifier was 
> incorrectly specified in the {{ivy.xml}}. Considering that we have most of 
> the {{avro-tools}} dependencies on classpath already, I think that it would 
> be cleaner to follow the intention.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Created] (PIG-3519) Remove dependency on uber avro-tools jar

[Jarek Jarcec Cecho (JIRA)]

Jarek Jarcec Cecho created PIG-3519:
---

 Summary: Remove dependency on uber avro-tools jar
 Key: PIG-3519
 URL: https://issues.apache.org/jira/browse/PIG-3519
 Project: Pig
  Issue Type: Task
Affects Versions: 0.12.0
Reporter: Jarek Jarcec Cecho
Assignee: Jarek Jarcec Cecho
 Fix For: 0.13.0


We are currently depending (only for tests though) on uber jar {{avro-tools}} 
that have 12 MB. It seems that the original intention was to use {{avro-tools}} 
artifact without all dependencies, however the classifier was incorrectly 
specified in the {{ivy.xml}}. Considering that we have most of the 
{{avro-tools}} dependencies on classpath already, I think that it would be 
cleaner to follow the intention.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

Re: [ANNOUNCE] Apache Pig 0.12.0 released

[Jarek Jarcec Cecho]

Thank you for driving the release Daniel!

Jarcec

On Mon, Oct 14, 2013 at 11:36:20AM -0700, Daniel Dai wrote:
> The Pig team is happy to announce the Pig 0.12.0 release.
> 
> Apache Pig provides a high-level data-flow language and execution framework
> for parallel computation on Hadoop clusters.
> More details about Pig can be found at http://pig.apache.org/.
> 
> The highlights of this release are ASSERT operator, Streaming UDF, new
> AvroStorage, IN/CASE operator, BigInteger/BigDecimal data type, support for
> Windows. The details of the release can be found at
> http://pig.apache.org/releases.html.
> 
> Note there are backward incompatible changes and known issues in this
> release. Please check
> https://cwiki.apache.org/confluence/display/PIG/Pig+0.12+Backward+Compatibilityand
> https://cwiki.apache.org/confluence/display/PIG/Pig+0.12.0+Known+Issues for
> details.
> 
> Thanks,
> Daniel
> 
> -- 
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to 
> which it is addressed and may contain information that is confidential, 
> privileged and exempt from disclosure under applicable law. If the reader 
> of this message is not the intended recipient, you are hereby notified that 
> any printing, copying, dissemination, distribution, disclosure or 
> forwarding of this communication is strictly prohibited. If you have 
> received this communication in error, please contact the sender immediately 
> and delete it from your system. Thank You.


signature.asc
Description: Digital signature

Re: [VOTE] Release Pig 0.12.0 (candidate 2)

[Jarek Jarcec Cecho]

+1

* Verified top level files
* Run commit tests

Jarcec

On Mon, Oct 07, 2013 at 12:28:57PM -0700, Daniel Dai wrote:
> Hi,
> 
> I have created a candidate build for Pig 0.12.0.
> 
> Keys used to sign the release are available at
> http://svn.apache.org/viewvc/pig/trunk/KEYS?view=markup
> 
> Please download, test, and try it out:
> 
> http://people.apache.org/~daijy/pig-0.12.0-candidate-2/
> 
> Should we release this? Vote closes on EOD this Thursday, Oct 10th.
> 
> Thanks,
> Daniel
> 
> -- 
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to 
> which it is addressed and may contain information that is confidential, 
> privileged and exempt from disclosure under applicable law. If the reader 
> of this message is not the intended recipient, you are hereby notified that 
> any printing, copying, dissemination, distribution, disclosure or 
> forwarding of this communication is strictly prohibited. If you have 
> received this communication in error, please contact the sender immediately 
> and delete it from your system. Thank You.


signature.asc
Description: Digital signature

[jira] [Created] (PIG-3498) Make pig binary work on both HBase version 0.94 and 0.95

[Jarek Jarcec Cecho (JIRA)]

Jarek Jarcec Cecho created PIG-3498:
---

 Summary: Make pig binary work on both HBase version 0.94 and 0.95
 Key: PIG-3498
 URL: https://issues.apache.org/jira/browse/PIG-3498
 Project: Pig
  Issue Type: Task
Affects Versions: 0.11
Reporter: Jarek Jarcec Cecho


HBase 0.95+ support has been added via PIG-3390. Whereas pig can be compiled 
against both 0.94 and 0.95, due to binary incompatibilities inside HBase, pig 
compiled against HBase 0.95 can't be used against 0.94 and vice versa. 

One of the issue we are facing is HBase class {{RowFilter}}, that changed 
constructor between the two HBase releases:

* HBase 0.94  {{RowFilter(CompareOp, WritableByteArrayComparable)}}
* HBase 0.95 {{RowFilter(CompareO, ByteArrayComparable)}}

We are using children of the classes used in second parameter and therefore the 
same code compiles against both HBase versions. However as the entire 
constructor signature is saved into compiled Java class, generated binaries are 
compatible with only one HBase version. 

As we're releasing only one pig binary, it would be useful to make Pig 
compatible with both versions at the same time.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Updated] (PIG-3496) Propagate HBase 0.95 jars to the backend

[Jarek Jarcec Cecho (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PIG-3496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jarek Jarcec Cecho updated PIG-3496:


Attachment: PIG-3496.patch

> Propagate HBase 0.95 jars to the backend
> 
>
> Key: PIG-3496
> URL: https://issues.apache.org/jira/browse/PIG-3496
> Project: Pig
>  Issue Type: Bug
>Affects Versions: 0.11
>    Reporter: Jarek Jarcec Cecho
>    Assignee: Jarek Jarcec Cecho
>Priority: Minor
> Fix For: 0.13.0
>
> Attachments: PIG-3496.patch
>
>
> In PIG-3390 we've introduced support for HBase 0.95 that introduced a lot of 
> significant changes to HBase. One of the biggest user facing changes was 
> splitting one uber jar file into multiple independent jars (such as 
> {{hbase-common}}, {{hbase-client}}, ...).  
> {{HBaseStore}} have [special 
> code|https://github.com/apache/pig/blob/trunk/src/org/apache/pig/backend/hadoop/hbase/HBaseStorage.java#L724]
>  for propagating HBase jar files and important dependencies to the backend. 
> This logic has not been altered to take into account the different HBase jars 
> after the split and as a result the HBase integration with 0.95 is not 
> working in fully distributed mode (it is work in local mode thought).



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Updated] (PIG-3496) Propagate HBase 0.95 jars to the backend

[Jarek Jarcec Cecho (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PIG-3496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jarek Jarcec Cecho updated PIG-3496:


Fix Version/s: 0.13.0
   Status: Patch Available  (was: Open)

> Propagate HBase 0.95 jars to the backend
> 
>
> Key: PIG-3496
> URL: https://issues.apache.org/jira/browse/PIG-3496
> Project: Pig
>  Issue Type: Bug
>Affects Versions: 0.11
>    Reporter: Jarek Jarcec Cecho
>    Assignee: Jarek Jarcec Cecho
>Priority: Minor
> Fix For: 0.13.0
>
> Attachments: PIG-3496.patch
>
>
> In PIG-3390 we've introduced support for HBase 0.95 that introduced a lot of 
> significant changes to HBase. One of the biggest user facing changes was 
> splitting one uber jar file into multiple independent jars (such as 
> {{hbase-common}}, {{hbase-client}}, ...).  
> {{HBaseStore}} have [special 
> code|https://github.com/apache/pig/blob/trunk/src/org/apache/pig/backend/hadoop/hbase/HBaseStorage.java#L724]
>  for propagating HBase jar files and important dependencies to the backend. 
> This logic has not been altered to take into account the different HBase jars 
> after the split and as a result the HBase integration with 0.95 is not 
> working in fully distributed mode (it is work in local mode thought).



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Created] (PIG-3496) Propagate HBase 0.95 jars to the backend

[Jarek Jarcec Cecho (JIRA)]

Jarek Jarcec Cecho created PIG-3496:
---

 Summary: Propagate HBase 0.95 jars to the backend
 Key: PIG-3496
 URL: https://issues.apache.org/jira/browse/PIG-3496
 Project: Pig
  Issue Type: Bug
Affects Versions: 0.11
Reporter: Jarek Jarcec Cecho
Assignee: Jarek Jarcec Cecho
Priority: Minor


In PIG-3390 we've introduced support for HBase 0.95 that introduced a lot of 
significant changes to HBase. One of the biggest user facing changes was 
splitting one uber jar file into multiple independent jars (such as 
{{hbase-common}}, {{hbase-client}}, ...).  

{{HBaseStore}} have [special 
code|https://github.com/apache/pig/blob/trunk/src/org/apache/pig/backend/hadoop/hbase/HBaseStorage.java#L724]
 for propagating HBase jar files and important dependencies to the backend. 
This logic has not been altered to take into account the different HBase jars 
after the split and as a result the HBase integration with 0.95 is not working 
in fully distributed mode (it is work in local mode thought).



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Updated] (PIG-3469) Skewed join can cause unrecoverable NullPointerException when one of its inputs is missing.

[Jarek Jarcec Cecho (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PIG-3469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jarek Jarcec Cecho updated PIG-3469:


Attachment: PIG-3469.patch

> Skewed join can cause unrecoverable NullPointerException when one of its 
> inputs is missing.
> ---
>
> Key: PIG-3469
> URL: https://issues.apache.org/jira/browse/PIG-3469
> Project: Pig
>  Issue Type: Bug
>Affects Versions: 0.11
> Environment: Apache Pig version 0.11.0-cdh4.4.0
> Happens in both local execution environment (os x) and cluster environment 
> (linux)
>Reporter: Christon DeWan
>Assignee: Jarek Jarcec Cecho
> Attachments: PIG-3469.patch, PIG-3469.patch, PIG-3469.patch
>
>
> Run this script in the local execution environment (affects cluster mode too):
> {noformat}
> %declare DATA_EXISTS /tmp/test_data_exists.tsv
> %declare DATA_MISSING /tmp/test_data_missing.tsv
> %declare DUMMY `bash -c '(for (( i=0; \$i < 10; i++ )); do echo \$i; done) > 
> /tmp/test_data_exists.tsv; true'`
> exists = LOAD '$DATA_EXISTS' AS (a:long);
> missing = LOAD '$DATA_MISSING' AS (a:long);
> missing = FOREACH ( GROUP missing BY a ) GENERATE $0 AS a, COUNT_STAR($1);
> joined = JOIN exists BY a, missing BY a USING 'skewed';
> STORE joined INTO '/tmp/test_out.tsv';
> {noformat}
> Results in NullPointerException which halts entire pig execution, including 
> unrelated jobs. Expected: only dependencies of the error'd LOAD statement 
> should fail. 
> Error:
> {noformat}
> 2013-09-18 11:42:31,518 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 
> 2017: Internal error creating job configuration.
> 2013-09-18 11:42:31,518 [main] ERROR org.apache.pig.tools.grunt.Grunt - 
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobCreationException:
>  ERROR 2017: Internal error creating job configuration.
>   at 
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.getJob(JobControlCompiler.java:848)
>   at 
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.compile(JobControlCompiler.java:294)
>   at 
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:177)
>   at org.apache.pig.PigServer.launchPlan(PigServer.java:1266)
>   at 
> org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1251)
>   at org.apache.pig.PigServer.execute(PigServer.java:1241)
>   at org.apache.pig.PigServer.executeBatch(PigServer.java:335)
>   at 
> org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:137)
>   at 
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:198)
>   at 
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:170)
>   at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:84)
>   at org.apache.pig.Main.run(Main.java:604)
>   at org.apache.pig.Main.main(Main.java:157)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>   at java.lang.reflect.Method.invoke(Method.java:597)
>   at org.apache.hadoop.util.RunJar.main(RunJar.java:208)
> Caused by: java.lang.NullPointerException
>   at 
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.adjustNumReducers(JobControlCompiler.java:868)
>   at 
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.getJob(JobControlCompiler.java:480)
>   ... 17 more
> {noformat}
> Script above is as small as I can make it while still reproducing the issue. 
> Removing the group-foreach causes the join to fail harmlessly (not stopping 
> pig execution), as does using the default join. Did not occur on 0.10.1.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Updated] (PIG-3469) Skewed join can cause unrecoverable NullPointerException when one of its inputs is missing.

[Jarek Jarcec Cecho (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PIG-3469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jarek Jarcec Cecho updated PIG-3469:


Attachment: PIG-3469.patch

Attached patch with test case to ensure that PigServer won't die on the 
incorrect input.

> Skewed join can cause unrecoverable NullPointerException when one of its 
> inputs is missing.
> ---
>
> Key: PIG-3469
> URL: https://issues.apache.org/jira/browse/PIG-3469
> Project: Pig
>  Issue Type: Bug
>Affects Versions: 0.11
> Environment: Apache Pig version 0.11.0-cdh4.4.0
> Happens in both local execution environment (os x) and cluster environment 
> (linux)
>Reporter: Christon DeWan
>Assignee: Jarek Jarcec Cecho
> Attachments: PIG-3469.patch, PIG-3469.patch
>
>
> Run this script in the local execution environment (affects cluster mode too):
> {noformat}
> %declare DATA_EXISTS /tmp/test_data_exists.tsv
> %declare DATA_MISSING /tmp/test_data_missing.tsv
> %declare DUMMY `bash -c '(for (( i=0; \$i < 10; i++ )); do echo \$i; done) > 
> /tmp/test_data_exists.tsv; true'`
> exists = LOAD '$DATA_EXISTS' AS (a:long);
> missing = LOAD '$DATA_MISSING' AS (a:long);
> missing = FOREACH ( GROUP missing BY a ) GENERATE $0 AS a, COUNT_STAR($1);
> joined = JOIN exists BY a, missing BY a USING 'skewed';
> STORE joined INTO '/tmp/test_out.tsv';
> {noformat}
> Results in NullPointerException which halts entire pig execution, including 
> unrelated jobs. Expected: only dependencies of the error'd LOAD statement 
> should fail. 
> Error:
> {noformat}
> 2013-09-18 11:42:31,518 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 
> 2017: Internal error creating job configuration.
> 2013-09-18 11:42:31,518 [main] ERROR org.apache.pig.tools.grunt.Grunt - 
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobCreationException:
>  ERROR 2017: Internal error creating job configuration.
>   at 
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.getJob(JobControlCompiler.java:848)
>   at 
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.compile(JobControlCompiler.java:294)
>   at 
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:177)
>   at org.apache.pig.PigServer.launchPlan(PigServer.java:1266)
>   at 
> org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1251)
>   at org.apache.pig.PigServer.execute(PigServer.java:1241)
>   at org.apache.pig.PigServer.executeBatch(PigServer.java:335)
>   at 
> org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:137)
>   at 
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:198)
>   at 
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:170)
>   at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:84)
>   at org.apache.pig.Main.run(Main.java:604)
>   at org.apache.pig.Main.main(Main.java:157)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>   at java.lang.reflect.Method.invoke(Method.java:597)
>   at org.apache.hadoop.util.RunJar.main(RunJar.java:208)
> Caused by: java.lang.NullPointerException
>   at 
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.adjustNumReducers(JobControlCompiler.java:868)
>   at 
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.getJob(JobControlCompiler.java:480)
>   ... 17 more
> {noformat}
> Script above is as small as I can make it while still reproducing the issue. 
> Removing the group-foreach causes the join to fail harmlessly (not stopping 
> pig execution), as does using the default join. Did not occur on 0.10.1.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (PIG-3469) Skewed join can cause unrecoverable NullPointerException when one of its inputs is missing.

[Jarek Jarcec Cecho (JIRA)]

[ 
https://issues.apache.org/jira/browse/PIG-3469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13778857#comment-13778857
 ] 

Jarek Jarcec Cecho commented on PIG-3469:
-

[~daijy], the patch is quite simple, so I'm wondering if there is any chance to 
get it into 0.12?

> Skewed join can cause unrecoverable NullPointerException when one of its 
> inputs is missing.
> ---
>
> Key: PIG-3469
> URL: https://issues.apache.org/jira/browse/PIG-3469
> Project: Pig
>  Issue Type: Bug
>Affects Versions: 0.11
> Environment: Apache Pig version 0.11.0-cdh4.4.0
> Happens in both local execution environment (os x) and cluster environment 
> (linux)
>Reporter: Christon DeWan
>Assignee: Jarek Jarcec Cecho
> Attachments: PIG-3469.patch
>
>
> Run this script in the local execution environment (affects cluster mode too):
> {noformat}
> %declare DATA_EXISTS /tmp/test_data_exists.tsv
> %declare DATA_MISSING /tmp/test_data_missing.tsv
> %declare DUMMY `bash -c '(for (( i=0; \$i < 10; i++ )); do echo \$i; done) > 
> /tmp/test_data_exists.tsv; true'`
> exists = LOAD '$DATA_EXISTS' AS (a:long);
> missing = LOAD '$DATA_MISSING' AS (a:long);
> missing = FOREACH ( GROUP missing BY a ) GENERATE $0 AS a, COUNT_STAR($1);
> joined = JOIN exists BY a, missing BY a USING 'skewed';
> STORE joined INTO '/tmp/test_out.tsv';
> {noformat}
> Results in NullPointerException which halts entire pig execution, including 
> unrelated jobs. Expected: only dependencies of the error'd LOAD statement 
> should fail. 
> Error:
> {noformat}
> 2013-09-18 11:42:31,518 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 
> 2017: Internal error creating job configuration.
> 2013-09-18 11:42:31,518 [main] ERROR org.apache.pig.tools.grunt.Grunt - 
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobCreationException:
>  ERROR 2017: Internal error creating job configuration.
>   at 
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.getJob(JobControlCompiler.java:848)
>   at 
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.compile(JobControlCompiler.java:294)
>   at 
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:177)
>   at org.apache.pig.PigServer.launchPlan(PigServer.java:1266)
>   at 
> org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1251)
>   at org.apache.pig.PigServer.execute(PigServer.java:1241)
>   at org.apache.pig.PigServer.executeBatch(PigServer.java:335)
>   at 
> org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:137)
>   at 
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:198)
>   at 
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:170)
>   at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:84)
>   at org.apache.pig.Main.run(Main.java:604)
>   at org.apache.pig.Main.main(Main.java:157)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>   at java.lang.reflect.Method.invoke(Method.java:597)
>   at org.apache.hadoop.util.RunJar.main(RunJar.java:208)
> Caused by: java.lang.NullPointerException
>   at 
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.adjustNumReducers(JobControlCompiler.java:868)
>   at 
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.getJob(JobControlCompiler.java:480)
>   ... 17 more
> {noformat}
> Script above is as small as I can make it while still reproducing the issue. 
> Removing the group-foreach causes the join to fail harmlessly (not stopping 
> pig execution), as does using the default join. Did not occur on 0.10.1.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (PIG-3469) Skewed join can cause unrecoverable NullPointerException when one of its inputs is missing.

[Jarek Jarcec Cecho (JIRA)]

[ 
https://issues.apache.org/jira/browse/PIG-3469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13778851#comment-13778851
 ] 

Jarek Jarcec Cecho commented on PIG-3469:
-

The NPE is thrown in {{JobControlCompiler.adjustNumReducers}} when getting 
successor nodes from plan. This method is called from 
{{JobControlCompiler.getJob()}} that is called from 
{{MapReduceLauncher.launchPig()}} through {{JobControlCompiler.compile()}}. 
There is a while loop in {{MapReduceLauncher.launchPig()}}, that is calling the 
{{compile()}} on every iteration. The loop itself is however changing the plan, 
by for example removing failed failed jobs with all their dependencies. As a 
result the call {{getSuccessors()}} can return {{NULL}} in case of a job 
failure as the nodes has been removed from the plan.

I'm attaching patch that will verify that the {{getSuccessors()}} is indeed not 
{{NULL}} prior using it. I've run all the tests and they seem to be passing.

> Skewed join can cause unrecoverable NullPointerException when one of its 
> inputs is missing.
> ---
>
> Key: PIG-3469
> URL: https://issues.apache.org/jira/browse/PIG-3469
> Project: Pig
>  Issue Type: Bug
>Affects Versions: 0.11
> Environment: Apache Pig version 0.11.0-cdh4.4.0
> Happens in both local execution environment (os x) and cluster environment 
> (linux)
>    Reporter: Christon DeWan
>Assignee: Jarek Jarcec Cecho
> Attachments: PIG-3469.patch
>
>
> Run this script in the local execution environment (affects cluster mode too):
> {noformat}
> %declare DATA_EXISTS /tmp/test_data_exists.tsv
> %declare DATA_MISSING /tmp/test_data_missing.tsv
> %declare DUMMY `bash -c '(for (( i=0; \$i < 10; i++ )); do echo \$i; done) > 
> /tmp/test_data_exists.tsv; true'`
> exists = LOAD '$DATA_EXISTS' AS (a:long);
> missing = LOAD '$DATA_MISSING' AS (a:long);
> missing = FOREACH ( GROUP missing BY a ) GENERATE $0 AS a, COUNT_STAR($1);
> joined = JOIN exists BY a, missing BY a USING 'skewed';
> STORE joined INTO '/tmp/test_out.tsv';
> {noformat}
> Results in NullPointerException which halts entire pig execution, including 
> unrelated jobs. Expected: only dependencies of the error'd LOAD statement 
> should fail. 
> Error:
> {noformat}
> 2013-09-18 11:42:31,518 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 
> 2017: Internal error creating job configuration.
> 2013-09-18 11:42:31,518 [main] ERROR org.apache.pig.tools.grunt.Grunt - 
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobCreationException:
>  ERROR 2017: Internal error creating job configuration.
>   at 
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.getJob(JobControlCompiler.java:848)
>   at 
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.compile(JobControlCompiler.java:294)
>   at 
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:177)
>   at org.apache.pig.PigServer.launchPlan(PigServer.java:1266)
>   at 
> org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1251)
>   at org.apache.pig.PigServer.execute(PigServer.java:1241)
>   at org.apache.pig.PigServer.executeBatch(PigServer.java:335)
>   at 
> org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:137)
>   at 
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:198)
>   at 
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:170)
>   at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:84)
>   at org.apache.pig.Main.run(Main.java:604)
>   at org.apache.pig.Main.main(Main.java:157)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>   at java.lang.reflect.Method.invoke(Method.java:597)
>   at org.apache.hadoop.util.RunJar.main(RunJar.java:208)
> Caused by: java.lang.NullPointerException
>   at 
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.adjustNumReducers(JobControlCompiler.java:868)
>   at 
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.getJob(JobControlCompiler.java:480)
>   ... 17 more
> {noformat}
> Script 

[jira] [Updated] (PIG-3469) Skewed join can cause unrecoverable NullPointerException when one of its inputs is missing.

[Jarek Jarcec Cecho (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PIG-3469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jarek Jarcec Cecho updated PIG-3469:


Status: Patch Available  (was: Open)

> Skewed join can cause unrecoverable NullPointerException when one of its 
> inputs is missing.
> ---
>
> Key: PIG-3469
> URL: https://issues.apache.org/jira/browse/PIG-3469
> Project: Pig
>  Issue Type: Bug
>Affects Versions: 0.11
> Environment: Apache Pig version 0.11.0-cdh4.4.0
> Happens in both local execution environment (os x) and cluster environment 
> (linux)
>Reporter: Christon DeWan
>Assignee: Jarek Jarcec Cecho
> Attachments: PIG-3469.patch
>
>
> Run this script in the local execution environment (affects cluster mode too):
> {noformat}
> %declare DATA_EXISTS /tmp/test_data_exists.tsv
> %declare DATA_MISSING /tmp/test_data_missing.tsv
> %declare DUMMY `bash -c '(for (( i=0; \$i < 10; i++ )); do echo \$i; done) > 
> /tmp/test_data_exists.tsv; true'`
> exists = LOAD '$DATA_EXISTS' AS (a:long);
> missing = LOAD '$DATA_MISSING' AS (a:long);
> missing = FOREACH ( GROUP missing BY a ) GENERATE $0 AS a, COUNT_STAR($1);
> joined = JOIN exists BY a, missing BY a USING 'skewed';
> STORE joined INTO '/tmp/test_out.tsv';
> {noformat}
> Results in NullPointerException which halts entire pig execution, including 
> unrelated jobs. Expected: only dependencies of the error'd LOAD statement 
> should fail. 
> Error:
> {noformat}
> 2013-09-18 11:42:31,518 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 
> 2017: Internal error creating job configuration.
> 2013-09-18 11:42:31,518 [main] ERROR org.apache.pig.tools.grunt.Grunt - 
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobCreationException:
>  ERROR 2017: Internal error creating job configuration.
>   at 
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.getJob(JobControlCompiler.java:848)
>   at 
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.compile(JobControlCompiler.java:294)
>   at 
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:177)
>   at org.apache.pig.PigServer.launchPlan(PigServer.java:1266)
>   at 
> org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1251)
>   at org.apache.pig.PigServer.execute(PigServer.java:1241)
>   at org.apache.pig.PigServer.executeBatch(PigServer.java:335)
>   at 
> org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:137)
>   at 
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:198)
>   at 
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:170)
>   at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:84)
>   at org.apache.pig.Main.run(Main.java:604)
>   at org.apache.pig.Main.main(Main.java:157)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>   at java.lang.reflect.Method.invoke(Method.java:597)
>   at org.apache.hadoop.util.RunJar.main(RunJar.java:208)
> Caused by: java.lang.NullPointerException
>   at 
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.adjustNumReducers(JobControlCompiler.java:868)
>   at 
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.getJob(JobControlCompiler.java:480)
>   ... 17 more
> {noformat}
> Script above is as small as I can make it while still reproducing the issue. 
> Removing the group-foreach causes the join to fail harmlessly (not stopping 
> pig execution), as does using the default join. Did not occur on 0.10.1.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (PIG-3469) Skewed join can cause unrecoverable NullPointerException when one of its inputs is missing.

[Jarek Jarcec Cecho (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PIG-3469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jarek Jarcec Cecho updated PIG-3469:


Attachment: PIG-3469.patch

> Skewed join can cause unrecoverable NullPointerException when one of its 
> inputs is missing.
> ---
>
> Key: PIG-3469
> URL: https://issues.apache.org/jira/browse/PIG-3469
> Project: Pig
>  Issue Type: Bug
>Affects Versions: 0.11
> Environment: Apache Pig version 0.11.0-cdh4.4.0
> Happens in both local execution environment (os x) and cluster environment 
> (linux)
>Reporter: Christon DeWan
>Assignee: Jarek Jarcec Cecho
> Attachments: PIG-3469.patch
>
>
> Run this script in the local execution environment (affects cluster mode too):
> {noformat}
> %declare DATA_EXISTS /tmp/test_data_exists.tsv
> %declare DATA_MISSING /tmp/test_data_missing.tsv
> %declare DUMMY `bash -c '(for (( i=0; \$i < 10; i++ )); do echo \$i; done) > 
> /tmp/test_data_exists.tsv; true'`
> exists = LOAD '$DATA_EXISTS' AS (a:long);
> missing = LOAD '$DATA_MISSING' AS (a:long);
> missing = FOREACH ( GROUP missing BY a ) GENERATE $0 AS a, COUNT_STAR($1);
> joined = JOIN exists BY a, missing BY a USING 'skewed';
> STORE joined INTO '/tmp/test_out.tsv';
> {noformat}
> Results in NullPointerException which halts entire pig execution, including 
> unrelated jobs. Expected: only dependencies of the error'd LOAD statement 
> should fail. 
> Error:
> {noformat}
> 2013-09-18 11:42:31,518 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 
> 2017: Internal error creating job configuration.
> 2013-09-18 11:42:31,518 [main] ERROR org.apache.pig.tools.grunt.Grunt - 
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobCreationException:
>  ERROR 2017: Internal error creating job configuration.
>   at 
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.getJob(JobControlCompiler.java:848)
>   at 
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.compile(JobControlCompiler.java:294)
>   at 
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:177)
>   at org.apache.pig.PigServer.launchPlan(PigServer.java:1266)
>   at 
> org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1251)
>   at org.apache.pig.PigServer.execute(PigServer.java:1241)
>   at org.apache.pig.PigServer.executeBatch(PigServer.java:335)
>   at 
> org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:137)
>   at 
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:198)
>   at 
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:170)
>   at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:84)
>   at org.apache.pig.Main.run(Main.java:604)
>   at org.apache.pig.Main.main(Main.java:157)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>   at java.lang.reflect.Method.invoke(Method.java:597)
>   at org.apache.hadoop.util.RunJar.main(RunJar.java:208)
> Caused by: java.lang.NullPointerException
>   at 
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.adjustNumReducers(JobControlCompiler.java:868)
>   at 
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.getJob(JobControlCompiler.java:480)
>   ... 17 more
> {noformat}
> Script above is as small as I can make it while still reproducing the issue. 
> Removing the group-foreach causes the join to fail harmlessly (not stopping 
> pig execution), as does using the default join. Did not occur on 0.10.1.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (PIG-3390) Make pig working with HBase 0.95

[Jarek Jarcec Cecho (JIRA)]

[ 
https://issues.apache.org/jira/browse/PIG-3390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13778803#comment-13778803
 ] 

Jarek Jarcec Cecho commented on PIG-3390:
-

That is correct [~ashishujjain], as per earlier comment the change currently 
works only for Hadoop 1. My intention is to tweak it for Hadoop 2 when 
upgrading the dependency from SNAPSHOT to final 0.96.0.

> Make pig working with HBase 0.95
> 
>
> Key: PIG-3390
> URL: https://issues.apache.org/jira/browse/PIG-3390
> Project: Pig
>  Issue Type: New Feature
>Affects Versions: 0.11
>        Reporter: Jarek Jarcec Cecho
>    Assignee: Jarek Jarcec Cecho
> Fix For: 0.12.0
>
> Attachments: PIG-3390.patch, PIG-3390.patch, PIG-3390.patch
>
>
> The HBase 0.95 changed API in incompatible way. Following APIs that 
> {{HBaseStorage}} in Pig uses are no longer available:
> * {{Mutation.setWriteToWAL(Boolean)}}
> * {{Scan.write(DataOutput)}}
> Also in addition the HBase is no longer available as one monolithic archive 
> with entire functionality, but was broken down into smaller pieces such as 
> {{hbase-client}}, {{hbase-server}}, ...

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (PIG-3469) Skewed join can cause unrecoverable NullPointerException when one of its inputs is missing.

[Jarek Jarcec Cecho (JIRA)]

[ 
https://issues.apache.org/jira/browse/PIG-3469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13778225#comment-13778225
 ] 

Jarek Jarcec Cecho commented on PIG-3469:
-

I believe that I do have understanding of this issue, will upload patch after 
running all tests.

> Skewed join can cause unrecoverable NullPointerException when one of its 
> inputs is missing.
> ---
>
> Key: PIG-3469
> URL: https://issues.apache.org/jira/browse/PIG-3469
> Project: Pig
>  Issue Type: Bug
>Affects Versions: 0.11
> Environment: Apache Pig version 0.11.0-cdh4.4.0
> Happens in both local execution environment (os x) and cluster environment 
> (linux)
>Reporter: Christon DeWan
>Assignee: Jarek Jarcec Cecho
>
> Run this script in the local execution environment (affects cluster mode too):
> {noformat}
> %declare DATA_EXISTS /tmp/test_data_exists.tsv
> %declare DATA_MISSING /tmp/test_data_missing.tsv
> %declare DUMMY `bash -c '(for (( i=0; \$i < 10; i++ )); do echo \$i; done) > 
> /tmp/test_data_exists.tsv; true'`
> exists = LOAD '$DATA_EXISTS' AS (a:long);
> missing = LOAD '$DATA_MISSING' AS (a:long);
> missing = FOREACH ( GROUP missing BY a ) GENERATE $0 AS a, COUNT_STAR($1);
> joined = JOIN exists BY a, missing BY a USING 'skewed';
> STORE joined INTO '/tmp/test_out.tsv';
> {noformat}
> Results in NullPointerException which halts entire pig execution, including 
> unrelated jobs. Expected: only dependencies of the error'd LOAD statement 
> should fail. 
> Error:
> {noformat}
> 2013-09-18 11:42:31,518 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 
> 2017: Internal error creating job configuration.
> 2013-09-18 11:42:31,518 [main] ERROR org.apache.pig.tools.grunt.Grunt - 
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobCreationException:
>  ERROR 2017: Internal error creating job configuration.
>   at 
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.getJob(JobControlCompiler.java:848)
>   at 
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.compile(JobControlCompiler.java:294)
>   at 
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:177)
>   at org.apache.pig.PigServer.launchPlan(PigServer.java:1266)
>   at 
> org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1251)
>   at org.apache.pig.PigServer.execute(PigServer.java:1241)
>   at org.apache.pig.PigServer.executeBatch(PigServer.java:335)
>   at 
> org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:137)
>   at 
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:198)
>   at 
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:170)
>   at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:84)
>   at org.apache.pig.Main.run(Main.java:604)
>   at org.apache.pig.Main.main(Main.java:157)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>   at java.lang.reflect.Method.invoke(Method.java:597)
>   at org.apache.hadoop.util.RunJar.main(RunJar.java:208)
> Caused by: java.lang.NullPointerException
>   at 
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.adjustNumReducers(JobControlCompiler.java:868)
>   at 
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.getJob(JobControlCompiler.java:480)
>   ... 17 more
> {noformat}
> Script above is as small as I can make it while still reproducing the issue. 
> Removing the group-foreach causes the join to fail harmlessly (not stopping 
> pig execution), as does using the default join. Did not occur on 0.10.1.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Assigned] (PIG-3469) Skewed join can cause unrecoverable NullPointerException when one of its inputs is missing.

[Jarek Jarcec Cecho (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PIG-3469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jarek Jarcec Cecho reassigned PIG-3469:
---

Assignee: Jarek Jarcec Cecho

> Skewed join can cause unrecoverable NullPointerException when one of its 
> inputs is missing.
> ---
>
> Key: PIG-3469
> URL: https://issues.apache.org/jira/browse/PIG-3469
> Project: Pig
>  Issue Type: Bug
>Affects Versions: 0.11
> Environment: Apache Pig version 0.11.0-cdh4.4.0
> Happens in both local execution environment (os x) and cluster environment 
> (linux)
>Reporter: Christon DeWan
>Assignee: Jarek Jarcec Cecho
>
> Run this script in the local execution environment (affects cluster mode too):
> {noformat}
> %declare DATA_EXISTS /tmp/test_data_exists.tsv
> %declare DATA_MISSING /tmp/test_data_missing.tsv
> %declare DUMMY `bash -c '(for (( i=0; \$i < 10; i++ )); do echo \$i; done) > 
> /tmp/test_data_exists.tsv; true'`
> exists = LOAD '$DATA_EXISTS' AS (a:long);
> missing = LOAD '$DATA_MISSING' AS (a:long);
> missing = FOREACH ( GROUP missing BY a ) GENERATE $0 AS a, COUNT_STAR($1);
> joined = JOIN exists BY a, missing BY a USING 'skewed';
> STORE joined INTO '/tmp/test_out.tsv';
> {noformat}
> Results in NullPointerException which halts entire pig execution, including 
> unrelated jobs. Expected: only dependencies of the error'd LOAD statement 
> should fail. 
> Error:
> {noformat}
> 2013-09-18 11:42:31,518 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 
> 2017: Internal error creating job configuration.
> 2013-09-18 11:42:31,518 [main] ERROR org.apache.pig.tools.grunt.Grunt - 
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobCreationException:
>  ERROR 2017: Internal error creating job configuration.
>   at 
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.getJob(JobControlCompiler.java:848)
>   at 
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.compile(JobControlCompiler.java:294)
>   at 
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:177)
>   at org.apache.pig.PigServer.launchPlan(PigServer.java:1266)
>   at 
> org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1251)
>   at org.apache.pig.PigServer.execute(PigServer.java:1241)
>   at org.apache.pig.PigServer.executeBatch(PigServer.java:335)
>   at 
> org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:137)
>   at 
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:198)
>   at 
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:170)
>   at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:84)
>   at org.apache.pig.Main.run(Main.java:604)
>   at org.apache.pig.Main.main(Main.java:157)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>   at java.lang.reflect.Method.invoke(Method.java:597)
>   at org.apache.hadoop.util.RunJar.main(RunJar.java:208)
> Caused by: java.lang.NullPointerException
>   at 
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.adjustNumReducers(JobControlCompiler.java:868)
>   at 
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.getJob(JobControlCompiler.java:480)
>   ... 17 more
> {noformat}
> Script above is as small as I can make it while still reproducing the issue. 
> Removing the group-foreach causes the join to fail harmlessly (not stopping 
> pig execution), as does using the default join. Did not occur on 0.10.1.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (PIG-3390) Make pig working with HBase 0.95

[Jarek Jarcec Cecho (JIRA)]

[ 
https://issues.apache.org/jira/browse/PIG-3390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13765776#comment-13765776
 ] 

Jarek Jarcec Cecho commented on PIG-3390:
-

Thank you [~daijy] for committing this patch! I'll keep an eye on HBase and 
will update the dependency as soon as it will be released!

> Make pig working with HBase 0.95
> 
>
> Key: PIG-3390
> URL: https://issues.apache.org/jira/browse/PIG-3390
> Project: Pig
>  Issue Type: New Feature
>Affects Versions: 0.11
>Reporter: Jarek Jarcec Cecho
>    Assignee: Jarek Jarcec Cecho
> Fix For: 0.12
>
> Attachments: PIG-3390.patch, PIG-3390.patch, PIG-3390.patch
>
>
> The HBase 0.95 changed API in incompatible way. Following APIs that 
> {{HBaseStorage}} in Pig uses are no longer available:
> * {{Mutation.setWriteToWAL(Boolean)}}
> * {{Scan.write(DataOutput)}}
> Also in addition the HBase is no longer available as one monolithic archive 
> with entire functionality, but was broken down into smaller pieces such as 
> {{hbase-client}}, {{hbase-server}}, ...

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (PIG-3390) Make pig working with HBase 0.95

[Jarek Jarcec Cecho (JIRA)]

[ 
https://issues.apache.org/jira/browse/PIG-3390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13765704#comment-13765704
 ] 

Jarek Jarcec Cecho commented on PIG-3390:
-

Hi [~daijy],
thank you very much for taking a look on the patch! Indeed the dependency on a 
0.95.3 snapshot is intentional - version 0.95.1 can't be used as it do not have 
HBASE-9093 and the 0.95.2 jars are corrupted (please see details in 
[HIVE-4388|https://issues.apache.org/jira/browse/HIVE-4388?focusedCommentId=13743075#comment-13743075]).
 

> Make pig working with HBase 0.95
> 
>
> Key: PIG-3390
> URL: https://issues.apache.org/jira/browse/PIG-3390
> Project: Pig
>  Issue Type: New Feature
>Affects Versions: 0.11
>Reporter: Jarek Jarcec Cecho
>    Assignee: Jarek Jarcec Cecho
> Fix For: 0.12
>
> Attachments: PIG-3390.patch, PIG-3390.patch, PIG-3390.patch
>
>
> The HBase 0.95 changed API in incompatible way. Following APIs that 
> {{HBaseStorage}} in Pig uses are no longer available:
> * {{Mutation.setWriteToWAL(Boolean)}}
> * {{Scan.write(DataOutput)}}
> Also in addition the HBase is no longer available as one monolithic archive 
> with entire functionality, but was broken down into smaller pieces such as 
> {{hbase-client}}, {{hbase-server}}, ...

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Re: [ANNOUNCE] Congratulations to our new PMC members Rohini Palaniswamy and Cheolsoo Park

[Jarek Jarcec Cecho]

Congratulations Rohini and Cheolsoo, awesome work!

Jarcec

On Wed, Sep 11, 2013 at 04:24:21PM -0700, Julien Le Dem wrote:
> Please welcome Rohini Palaniswamy and Cheolsoo Park as our latest Pig PMC 
> members.
> 
> Congrats Rohini and Cheolsoo !


signature.asc
Description: Digital signature

Re: Welcome new Pig Committer - Koji Noguchi

[Jarek Jarcec Cecho]

Congratulations Koji, awesome job!

Jarcec

On Tue, Sep 10, 2013 at 08:58:55AM -0700, Olga Natkovich wrote:
> It is my pleasure to announce that Koji Noguchi became the newest addition to 
> the Pig Committers!
> 
> Koji has been actively contributing to Pig for over a year now and has been a 
> part of larger Hadoop community (including Hadoop Committer) for many years 
> now.
> 
> Please, join me in congratulating Koji!
> 
> Olga


signature.asc
Description: Digital signature

Re: Are we ready for Pig 0.12.0 release?

[Jarek Jarcec Cecho]

I've cleaned up PIG-3390 and uploaded patch.

Jarcec

On Tue, Sep 03, 2013 at 11:08:07AM -0700, Jarek Jarcec Cecho wrote:
> I'll try to clean up and finish PIG-3390 (HBase 0.95 support) this week, to 
> see if it can be included.
> 
> Jarcec
> 
> On Tue, Sep 03, 2013 at 10:56:42AM -0700, Cheolsoo Park wrote:
> > +1. I will go through my jiras this week.
> > 
> > 
> > On Tue, Sep 3, 2013 at 10:34 AM, Daniel Dai  wrote:
> > 
> > > Hi, All,
> > > It has been more than half a year since initial Pig 0.11 release. I'd like
> > > roll a Pig 0.12 release around the end of September or the beginning of
> > > October. Let me know if it is possible.
> > >
> > > Proposed schedule:
> > > 1. Commit all major features (1-2 weeks)
> > > 2. Branching Pig 0.12
> > > 3. Commit remaining patches (1-2 weeks)
> > > 4. Wrapping up, document (1 week)
> > >
> > > If you have patches want to get in, please make sure the Jira ticket has
> > > fix version set to 0.12. If the patches originally set to 0.12 and you
> > > think you can delay, please mark the fix version to either 0.13.0 or
> > > 0.12.1.
> > >
> > > Thanks,
> > > Daniel
> > >
> > > --
> > > CONFIDENTIALITY NOTICE
> > > NOTICE: This message is intended for the use of the individual or entity 
> > > to
> > > which it is addressed and may contain information that is confidential,
> > > privileged and exempt from disclosure under applicable law. If the reader
> > > of this message is not the intended recipient, you are hereby notified 
> > > that
> > > any printing, copying, dissemination, distribution, disclosure or
> > > forwarding of this communication is strictly prohibited. If you have
> > > received this communication in error, please contact the sender 
> > > immediately
> > > and delete it from your system. Thank You.
> > >




signature.asc
Description: Digital signature

[jira] [Updated] (PIG-3390) Make pig working with HBase 0.95

[Jarek Jarcec Cecho (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PIG-3390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jarek Jarcec Cecho updated PIG-3390:


Attachment: PIG-3390.patch

Attaching completed patch.

The {{Mutation.setWriteToWAL(Boolean)}} has been returned via HBASE-9093 in 
0.95.2, so I've removed the reflection from the patch. As maven jars for HBase 
0.95.2 are corrupted (see details in 
[HIVE-4388|https://issues.apache.org/jira/browse/HIVE-4388?focusedCommentId=13743075#comment-13743075])
 I'm currently using version 0.95.3-SNAPSHOT.

I've also had to change the default value of {{minTimestamp}} from 
{{Long.MIN_VALUE}} to zero as negative values in timestamp are no longer 
permitted (per HBASE-8462).


> Make pig working with HBase 0.95
> 
>
> Key: PIG-3390
> URL: https://issues.apache.org/jira/browse/PIG-3390
> Project: Pig
>  Issue Type: New Feature
>    Reporter: Jarek Jarcec Cecho
>Assignee: Jarek Jarcec Cecho
> Attachments: PIG-3390.patch, PIG-3390.patch, PIG-3390.patch
>
>
> The HBase 0.95 changed API in incompatible way. Following APIs that 
> {{HBaseStorage}} in Pig uses are no longer available:
> * {{Mutation.setWriteToWAL(Boolean)}}
> * {{Scan.write(DataOutput)}}
> Also in addition the HBase is no longer available as one monolithic archive 
> with entire functionality, but was broken down into smaller pieces such as 
> {{hbase-client}}, {{hbase-server}}, ...

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (PIG-3390) Make pig working with HBase 0.95

[Jarek Jarcec Cecho (JIRA)]

[ 
https://issues.apache.org/jira/browse/PIG-3390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13758900#comment-13758900
 ] 

Jarek Jarcec Cecho commented on PIG-3390:
-

Steps to test the latest patch:

{code}
# HBase 0.94, Hadoop 1
ant clean test -Dtestcase=TestHBaseStorage -Dhbaseversion=94

# HBase 0.95, Hadoop 1
ant clean test -Dtestcase=TestHBaseStorage -Dhbaseversion=95 
-Dprotobuf-java.version=2.5.0
{code}

> Make pig working with HBase 0.95
> 
>
> Key: PIG-3390
> URL: https://issues.apache.org/jira/browse/PIG-3390
> Project: Pig
>  Issue Type: New Feature
>Affects Versions: 0.11
>        Reporter: Jarek Jarcec Cecho
>    Assignee: Jarek Jarcec Cecho
> Fix For: 0.12
>
> Attachments: PIG-3390.patch, PIG-3390.patch, PIG-3390.patch
>
>
> The HBase 0.95 changed API in incompatible way. Following APIs that 
> {{HBaseStorage}} in Pig uses are no longer available:
> * {{Mutation.setWriteToWAL(Boolean)}}
> * {{Scan.write(DataOutput)}}
> Also in addition the HBase is no longer available as one monolithic archive 
> with entire functionality, but was broken down into smaller pieces such as 
> {{hbase-client}}, {{hbase-server}}, ...

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (PIG-3390) Make pig working with HBase 0.95

[Jarek Jarcec Cecho (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PIG-3390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jarek Jarcec Cecho updated PIG-3390:


Fix Version/s: 0.12
Affects Version/s: 0.11
   Status: Patch Available  (was: Open)

> Make pig working with HBase 0.95
> 
>
> Key: PIG-3390
> URL: https://issues.apache.org/jira/browse/PIG-3390
> Project: Pig
>  Issue Type: New Feature
>Affects Versions: 0.11
>    Reporter: Jarek Jarcec Cecho
>    Assignee: Jarek Jarcec Cecho
> Fix For: 0.12
>
> Attachments: PIG-3390.patch, PIG-3390.patch, PIG-3390.patch
>
>
> The HBase 0.95 changed API in incompatible way. Following APIs that 
> {{HBaseStorage}} in Pig uses are no longer available:
> * {{Mutation.setWriteToWAL(Boolean)}}
> * {{Scan.write(DataOutput)}}
> Also in addition the HBase is no longer available as one monolithic archive 
> with entire functionality, but was broken down into smaller pieces such as 
> {{hbase-client}}, {{hbase-server}}, ...

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Re: Are we ready for Pig 0.12.0 release?

[Jarek Jarcec Cecho]

I'll try to clean up and finish PIG-3390 (HBase 0.95 support) this week, to see 
if it can be included.

Jarcec

On Tue, Sep 03, 2013 at 10:56:42AM -0700, Cheolsoo Park wrote:
> +1. I will go through my jiras this week.
> 
> 
> On Tue, Sep 3, 2013 at 10:34 AM, Daniel Dai  wrote:
> 
> > Hi, All,
> > It has been more than half a year since initial Pig 0.11 release. I'd like
> > roll a Pig 0.12 release around the end of September or the beginning of
> > October. Let me know if it is possible.
> >
> > Proposed schedule:
> > 1. Commit all major features (1-2 weeks)
> > 2. Branching Pig 0.12
> > 3. Commit remaining patches (1-2 weeks)
> > 4. Wrapping up, document (1 week)
> >
> > If you have patches want to get in, please make sure the Jira ticket has
> > fix version set to 0.12. If the patches originally set to 0.12 and you
> > think you can delay, please mark the fix version to either 0.13.0 or
> > 0.12.1.
> >
> > Thanks,
> > Daniel
> >
> > --
> > CONFIDENTIALITY NOTICE
> > NOTICE: This message is intended for the use of the individual or entity to
> > which it is addressed and may contain information that is confidential,
> > privileged and exempt from disclosure under applicable law. If the reader
> > of this message is not the intended recipient, you are hereby notified that
> > any printing, copying, dissemination, distribution, disclosure or
> > forwarding of this communication is strictly prohibited. If you have
> > received this communication in error, please contact the sender immediately
> > and delete it from your system. Thank You.
> >


signature.asc
Description: Digital signature

[jira] [Commented] (PIG-3390) Make pig working with HBase 0.95

[Jarek Jarcec Cecho (JIRA)]

[ 
https://issues.apache.org/jira/browse/PIG-3390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13724634#comment-13724634
 ] 

Jarek Jarcec Cecho commented on PIG-3390:
-

It seems that HBase team might be open to return the 
{{Mutation.setWriteToWAL(Boolean)}} API back in HBASE-9093, so I'll refactore 
the patch if that gets committed.

> Make pig working with HBase 0.95
> 
>
> Key: PIG-3390
> URL: https://issues.apache.org/jira/browse/PIG-3390
> Project: Pig
>  Issue Type: New Feature
>    Reporter: Jarek Jarcec Cecho
>Assignee: Jarek Jarcec Cecho
> Attachments: PIG-3390.patch, PIG-3390.patch
>
>
> The HBase 0.95 changed API in incompatible way. Following APIs that 
> {{HBaseStorage}} in Pig uses are no longer available:
> * {{Mutation.setWriteToWAL(Boolean)}}
> * {{Scan.write(DataOutput)}}
> Also in addition the HBase is no longer available as one monolithic archive 
> with entire functionality, but was broken down into smaller pieces such as 
> {{hbase-client}}, {{hbase-server}}, ...

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (PIG-3390) Make pig working with HBase 0.95

[Jarek Jarcec Cecho (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PIG-3390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jarek Jarcec Cecho updated PIG-3390:


Attachment: PIG-3390.patch

Attaching updated patch incorporating feedback from [~cheolsoo]. To run the 
tests:

{code}
ant clean test -Dtestcase=TestHBaseStorage -Dhbaseversion=95
ant clean test -Dtestcase=TestHBaseStorage -Dhbaseversion=94 # default
{code}

> Make pig working with HBase 0.95
> 
>
> Key: PIG-3390
> URL: https://issues.apache.org/jira/browse/PIG-3390
> Project: Pig
>  Issue Type: New Feature
>        Reporter: Jarek Jarcec Cecho
>    Assignee: Jarek Jarcec Cecho
> Attachments: PIG-3390.patch, PIG-3390.patch
>
>
> The HBase 0.95 changed API in incompatible way. Following APIs that 
> {{HBaseStorage}} in Pig uses are no longer available:
> * {{Mutation.setWriteToWAL(Boolean)}}
> * {{Scan.write(DataOutput)}}
> Also in addition the HBase is no longer available as one monolithic archive 
> with entire functionality, but was broken down into smaller pieces such as 
> {{hbase-client}}, {{hbase-server}}, ...

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (PIG-3390) Make pig working with HBase 0.95

[Jarek Jarcec Cecho (JIRA)]

[ 
https://issues.apache.org/jira/browse/PIG-3390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13722536#comment-13722536
 ] 

Jarek Jarcec Cecho commented on PIG-3390:
-

Thank you for taking a look [~cheolsoo]!

bq. 1. I think you made mistakes in ivy.xml. It doesn't compile.

It seems that I've corrupted the patch during clean up, will fix that. Please 
accept my apologies for the inconvenience. 

bq. 2. Can you change "hbase.version" to "hbase94.version" in 
libraries.properties?

Of course, will do.

bq. 3. Do you mind explaining what these jars are for? Are they needed?

The HBase pom file seems to be corrupted as is not downloading transitive 
dependencies (covered by HBASE-8488). I had to enumerate all dependencies 
myself to overcome that. The HTrace is one of such dependencies.

bq. 4. Let's not add support for hadoop2 yet. We can revisit when hbase jars 
are available. TestHBaseStorage is excluded in hadoop2 unit tests anyway.

Understood, will clean up the patch then.

bq. 5. Can you remove unused imports in TestHbaseStorage.java and 
HbaseStorage.java?

Will do!

bq. 6. Can you use 4-space indentation instead of 2-space?

Will do!

Jarcec

> Make pig working with HBase 0.95
> 
>
> Key: PIG-3390
> URL: https://issues.apache.org/jira/browse/PIG-3390
> Project: Pig
>      Issue Type: New Feature
>    Reporter: Jarek Jarcec Cecho
>Assignee: Jarek Jarcec Cecho
> Attachments: PIG-3390.patch
>
>
> The HBase 0.95 changed API in incompatible way. Following APIs that 
> {{HBaseStorage}} in Pig uses are no longer available:
> * {{Mutation.setWriteToWAL(Boolean)}}
> * {{Scan.write(DataOutput)}}
> Also in addition the HBase is no longer available as one monolithic archive 
> with entire functionality, but was broken down into smaller pieces such as 
> {{hbase-client}}, {{hbase-server}}, ...

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (PIG-3390) Make pig working with HBase 0.95

[Jarek Jarcec Cecho (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PIG-3390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jarek Jarcec Cecho updated PIG-3390:


Attachment: PIG-3390.patch

I'm attaching preliminary patch that get basic support for HBase 0.95 working. 
I'm not changing status to {{Patch available}} as the patch is not yet ready 
for commit. Nevertheless, I would appreciate any feedback.

I've tweaked the {{ivy}} to include two HBase profiles, one for HBase 0.94- and 
second for 0.95+. It seems transitive dependencies of 0.95 are not currently 
resolved properly, so I had to temporarily specify all of them manually (seems 
to be tracked by HBASE-8488).

For the missing APis:

* {{Scan.write(DataOutput)}} It seems that we used this to manually serialize 
the {{Scan}} into Mapreduce job. I've used {{TableInputFormat}} to that for us. 
This way seems to be working for both 0.94- and 0.95+.
* {{Mutation.setWriteToWAL(Boolean}} was superseded by 
{{Mutation.setDurability(Durability)}}. Unfortunately I did not find clean way 
how to overcome this API change. Current patch uses reflection to detect the 
HBase version and to call the proper API.

To test it out you can use following commands:

{code}
ant clean test -Dtestcase=TestHBaseStorage -Dhbaseversion=95
ant clean test -Dtestcase=TestHBaseStorage -Dhbaseversion=94 # default
{code}

Using Hadoop 2 won't currently work as the HBase artifacts for Hadoop 2 are not 
published, but the in the future it should work the following way:

{code}
ant clean test -Dtestcase=TestHBaseStorage -Dhbaseversion=95 -Dhadoopversion=23 
-Dhbasecompat=2
ant clean test -Dtestcase=TestHBaseStorage -Dhbaseversion=94 -Dhadoopversion=23 
-Dhbasecompat=2
{code}

I'll be more than happy to hear any feedback on my approach!

> Make pig working with HBase 0.95
> 
>
> Key: PIG-3390
> URL: https://issues.apache.org/jira/browse/PIG-3390
> Project: Pig
>  Issue Type: New Feature
>    Reporter: Jarek Jarcec Cecho
>Assignee: Jarek Jarcec Cecho
> Attachments: PIG-3390.patch
>
>
> The HBase 0.95 changed API in incompatible way. Following APIs that 
> {{HBaseStorage}} in Pig uses are no longer available:
> * {{Mutation.setWriteToWAL(Boolean)}}
> * {{Scan.write(DataOutput)}}
> Also in addition the HBase is no longer available as one monolithic archive 
> with entire functionality, but was broken down into smaller pieces such as 
> {{hbase-client}}, {{hbase-server}}, ...

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (PIG-3390) Make pig working with HBase 0.95

[Jarek Jarcec Cecho (JIRA)]

Jarek Jarcec Cecho created PIG-3390:
---

 Summary: Make pig working with HBase 0.95
 Key: PIG-3390
 URL: https://issues.apache.org/jira/browse/PIG-3390
 Project: Pig
  Issue Type: New Feature
Reporter: Jarek Jarcec Cecho
Assignee: Jarek Jarcec Cecho


The HBase 0.95 changed API in incompatible way. Following APIs that 
{{HBaseStorage}} in Pig uses are no longer available:

* {{Mutation.setWriteToWAL(Boolean)}}
* {{Scan.write(DataOutput)}}

Also in addition the HBase is no longer available as one monolithic archive 
with entire functionality, but was broken down into smaller pieces such as 
{{hbase-client}}, {{hbase-server}}, ...

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Re: Uploading patches for review

[Jarek Jarcec Cecho]

I can definitely recommend using the review board. We are using it already in 
all other projects that I'm participating and it's really helping us out.

Jarcec

On Wed, Jun 05, 2013 at 06:21:32PM -0700, Rohini Palaniswamy wrote:
> Hi,
> Reviewing uploaded patches for few lines of change is easy. But when
> the change is more it is hard to read, review is more time consuming and at
> times you have to switch between the patch and eclipse to get more context.
> Without the surrounding code it is also easy to miss things on review. Can
> we make it a practice and decide on putting up the patches in review board
> for review if it is slightly bigger? Commenting on the patch is also a
> breeze in the review board.
> 
> Thoughts ???
> 
> Regards,
> Rohini


signature.asc
Description: Digital signature

[jira] [Commented] (PIG-2591) Unit tests should not write to /tmp but respect java.io.tmpdir

[Jarek Jarcec Cecho (JIRA)]

[ 
https://issues.apache.org/jira/browse/PIG-2591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13598563#comment-13598563
 ] 

Jarek Jarcec Cecho commented on PIG-2591:
-

Hi [~cheolsoo] and [~prkommireddi],
thank you very much for your feedback. I agree with proposed steps and I'll be 
more than happy to execute them myself. I just have small comment, direct usage 
of {{FileLocalizer}} is not simple as it requires a lot of initialization (for 
example that class needs valid {{PigContext}}). Thus I would propose to move 
the actual logic of getting temporary directory into standalone easily reusable 
class {{TmpUtil}} (which is part of {{bugPIG-2591.patch}}).

Jarcec

> Unit tests should not write to /tmp but respect java.io.tmpdir
> --
>
> Key: PIG-2591
> URL: https://issues.apache.org/jira/browse/PIG-2591
> Project: Pig
>  Issue Type: Bug
>  Components: tools
>Reporter: Thomas Weise
>    Assignee: Jarek Jarcec Cecho
> Fix For: 0.12
>
> Attachments: bugPIG-2591.patch, PIG-2495.patch
>
>
> Several tests use /tmp but should derive temporary file location from 
> java.io.tmpdir to avoid side effects (java.io.tmpdir is already set to a test 
> run specific location in build.xml)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Re: Welcome our new PMC chair, Julien Le Dem

[Jarek Jarcec Cecho]

Congratulations sir!

Jarcec

On Sun, Mar 10, 2013 at 08:55:55PM -0700, Aniket Mokashi wrote:
> Congrats Julien!
> 
> 
> On Sun, Mar 10, 2013 at 8:54 PM, Russell Jurney 
> wrote:
> 
> > Congrats!
> >
> > Russell Jurney http://datasyndrome.com
> >
> > On Mar 10, 2013, at 8:53 PM, Daniel Dai  wrote:
> >
> > > It is a bit late, Apache board approved the nomination of Julien Le
> > > Dem as our Pig PMC Chair last month. Welcome Julien!
> > >
> > > Thanks,
> > > Daniel
> >
> 
> 
> 
> -- 
> "...:::Aniket:::... Quetzalco@tl"


signature.asc
Description: Digital signature

[jira] [Commented] (PIG-3002) Pig client should handle CountersExceededException

[Jarek Jarcec Cecho (JIRA)]

[ 
https://issues.apache.org/jira/browse/PIG-3002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13589842#comment-13589842
 ] 

Jarek Jarcec Cecho commented on PIG-3002:
-

Understood. Thank you for your time [~billgraham], I appreciate your help and 
support.

I'm +1 on the patch "PIG-3002.2.patch" (non-binding).

Jarcec

> Pig client should handle CountersExceededException
> --
>
> Key: PIG-3002
> URL: https://issues.apache.org/jira/browse/PIG-3002
> Project: Pig
>  Issue Type: Bug
>Reporter: Bill Graham
>Assignee: Jarek Jarcec Cecho
>  Labels: newbie, simple
> Attachments: PIG-3002.2.patch, PIG-3002.patch
>
>
> Running a pig job that uses more than 120 counters will succeed, but a grunt 
> exception will occur when trying to output counter info to the console. This 
> exception should be caught and handled with friendly messaging:
> {noformat}
> org.apache.pig.backend.executionengine.ExecException: ERROR 2043: Unexpected 
> error during execution.
> at org.apache.pig.PigServer.launchPlan(PigServer.java:1275)
> at 
> org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1249)
> at org.apache.pig.PigServer.execute(PigServer.java:1239)
> at org.apache.pig.PigServer.executeBatch(PigServer.java:333)
> at 
> org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:136)
> at 
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:197)
> at 
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:169)
> at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:84)
> at org.apache.pig.Main.run(Main.java:604)
> at org.apache.pig.Main.main(Main.java:154)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:186)
> Caused by: org.apache.hadoop.mapred.Counters$CountersExceededException: 
> Error: Exceeded limits on number of counters - Counters=120 Limit=120
> at 
> org.apache.hadoop.mapred.Counters$Group.getCounterForName(Counters.java:312)
> at org.apache.hadoop.mapred.Counters.findCounter(Counters.java:431)
> at org.apache.hadoop.mapred.Counters.getCounter(Counters.java:495)
> at 
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.computeWarningAggregate(MapReduceLauncher.java:707)
> at 
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:442)
> at org.apache.pig.PigServer.launchPlan(PigServer.java:1264)
> {noformat}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (PIG-2591) Unit tests should not write to /tmp but respect java.io.tmpdir

[Jarek Jarcec Cecho (JIRA)]

[ 
https://issues.apache.org/jira/browse/PIG-2591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13589126#comment-13589126
 ] 

Jarek Jarcec Cecho commented on PIG-2591:
-

[~cheolsoo], thank you very much for your review, I appreciate your time. There 
is a lot of tests that needs to be fixed and they seems to be changing, so I 
would prefer to fix all of them incrementally rather than all at once. Do you 
think that it would be feasible to get this one committed and create subtask 
for each additional chunk of fixed test cases?

> Unit tests should not write to /tmp but respect java.io.tmpdir
> --
>
> Key: PIG-2591
> URL: https://issues.apache.org/jira/browse/PIG-2591
> Project: Pig
>  Issue Type: Bug
>  Components: tools
>Reporter: Thomas Weise
>    Assignee: Jarek Jarcec Cecho
> Fix For: 0.12
>
> Attachments: bugPIG-2591.patch, PIG-2495.patch
>
>
> Several tests use /tmp but should derive temporary file location from 
> java.io.tmpdir to avoid side effects (java.io.tmpdir is already set to a test 
> run specific location in build.xml)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (PIG-3002) Pig client should handle CountersExceededException

[Jarek Jarcec Cecho (JIRA)]

[ 
https://issues.apache.org/jira/browse/PIG-3002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13587246#comment-13587246
 ] 

Jarek Jarcec Cecho commented on PIG-3002:
-

Hi [~billgraham], 
thank you very much for taking a look on this Jira and my patch. I was 
considering similar solution as you proposed in my early work, but I've notice 
one side effect during experiments with my early patches.

I've created quite pathological case when my cluster was using default 
configuration, but I've limit the number of allowed counters to 3 on machine 
where I've executed pig. I've noticed that with similar fix, pig will print out 
couple of counters and than bail out on exception on first non existing 
Counter. As a result not all the counters will be printed out even though they 
are available in the {{Couter}} object.

My experiment is obviously not entirely real as it's unlikely that users will 
have different hadoop configuration. However I believe that it model the edge 
situation when mapreduce job will create almost all available counters, but 
because client is iterating over predefined set, not all of them will be 
printed out.

I've also did one step further and put the {{try-catch}} block inside the 
{{for}} iteration. I've noticed that in this situation we might print out the 
error message several times, which is kind of distracting. This lead me to the 
idea of doing the changes on the shim layer that I've submitted.

Jarcec

> Pig client should handle CountersExceededException
> --
>
> Key: PIG-3002
> URL: https://issues.apache.org/jira/browse/PIG-3002
> Project: Pig
>  Issue Type: Bug
>    Reporter: Bill Graham
>Assignee: Jarek Jarcec Cecho
>  Labels: newbie, simple
> Attachments: PIG-3002.2.patch, PIG-3002.patch
>
>
> Running a pig job that uses more than 120 counters will succeed, but a grunt 
> exception will occur when trying to output counter info to the console. This 
> exception should be caught and handled with friendly messaging:
> {noformat}
> org.apache.pig.backend.executionengine.ExecException: ERROR 2043: Unexpected 
> error during execution.
> at org.apache.pig.PigServer.launchPlan(PigServer.java:1275)
> at 
> org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1249)
> at org.apache.pig.PigServer.execute(PigServer.java:1239)
> at org.apache.pig.PigServer.executeBatch(PigServer.java:333)
> at 
> org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:136)
> at 
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:197)
> at 
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:169)
> at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:84)
> at org.apache.pig.Main.run(Main.java:604)
> at org.apache.pig.Main.main(Main.java:154)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:186)
> Caused by: org.apache.hadoop.mapred.Counters$CountersExceededException: 
> Error: Exceeded limits on number of counters - Counters=120 Limit=120
> at 
> org.apache.hadoop.mapred.Counters$Group.getCounterForName(Counters.java:312)
> at org.apache.hadoop.mapred.Counters.findCounter(Counters.java:431)
> at org.apache.hadoop.mapred.Counters.getCounter(Counters.java:495)
> at 
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.computeWarningAggregate(MapReduceLauncher.java:707)
> at 
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:442)
> at org.apache.pig.PigServer.launchPlan(PigServer.java:1264)
> {noformat}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (PIG-3002) Pig client should handle CountersExceededException

[Jarek Jarcec Cecho (JIRA)]

[ 
https://issues.apache.org/jira/browse/PIG-3002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13585189#comment-13585189
 ] 

Jarek Jarcec Cecho commented on PIG-3002:
-

Hi [~billgraham],
I know that you are super busy with driving the pig release and all other 
things. I just wanted to check whether you had by any chance time to take a 
look into my proposed patch?

Jarcec

> Pig client should handle CountersExceededException
> --
>
> Key: PIG-3002
> URL: https://issues.apache.org/jira/browse/PIG-3002
> Project: Pig
>  Issue Type: Bug
>Reporter: Bill Graham
>        Assignee: Jarek Jarcec Cecho
>  Labels: newbie, simple
> Attachments: PIG-3002.patch
>
>
> Running a pig job that uses more than 120 counters will succeed, but a grunt 
> exception will occur when trying to output counter info to the console. This 
> exception should be caught and handled with friendly messaging:
> {noformat}
> org.apache.pig.backend.executionengine.ExecException: ERROR 2043: Unexpected 
> error during execution.
> at org.apache.pig.PigServer.launchPlan(PigServer.java:1275)
> at 
> org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1249)
> at org.apache.pig.PigServer.execute(PigServer.java:1239)
> at org.apache.pig.PigServer.executeBatch(PigServer.java:333)
> at 
> org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:136)
> at 
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:197)
> at 
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:169)
> at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:84)
> at org.apache.pig.Main.run(Main.java:604)
> at org.apache.pig.Main.main(Main.java:154)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:186)
> Caused by: org.apache.hadoop.mapred.Counters$CountersExceededException: 
> Error: Exceeded limits on number of counters - Counters=120 Limit=120
> at 
> org.apache.hadoop.mapred.Counters$Group.getCounterForName(Counters.java:312)
> at org.apache.hadoop.mapred.Counters.findCounter(Counters.java:431)
> at org.apache.hadoop.mapred.Counters.getCounter(Counters.java:495)
> at 
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.computeWarningAggregate(MapReduceLauncher.java:707)
> at 
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:442)
> at org.apache.pig.PigServer.launchPlan(PigServer.java:1264)
> {noformat}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira