[jira] [Work started] (PIG-2934) HBaseStorage filter optimizations

2012-11-09 Thread Bill Graham (JIRA) 

 [ 
https://issues.apache.org/jira/browse/PIG-2934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on PIG-2934 started by Bill Graham.

> HBaseStorage filter optimizations
> -
>
> Key: PIG-2934
> URL: https://issues.apache.org/jira/browse/PIG-2934
> Project: Pig
>  Issue Type: Improvement
>Reporter: Bill Graham
>Assignee: Bill Graham
>  Labels: hbase
>
> Our HBase pal/guru Gary Helmling was kind enough to do a code review of 
> HBaseStorage. He suggested some good filter optimizations:
> * when using the "lt*" and "gt*" options, set the start/stop rows on the Scan 
> instance, at least in addition to the RowFilters. Without this you're doing a 
> full table scan, regardless of the RowFilters.
> * when selecting specific columns or entire families to return, it would be 
> more efficient to set the family + columns on the Scan object (addFamily(), 
> addColumn()), instead of using a FilterList. I'm not familiar with the 
> family:prefix handling you mention, but that would still seem to require 
> filters. But if that's not being used, it would be better to avoid the 
> FilterList for columns. At minimum, we should probably call Scan.addFamily() 
> with the distinct families, so we can skip entire column families that are 
> not being used. In the case of a table with 4 CFs, if, say, only 1 is being 
> used, this could be a big gain.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] Subscription: PIG patch available

2012-11-09 Thread jira 
Issue Subscription
Filter: PIG patch available (30 issues)

Subscriber: pigdaily

Key Summary
PIG-3039Not possible to use custom version of jackson jars
https://issues.apache.org/jira/browse/PIG-3039
PIG-3034Remove Penny code from Pig repository
https://issues.apache.org/jira/browse/PIG-3034
PIG-3029TestTypeCheckingValidatorNewLP has some path reference issues for 
cross-platform execution
https://issues.apache.org/jira/browse/PIG-3029
PIG-3028testGrunt dev test needs some command filters to run correctly 
without cygwin
https://issues.apache.org/jira/browse/PIG-3028
PIG-3027pigTest unit test needs a newline filter for comparisons of golden 
multi-line
https://issues.apache.org/jira/browse/PIG-3027
PIG-3026Pig checked-in baseline comparisons need a pre-filter to address 
OS-specific newline differences
https://issues.apache.org/jira/browse/PIG-3026
PIG-3025TestPruneColumn unit test - SimpleEchoStreamingCommand perl inline 
script needs simplification
https://issues.apache.org/jira/browse/PIG-3025
PIG-3024TestEmptyInputDir unit test - hadoop version detection logic is 
brittle
https://issues.apache.org/jira/browse/PIG-3024
PIG-3014CurrentTime() UDF has undesirable characteristics
https://issues.apache.org/jira/browse/PIG-3014
PIG-3010Allow UDF's to flatten themselves
https://issues.apache.org/jira/browse/PIG-3010
PIG-2978TestLoadStoreFuncLifeCycle fails with hadoop-2.0.x
https://issues.apache.org/jira/browse/PIG-2978
PIG-2959Add a pig.cmd for Pig to run under Windows
https://issues.apache.org/jira/browse/PIG-2959
PIG-2957TetsScriptUDF fail due to volume prefix in jar
https://issues.apache.org/jira/browse/PIG-2957
PIG-2956Invalid cache specification for some streaming statement
https://issues.apache.org/jira/browse/PIG-2956
PIG-2955 Fix bunch of Pig e2e tests on Windows 
https://issues.apache.org/jira/browse/PIG-2955
PIG-2937generated field in nested foreach does not inherit the variable 
name as the field name
https://issues.apache.org/jira/browse/PIG-2937
PIG-2924PigStats should not be assuming all Storage classes to be 
file-based storage
https://issues.apache.org/jira/browse/PIG-2924
PIG-2873Converting bin/pig shell script to python
https://issues.apache.org/jira/browse/PIG-2873
PIG-2834MultiStorage requires unused constructor argument
https://issues.apache.org/jira/browse/PIG-2834
PIG-2824Pushing checking number of fields into LoadFunc
https://issues.apache.org/jira/browse/PIG-2824
PIG-2661Pig uses an extra job for loading data in Pigmix L9
https://issues.apache.org/jira/browse/PIG-2661
PIG-2657Print warning if using wrong jython version
https://issues.apache.org/jira/browse/PIG-2657
PIG-2507Semicolon in paramenters for UDF results in parsing error
https://issues.apache.org/jira/browse/PIG-2507
PIG-2433Jython import module not working if module path is in classpath
https://issues.apache.org/jira/browse/PIG-2433
PIG-2417Streaming UDFs -  allow users to easily write UDFs in scripting 
languages with no JVM implementation.
https://issues.apache.org/jira/browse/PIG-2417
PIG-2362Rework Ant build.xml to use macrodef instead of antcall
https://issues.apache.org/jira/browse/PIG-2362
PIG-2312NPE when relation and column share the same name and used in Nested 
Foreach 
https://issues.apache.org/jira/browse/PIG-2312
PIG-1942script UDF (jython) should utilize the intended output schema to 
more directly convert Py objects to Pig objects
https://issues.apache.org/jira/browse/PIG-1942
PIG-1431Current DateTime UDFs: ISONOW(), UNIXNOW()
https://issues.apache.org/jira/browse/PIG-1431
PIG-1237Piggybank MutliStorage - specify field to write in output
https://issues.apache.org/jira/browse/PIG-1237

You may edit this subscription at:
https://issues.apache.org/jira/secure/FilterSubscription!default.jspa?subId=13225&filterId=12322384

[jira] [Resolved] (PIG-2325) Make e2e test directory for data configurable in HDFS

2012-11-09 Thread Gianmarco De Francisci Morales (JIRA) 

 [ 
https://issues.apache.org/jira/browse/PIG-2325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gianmarco De Francisci Morales resolved PIG-2325.
-

   Resolution: Invalid
Fix Version/s: 0.12
 Assignee: Gianmarco De Francisci Morales

Thanks!

> Make e2e test directory for data configurable in HDFS
> -
>
> Key: PIG-2325
> URL: https://issues.apache.org/jira/browse/PIG-2325
> Project: Pig
>  Issue Type: Improvement
>Reporter: Gianmarco De Francisci Morales
>Assignee: Gianmarco De Francisci Morales
>Priority: Minor
> Fix For: 0.12
>
>
> Right now the place for the data generated in e2e tests is hardcoded in 
> test/e2e/pig/conf/default.conf as:
> {code}
>  $cfg = {
>  #HDFS
>   'inpathbase' => '/user/pig/tests/data'
> , 'outpathbase'=> '/user/pig/out'
> {code}
> It would be better to make it configurable (with an environment variable?) as 
> the rest of the paths.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (PIG-2325) Make e2e test directory for data configurable in HDFS

2012-11-09 Thread Johnny Zhang (JIRA) 

[ 
https://issues.apache.org/jira/browse/PIG-2325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13494454#comment-13494454
 ] 

Johnny Zhang commented on PIG-2325:
---

[~azaroth], I think this issue can be closed, it is already been fixed in trunk
in test/e2e/pig/conf/default.conf
{noformat}
my $hdfsBase = $ENV{PH_HDFS_BASE} || "/user/pig";

$cfg = {
#HDFS
  'inpathbase' => "$hdfsBase/tests/data"
, 'outpathbase'=> "$hdfsBase/out"
{noformat}

> Make e2e test directory for data configurable in HDFS
> -
>
> Key: PIG-2325
> URL: https://issues.apache.org/jira/browse/PIG-2325
> Project: Pig
>  Issue Type: Improvement
>Reporter: Gianmarco De Francisci Morales
>Priority: Minor
>
> Right now the place for the data generated in e2e tests is hardcoded in 
> test/e2e/pig/conf/default.conf as:
> {code}
>  $cfg = {
>  #HDFS
>   'inpathbase' => '/user/pig/tests/data'
> , 'outpathbase'=> '/user/pig/out'
> {code}
> It would be better to make it configurable (with an environment variable?) as 
> the rest of the paths.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (PIG-3044) Trigger POPartialAgg compaction under GC pressure

2012-11-09 Thread Dmitriy V. Ryaboy (JIRA) 

[ 
https://issues.apache.org/jira/browse/PIG-3044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13494382#comment-13494382
 ] 

Dmitriy V. Ryaboy commented on PIG-3044:


Since this is preventing some jobs from migrating from Pig 9 to Pig 11 in our 
environment, I would like to add it to 11, but an argument can be made that 
this is not a critical bug and should therefore not be applied to the 0.11 
branch. Committers, please weigh in if you have an opinion on this.

> Trigger POPartialAgg compaction under GC pressure
> -
>
> Key: PIG-3044
> URL: https://issues.apache.org/jira/browse/PIG-3044
> Project: Pig
>  Issue Type: Improvement
>Affects Versions: 0.10.0, 0.11, 0.10.1
>Reporter: Dmitriy V. Ryaboy
>Assignee: Dmitriy V. Ryaboy
> Fix For: 0.12
>
>
> If partial aggregation is turned on in pig 10 and 11, 20% (by default) of the 
> available heap can be consumed by the POPartialAgg operator. This can cause 
> memory issues for jobs that use all, or nearly all, of the heap already.
> If we make POPartialAgg "spillable" (trigger compaction when memory reduction 
> is required), we would be much nicer to high-memory jobs.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (PIG-3044) Trigger POPartialAgg compaction under GC pressure

2012-11-09 Thread Dmitriy V. Ryaboy (JIRA) 
Dmitriy V. Ryaboy created PIG-3044:
--

 Summary: Trigger POPartialAgg compaction under GC pressure
 Key: PIG-3044
 URL: https://issues.apache.org/jira/browse/PIG-3044
 Project: Pig
  Issue Type: Improvement
Affects Versions: 0.10.0, 0.11, 0.10.1
Reporter: Dmitriy V. Ryaboy
Assignee: Dmitriy V. Ryaboy
 Fix For: 0.12


If partial aggregation is turned on in pig 10 and 11, 20% (by default) of the 
available heap can be consumed by the POPartialAgg operator. This can cause 
memory issues for jobs that use all, or nearly all, of the heap already.

If we make POPartialAgg "spillable" (trigger compaction when memory reduction 
is required), we would be much nicer to high-memory jobs.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (PIG-3039) Not possible to use custom version of jackson jars

2012-11-09 Thread Julien Le Dem (JIRA) 

[ 
https://issues.apache.org/jira/browse/PIG-3039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13494325#comment-13494325
 ] 

Julien Le Dem commented on PIG-3039:


I see. Thanks!

> Not possible to use custom version of jackson jars
> --
>
> Key: PIG-3039
> URL: https://issues.apache.org/jira/browse/PIG-3039
> Project: Pig
>  Issue Type: Bug
>Affects Versions: 0.10.0
>Reporter: Rohini Palaniswamy
>Assignee: Rohini Palaniswamy
> Fix For: 0.12
>
> Attachments: PIG-3039-trunk.patch
>
>
> User is trying
> register jackson_core_asl-1.9.4_1.jar;
> register jackson_mapper_asl-1.9.4_1.jar;
> register jackson_xc-1.9.4_1.jar;
> But pig.jar/pig-withouthadoop.jar has jackson jars and JarManager packages 
> the jackson from pig.jar into job.jar(PIG-2457). We could not find any 
> possible workaround with mapreduce framework to put the user jar first in the 
> classpath as job.jar always takes precedence.
>  The pig script works fine with 0.9 and is a regression in 0.10.  

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (PIG-3039) Not possible to use custom version of jackson jars

2012-11-09 Thread Rohini Palaniswamy (JIRA) 

[ 
https://issues.apache.org/jira/browse/PIG-3039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13494320#comment-13494320
 ] 

Rohini Palaniswamy commented on PIG-3039:
-

You mean how it works in 0.9? It is simple. Before PIG-2457, jackson was not 
packaged as part of pig.jar. So users could register their version of jackson 
jar in pig with mapreduce.user.classpath.first=true to avoid hadoop's jackson 
jars being in classpath.  

> Not possible to use custom version of jackson jars
> --
>
> Key: PIG-3039
> URL: https://issues.apache.org/jira/browse/PIG-3039
> Project: Pig
>  Issue Type: Bug
>Affects Versions: 0.10.0
>Reporter: Rohini Palaniswamy
>Assignee: Rohini Palaniswamy
> Fix For: 0.12
>
> Attachments: PIG-3039-trunk.patch
>
>
> User is trying
> register jackson_core_asl-1.9.4_1.jar;
> register jackson_mapper_asl-1.9.4_1.jar;
> register jackson_xc-1.9.4_1.jar;
> But pig.jar/pig-withouthadoop.jar has jackson jars and JarManager packages 
> the jackson from pig.jar into job.jar(PIG-2457). We could not find any 
> possible workaround with mapreduce framework to put the user jar first in the 
> classpath as job.jar always takes precedence.
>  The pig script works fine with 0.9 and is a regression in 0.10.  

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (PIG-3039) Not possible to use custom version of jackson jars

2012-11-09 Thread Julien Le Dem (JIRA) 

[ 
https://issues.apache.org/jira/browse/PIG-3039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13494318#comment-13494318
 ] 

Julien Le Dem commented on PIG-3039:


Could you explain how it was working before ? Were the classes from the 
registered jar added to the same job jar and overwriting the pig.jar classes?


> Not possible to use custom version of jackson jars
> --
>
> Key: PIG-3039
> URL: https://issues.apache.org/jira/browse/PIG-3039
> Project: Pig
>  Issue Type: Bug
>Affects Versions: 0.10.0
>Reporter: Rohini Palaniswamy
>Assignee: Rohini Palaniswamy
> Fix For: 0.12
>
> Attachments: PIG-3039-trunk.patch
>
>
> User is trying
> register jackson_core_asl-1.9.4_1.jar;
> register jackson_mapper_asl-1.9.4_1.jar;
> register jackson_xc-1.9.4_1.jar;
> But pig.jar/pig-withouthadoop.jar has jackson jars and JarManager packages 
> the jackson from pig.jar into job.jar(PIG-2457). We could not find any 
> possible workaround with mapreduce framework to put the user jar first in the 
> classpath as job.jar always takes precedence.
>  The pig script works fine with 0.9 and is a regression in 0.10.  

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (PIG-3043) Modify the UrlClassloader in PigContext so that classes from the same classloader are used first instead of the parent

2012-11-09 Thread Rohini Palaniswamy (JIRA) 

[ 
https://issues.apache.org/jira/browse/PIG-3043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13494296#comment-13494296
 ] 

Rohini Palaniswamy commented on PIG-3043:
-

That's right. Thanks for taking a look Julien. My concern was whether you 
wanted to keep both versions of the jar in the backend too as my patch just 
ships one version to the backend in case of packages inside pig.jar.

> Modify the UrlClassloader in PigContext so that classes from the same 
> classloader are used first instead of the parent
> --
>
> Key: PIG-3043
> URL: https://issues.apache.org/jira/browse/PIG-3043
> Project: Pig
>  Issue Type: Improvement
>Reporter: Julien Le Dem
>
> This behavior would be similar to what application servers do (Tomcat, Jetty, 
> ...) and would allow classes from registered jars to use their own version of 
> a class. It also avoid problems when adding a jar to pig break libraries that 
> make use of dynamic class lookup.
> example of a common pattern that regularly is broken by the current mechanism:
> register lib.jar
> register my.jar
> define blah as my.UDF('my.Implementation')
> my.UDF is in my.jar and uses classes in lib.jar that use Class.forName() to 
> resolve my.Implementation. It works fine until lib.jar is added as a 
> dependency of pig or in the PIG_CLASSPATH. Then classes in lib.jar do not see 
> the classes in registered jars.
> I thinks that overriding loadClass(String name, boolean resolve) would allow 
> doing that.
> We should make an exception for anything in org.apache.pig just like 
> servlet.jar is excluded in app servers.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (PIG-2980) documentation for DateTime datatype

2012-11-09 Thread Thejas M Nair (JIRA) 

[ 
https://issues.apache.org/jira/browse/PIG-2980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13494274#comment-13494274
 ] 

Thejas M Nair commented on PIG-2980:


bq. Yes, I mean ToDate('1970-01-01T00:00:00.000+00:00'). where users can 
specify a constant string to create a datetime object. Let me rephrase the 
description here.
I think we can just remove datestamp from the constants table and add a note 
under the table, that users should use ToDate udf to generate datetime from 
string constants. 
 

> documentation for DateTime datatype
> ---
>
> Key: PIG-2980
> URL: https://issues.apache.org/jira/browse/PIG-2980
> Project: Pig
>  Issue Type: Bug
>  Components: documentation
>Reporter: Thejas M Nair
>Assignee: Zhijie Shen
> Fix For: 0.11
>
> Attachments: PIG-2980.patch
>
>
> Documentation for new DateTime type needs to be added.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (PIG-3043) Modify the UrlClassloader in PigContext so that classes from the same classloader are used first instead of the parent

2012-11-09 Thread Julien Le Dem (JIRA) 

[ 
https://issues.apache.org/jira/browse/PIG-3043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13494231#comment-13494231
 ] 

Julien Le Dem commented on PIG-3043:


Rohini, I just looked at your patch in PIG-3039 and I think it is compatible 
with what I am suggesting here. register jars take precedence in the frontend 
and the backend, even if the mechanism for this differs.

> Modify the UrlClassloader in PigContext so that classes from the same 
> classloader are used first instead of the parent
> --
>
> Key: PIG-3043
> URL: https://issues.apache.org/jira/browse/PIG-3043
> Project: Pig
>  Issue Type: Improvement
>Reporter: Julien Le Dem
>
> This behavior would be similar to what application servers do (Tomcat, Jetty, 
> ...) and would allow classes from registered jars to use their own version of 
> a class. It also avoid problems when adding a jar to pig break libraries that 
> make use of dynamic class lookup.
> example of a common pattern that regularly is broken by the current mechanism:
> register lib.jar
> register my.jar
> define blah as my.UDF('my.Implementation')
> my.UDF is in my.jar and uses classes in lib.jar that use Class.forName() to 
> resolve my.Implementation. It works fine until lib.jar is added as a 
> dependency of pig or in the PIG_CLASSPATH. Then classes in lib.jar do not see 
> the classes in registered jars.
> I thinks that overriding loadClass(String name, boolean resolve) would allow 
> doing that.
> We should make an exception for anything in org.apache.pig just like 
> servlet.jar is excluded in app servers.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (PIG-3043) Modify the UrlClassloader in PigContext so that classes from the same classloader are used first instead of the parent

2012-11-09 Thread Julien Le Dem (JIRA) 

[ 
https://issues.apache.org/jira/browse/PIG-3043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13494224#comment-13494224
 ] 

Julien Le Dem commented on PIG-3043:


I tried the Classloader in the backend in the past and ended up reverting it 
because of similar issues to the one described in this ticket. For example, a 
Loader (in a registered jar) not seeing files from the registered jar because 
the Configuration object uses getResourceAsStream on its own classloader. 
Having 2 classloaders where there was 1 has side effects.
see: https://issues.apache.org/jira/browse/PIG-2318

> Modify the UrlClassloader in PigContext so that classes from the same 
> classloader are used first instead of the parent
> --
>
> Key: PIG-3043
> URL: https://issues.apache.org/jira/browse/PIG-3043
> Project: Pig
>  Issue Type: Improvement
>Reporter: Julien Le Dem
>
> This behavior would be similar to what application servers do (Tomcat, Jetty, 
> ...) and would allow classes from registered jars to use their own version of 
> a class. It also avoid problems when adding a jar to pig break libraries that 
> make use of dynamic class lookup.
> example of a common pattern that regularly is broken by the current mechanism:
> register lib.jar
> register my.jar
> define blah as my.UDF('my.Implementation')
> my.UDF is in my.jar and uses classes in lib.jar that use Class.forName() to 
> resolve my.Implementation. It works fine until lib.jar is added as a 
> dependency of pig or in the PIG_CLASSPATH. Then classes in lib.jar do not see 
> the classes in registered jars.
> I thinks that overriding loadClass(String name, boolean resolve) would allow 
> doing that.
> We should make an exception for anything in org.apache.pig just like 
> servlet.jar is excluded in app servers.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (PIG-3043) Modify the UrlClassloader in PigContext so that classes from the same classloader are used first instead of the parent

2012-11-09 Thread Rohini Palaniswamy (JIRA) 

[ 
https://issues.apache.org/jira/browse/PIG-3043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13494038#comment-13494038
 ] 

Rohini Palaniswamy commented on PIG-3043:
-

Asked in case you are attempting to achieve something like that in the backend 
too. That did not seem to be the intention but wanted to confirm. In PIG-3039, 
I put up a patch which gives preference to the registered jars while shipping 
jars to the job. That would conflict in case you were trying something like 
that. 

Just FYI. It is possible to do class loaders at the backend too. Instead of 
archive you can ship the jar as a file without adding to class path and then 
use a URLClassloader to put them in classpath. Have tried that in the past to 
load a different version of Hadoop jar in h20. But anyways with the way 
classpath handling is done in h23(wildcard inclusion of jars) that approach 
will not work.

> Modify the UrlClassloader in PigContext so that classes from the same 
> classloader are used first instead of the parent
> --
>
> Key: PIG-3043
> URL: https://issues.apache.org/jira/browse/PIG-3043
> Project: Pig
>  Issue Type: Improvement
>Reporter: Julien Le Dem
>
> This behavior would be similar to what application servers do (Tomcat, Jetty, 
> ...) and would allow classes from registered jars to use their own version of 
> a class. It also avoid problems when adding a jar to pig break libraries that 
> make use of dynamic class lookup.
> example of a common pattern that regularly is broken by the current mechanism:
> register lib.jar
> register my.jar
> define blah as my.UDF('my.Implementation')
> my.UDF is in my.jar and uses classes in lib.jar that use Class.forName() to 
> resolve my.Implementation. It works fine until lib.jar is added as a 
> dependency of pig or in the PIG_CLASSPATH. Then classes in lib.jar do not see 
> the classes in registered jars.
> I thinks that overriding loadClass(String name, boolean resolve) would allow 
> doing that.
> We should make an exception for anything in org.apache.pig just like 
> servlet.jar is excluded in app servers.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Build failed in Jenkins: Pig-trunk #1358

2012-11-09 Thread Apache Jenkins Server 
See 

Changes:

[cheolsoo] PIG-2979: Pig.jar doesn't work with hadoop-2.0.x (cheolsoo)

--
[...truncated 38428 lines...]
[junit] at 
org.apache.hadoop.hdfs.MiniDFSCluster.shutdownDataNodes(MiniDFSCluster.java:566)
[junit] at 
org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:550)
[junit] at 
org.apache.pig.test.MiniGenericCluster.shutdownMiniDfsClusters(MiniGenericCluster.java:87)
[junit] at 
org.apache.pig.test.MiniGenericCluster.shutdownMiniDfsAndMrClusters(MiniGenericCluster.java:77)
[junit] at 
org.apache.pig.test.MiniGenericCluster.shutDown(MiniGenericCluster.java:68)
[junit] at 
org.apache.pig.test.TestStore.oneTimeTearDown(TestStore.java:138)
[junit] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[junit] at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
[junit] at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
[junit] at java.lang.reflect.Method.invoke(Method.java:597)
[junit] at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
[junit] at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
[junit] at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
[junit] at 
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:37)
[junit] at org.junit.runners.ParentRunner.run(ParentRunner.java:220)
[junit] at 
junit.framework.JUnit4TestAdapter.run(JUnit4TestAdapter.java:39)
[junit] at 
org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:420)
[junit] at 
org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:911)
[junit] at 
org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:768)
[junit] 12/11/09 10:34:16 WARN datanode.FSDatasetAsyncDiskService: 
AsyncDiskService has already shut down.
[junit] 12/11/09 10:34:16 INFO mortbay.log: Stopped 
SelectChannelConnector@localhost:0
[junit] 12/11/09 10:34:16 INFO ipc.Server: Stopping server on 33361
[junit] 12/11/09 10:34:16 INFO ipc.Server: IPC Server handler 1 on 33361: 
exiting
[junit] 12/11/09 10:34:16 INFO ipc.Server: IPC Server handler 2 on 33361: 
exiting
[junit] 12/11/09 10:34:16 INFO ipc.Server: Stopping IPC Server listener on 
33361
[junit] 12/11/09 10:34:16 INFO ipc.Server: Stopping IPC Server Responder
[junit] 12/11/09 10:34:16 INFO ipc.Server: IPC Server handler 0 on 33361: 
exiting
[junit] 12/11/09 10:34:16 INFO metrics.RpcInstrumentation: shut down
[junit] 12/11/09 10:34:16 INFO datanode.DataNode: Waiting for threadgroup 
to exit, active threads is 1
[junit] 12/11/09 10:34:16 WARN datanode.DataNode: 
DatanodeRegistration(127.0.0.1:42881, 
storageID=DS-754274827-67.195.138.20-42881-1352456759286, infoPort=41770, 
ipcPort=33361):DataXceiveServer:java.nio.channels.AsynchronousCloseException
[junit] at 
java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:185)
[junit] at 
sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:159)
[junit] at 
sun.nio.ch.ServerSocketAdaptor.accept(ServerSocketAdaptor.java:84)
[junit] at 
org.apache.hadoop.hdfs.server.datanode.DataXceiverServer.run(DataXceiverServer.java:131)
[junit] at java.lang.Thread.run(Thread.java:662)
[junit] 
[junit] 12/11/09 10:34:16 INFO datanode.DataNode: Exiting DataXceiveServer
[junit] 12/11/09 10:34:16 INFO datanode.DataNode: Scheduling block 
blk_4945455753790126000_1134 file 
build/test/data/dfs/data/data1/current/blk_4945455753790126000 for deletion
[junit] 12/11/09 10:34:16 INFO datanode.DataNode: Deleted block 
blk_4945455753790126000_1134 at file 
build/test/data/dfs/data/data1/current/blk_4945455753790126000
[junit] 12/11/09 10:34:17 INFO mapred.TaskTracker: Received 'KillJobAction' 
for job: job_20121109102559702_0012
[junit] 12/11/09 10:34:17 WARN mapred.TaskTracker: Unknown job 
job_20121109102559702_0012 being deleted.
[junit] 12/11/09 10:34:17 INFO mapred.TaskTracker: Received 'KillJobAction' 
for job: job_20121109102559702_0012
[junit] 12/11/09 10:34:17 WARN mapred.TaskTracker: Unknown job 
job_20121109102559702_0012 being deleted.
[junit] 12/11/09 10:34:17 INFO datanode.DataBlockScanner: Exiting 
DataBlockScanner thread.
[junit] 12/11/09 10:34:17 INFO datanode.DataNode: 
DatanodeRegistration(127.0.0.1:42881, 
storageID=DS-754274827-67.195.138.20-42881-1352456759286, infoPort=41770, 
ipcPort=33361):Finishing DataNode in: 
FSDataset{dirpath='