It took a deep debugging session to find the reason why my test failed.
After two hours of tracing step by step through the insides of Pig itself I
noticed it was going to the "cleanupOnFailure" method.
It turns out that the entire job failed in the unit test and the error was
hidden to me.
The error was:
Unexpected System Error Occured:
java.lang.IncompatibleClassChangeError: Found interface
org.apache.hadoop.mapreduce.JobContext, but class was expected
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat.setupUdfEnvAndStores(PigOutputFormat.java:243)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat.checkOutputSpecs(PigOutputFormat.java:190)
Sigh ... So it's the Hadoop 1.x vs 2.x problem again.
So why was the message hidden?
Turns out I forgot to put in a log4j.properties file and even though the
junit errors are put on the screen; this error wasn't.
The fix was easy:
Change the pig dependency in the pom.xml from this
<dependency>
<groupId>org.apache.pig</groupId>
<artifactId>pig</artifactId>
<version>0.13.0</version>
</dependency>
into this
<dependency>
<groupId>org.apache.pig</groupId>
<artifactId>pig</artifactId>
<version>0.13.0</version>
<classifier>h2</classifier>
</dependency>
Niels Basjes
On Sun, Nov 23, 2014 at 8:29 PM, Niels Basjes <[email protected]> wrote:
> I did some additional testing myself and I now have the "smallest
> possible" way to reproduce the effect I see.
>
> I was able to reproduce the problem I'm facing by simply trying to run an
> existing unit test from pig without modification of the code at all.
>
> https://github.com/nielsbasjes/pig-loader-testing
>
> So what I did:
>
> Create a very simple pom.xml
> Download
> https://raw.githubusercontent.com/apache/pig/branch-0.13/test/org/apache/pig/builtin/mock/TestMockStorage.java
> and put it in the right directory.
>
> Now when you run
>
> mvn test
>
>
> I see:
>
> java.lang.RuntimeException: No data for location 'bar'
>> at org.apache.pig.builtin.mock.Storage$Data.get(Storage.java:327)
>> at
>> org.apache.pig.builtin.mock.TestMockStorage.testMockStoreAndLoad(TestMockStorage.java:55)
>
>
> and
>
> junit.framework.AssertionFailedError:
>> Expected :{a: chararray,b: chararray}
>> Actual :null
>
>
> and
>
> java.lang.RuntimeException: No data for location 'output'
>> at org.apache.pig.builtin.mock.Storage$Data.get(Storage.java:327)
>> at
>> org.apache.pig.builtin.mock.TestMockStorage.testMockStoreUnion(TestMockStorage.java:106)
>
>
> Any help suggestions to what I'm doing wrong is very welcome.
>
> Niels Basjes
>
>
> On Sun, Nov 23, 2014 at 12:29 AM, Niels Basjes <[email protected]> wrote:
>
>> Hi,
>>
>> I have written a Pig Loader and I want to make unit tests for it that
>> actually use it from Pig.
>>
>> It's a Maven project and as a first attempt I created a testing class and
>> tried to create a java based unit test by doing something similar to this:
>>
>> https://svn.apache.org/repos/asf/pig/trunk/test/org/apache/pig/builtin/mock/TestMockStorage.java
>>
>> I couldn't get it to work so I copied the exact method
>> testMockStoreAndLoad and I still get this error:
>> *java.lang.RuntimeException: No data for location 'bar'*
>>
>> Does anyone have a suggestion on how to proceed?
>>
>> --
>> Best regards
>>
>> Niels Basjes
>>
>
>
>
> --
> Best regards / Met vriendelijke groeten,
>
> Niels Basjes
>
--
Best regards / Met vriendelijke groeten,
Niels Basjes
