[
https://issues.apache.org/jira/browse/PIG-2540?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Coveney updated PIG-2540:
----------------------------------
Attachment: PIG-2540_almost_there.patch
Russell, here is the change. Previously, there was this function in
TestAvroStorage:
{code}
private static String getInputFile(String file) {
return "file:///" + System.getProperty("user.dir") + "/" + basedir +
file;
}
{code}
The issue is that System.getProperty("user.dir") returns a directory that
begins with a /, so you were getting
{code}
file:////etc
{code}
By changing it accordingly, now all but one test run. The last test errors out,
but this is because the file "test_no_extension" doesn't exist.
> AvroStorage can't read schema on amazon s3 in elastic mapreduce
> ---------------------------------------------------------------
>
> Key: PIG-2540
> URL: https://issues.apache.org/jira/browse/PIG-2540
> Project: Pig
> Issue Type: Bug
> Components: data, piggybank
> Affects Versions: 0.9.1, 0.10
> Environment: Amazon Elastic MapReduce
> Reporter: Russell Jurney
> Priority: Critical
> Labels: avro, avro_udf, aws, emr, pants, pig, s3, sad, unhappy
> Fix For: 0.10
>
> Attachments: PIG-2540.tests_fail.patch, PIG-2540_almost_there.patch,
> TEST-org.apache.pig.piggybank.test.storage.avro.TestAvroStorage.txt
>
>
> grunt> emails = load 's3://agile.data/again_inbox' using AvroStorage();
> grunt> describe emails
> Schema for emails unknown.
> grunt> a = limit emails 10;
> grunt> dump a
> 2012-02-16 22:15:58,347 [main] INFO
> org.apache.pig.tools.pigstats.ScriptState - Pig features used in the script:
> LIMIT
> 2012-02-16 22:15:58,483 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler -
> File concatenation threshold: 100 optimistic? false
> 2012-02-16 22:15:58,542 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer
> - MR plan size before optimization: 1
> 2012-02-16 22:15:58,542 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer
> - MR plan size after optimization: 1
> 2012-02-16 22:15:58,632 [main] INFO
> org.apache.pig.tools.pigstats.ScriptState - Pig script settings are added to
> the job
> 2012-02-16 22:15:58,658 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
> - mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3
> 2012-02-16 22:15:58,665 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR
> 2017: Internal error creating job configuration.
> 2012-02-16 22:15:58,665 [main] ERROR org.apache.pig.tools.grunt.Grunt -
> org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066: Unable to
> open iterator for alias a
> at org.apache.pig.PigServer.openIterator(PigServer.java:901)
> at
> org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:652)
> at
> org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:303)
> at
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:188)
> at
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:164)
> at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:67)
> at org.apache.pig.Main.run(Main.java:497)
> at org.apache.pig.Main.main(Main.java:111)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
> Caused by: org.apache.pig.PigException: ERROR 1002: Unable to store alias a
> at org.apache.pig.PigServer.storeEx(PigServer.java:1000)
> at org.apache.pig.PigServer.store(PigServer.java:963)
> at org.apache.pig.PigServer.openIterator(PigServer.java:876)
> ... 12 more
> Caused by:
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobCreationException:
> ERROR 2017: Internal error creating job configuration.
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.getJob(JobControlCompiler.java:731)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.compile(JobControlCompiler.java:263)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:149)
> at org.apache.pig.PigServer.launchPlan(PigServer.java:1314)
> at
> org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1299)
> at org.apache.pig.PigServer.storeEx(PigServer.java:996)
> ... 14 more
> Caused by: java.lang.ArrayIndexOutOfBoundsException: 0
> at
> org.apache.hadoop.mapreduce.lib.input.FileInputFormat.setInputPaths(FileInputFormat.java:352)
> at
> org.apache.pig.piggybank.storage.avro.AvroStorage.setLocation(AvroStorage.java:138)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.getJob(JobControlCompiler.java:387)
> ... 19 more
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
