Hey Johannes! Have you solved the problem? I also see it. But I don't see it when I use the schema as a string to AvroStorage parameter. I see it only when I try to use an external schema file. And if I specify a non-existent external schema file, the error is the same.
Ruslan On Tue, Oct 22, 2013 at 12:16 PM, Johannes Schwenk < [email protected]> wrote: > Thanks for your answer! > > Actually the Avro schema is valid and I can load data with it. The error > message states, that pig has a problem with the Pig schema, which has no > duplicate names. > > Johannes > > Am 21.10.2013 19:29, schrieb j.barrett Strausser: > > I'd imagime it is having an issue with the duplicate 'd' names? That is > my > > un-informed guess though. > > > > > > On Mon, Oct 21, 2013 at 1:05 PM, Johannes Schwenk < > > [email protected]> wrote: > > > >> Hi! > >> > >> I'm getting the following error when running the script below in grunt > >> (pig 0.12.0): > >> > >> DEFINE AvroStorageSTORE AvroStorage('-f schema.avsc'); > >> data = LOAD '/path/to/data' AS(a:long, b:long, c:int, d:chararray, > >> e:int, f:int, g:int, h:long, i:chararray, j:int, k:int, l:int, m:int, > >> n:int, o:int, p:int, q:int, r:int, s:int, t:int, u:int, v:chararray, > >> w:chararray, x:chararray); > >> STORE data into '/path/to/output' USING AvroStorageSTORE(); > >> > >> 2013-10-21 18:50:15,491 [main] INFO > >> org.apache.pig.tools.pigstats.ScriptState - Pig features used in the > >> script: UNKNOWN > >> 2013-10-21 18:50:15,554 [main] ERROR org.apache.pig.tools.grunt.Grunt - > >> ERROR 2116: > >> <line 3, column 0> Output Location Validation Failed for: > >> 'hdfs://path/to/output More info to follow: > >> Pig Schema contains a name that is not allowed in Avro > >> Details at logfile: pig_1382374188771.log > >> > >> Logfile contains: > >> > >> Pig Schema contains a name that is not allowed in Avro > >> > >> org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1002: Unable > >> to store alias data > >> at > >> org.apache.pig.PigServer$Graph.registerQuery(PigServer.java:1596) > >> at org.apache.pig.PigServer.registerQuery(PigServer.java:584) > >> at > >> org.apache.pig.tools.grunt.GruntParser.processPig(GruntParser.java:967) > >> at > >> > >> > org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:386) > >> at > >> > >> > org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:189) > >> at > >> > >> > org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:165) > >> at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:69) > >> at org.apache.pig.Main.run(Main.java:495) > >> at org.apache.pig.Main.main(Main.java:111) > >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > >> at > >> > >> > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > >> at > >> > >> > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > >> at java.lang.reflect.Method.invoke(Method.java:597) > >> at org.apache.hadoop.util.RunJar.main(RunJar.java:208) > >> Caused by: org.apache.pig.impl.plan.VisitorException: ERROR 2116: > >> <line 3, column 0> Output Location Validation Failed for: > >> 'hdfs://path/to/output More info to follow: > >> Pig Schema contains a name that is not allowed in Avro > >> at > >> > >> > org.apache.pig.newplan.logical.rules.InputOutputFileValidator$InputOutputFileVisitor.visit(InputOutputFileValidator.java:75) > >> at > >> > org.apache.pig.newplan.logical.relational.LOStore.accept(LOStore.java:77) > >> at > >> > >> > org.apache.pig.newplan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:64) > >> at > >> > >> > org.apache.pig.newplan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:66) > >> at > >> > >> > org.apache.pig.newplan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:66) > >> at > >> org.apache.pig.newplan.DepthFirstWalker.walk(DepthFirstWalker.java:53) > >> at org.apache.pig.newplan.PlanVisitor.visit(PlanVisitor.java:50) > >> at > >> > >> > org.apache.pig.newplan.logical.rules.InputOutputFileValidator.validate(InputOutputFileValidator.java:45) > >> at > >> > >> > org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.compile(HExecutionEngine.java:294) > >> at org.apache.pig.PigServer.compilePp(PigServer.java:1360) > >> at > >> org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1297) > >> at org.apache.pig.PigServer.execute(PigServer.java:1289) > >> at org.apache.pig.PigServer.access$400(PigServer.java:125) > >> at > >> org.apache.pig.PigServer$Graph.registerQuery(PigServer.java:1591) > >> ... 13 more > >> Caused by: java.io.IOException: Pig Schema contains a name that is not > >> allowed in Avro > >> at > >> > >> > org.apache.pig.impl.util.avro.AvroStorageSchemaConversionUtilities.toAvroName(AvroStorageSchemaConversionUtilities.java:380) > >> at > >> > >> > org.apache.pig.impl.util.avro.AvroStorageSchemaConversionUtilities.resourceSchemaToAvroSchema(AvroStorageSchemaConversionUtilities.java:326) > >> at > >> org.apache.pig.builtin.AvroStorage.checkSchema(AvroStorage.java:485) > >> at > >> > >> > org.apache.pig.newplan.logical.rules.InputOutputFileValidator$InputOutputFileVisitor.visit(InputOutputFileValidator.java:65) > >> ... 26 more > >> > >> > >> Avro schema looks like: > >> > >> { > >> "name": "entry", > >> "namespace": "a.b.c", > >> "type": "record", > >> "doc": "A Log", > >> "fields": [ > >> {"name": "a", "type": "long"}, > >> {"name": "b", "type": "long"}, > >> {"name": "c" "type": "int"}, > >> {"name": "d" "type": { > >> "name": "d" > >> "type": "enum", > >> "symbols": [ > >> "A", > >> "B", > >> "C", > >> "D", > >> "E" > >> ], > >> "doc": "Some doc" > >> } > >> }, > >> {"name": "e" "type": "int"}, > >> {"name": "f", "type": "int"}, > >> {"name": "g", "type": "int"}, > >> {"name": "h", "type": ["null", "long"], "default": > null}, > >> {"name": "i", "type": ["null", "string"], "default": > null}, > >> {"name": "j", "type": "int"}, > >> {"name": "k", "type": "int"}, > >> {"name": "l", "type": "int"}, > >> {"name": "m", "type": ["null", "int"], "default": null}, > >> {"name": "n", "type": ["null", "int"], "default": null}, > >> {"name": "o", "type": ["null", "int"], "default": null}, > >> {"name": "p", "type": ["null", "int"], "default": null}, > >> {"name": "q", "type": ["null", "int"], "default": null}, > >> {"name": "r", "type": ["null", "int"], "default": null}, > >> {"name": "s", "type": ["null", "int"], "default": null}, > >> {"name": "t", "type": "boolean"}, > >> {"name": "u", "type": "int"}, > >> {"name": "v", "type": "string"}, > >> {"name": "w", "type": ["null", "string"], "default": > null}, > >> {"name": "x", "type": "string"} > >> ] > >> } > >> > >> > >> Had to redact the names because of confidentiality. > >> > >> Do you have any idea what the problem could be? The error message seems > >> misleading... > >> > >> Greetings, > >> > >> Johannes Schwenk > >> > >> -- > >> Softwareentwickler (Reporting) > >> ________________________________________________________ > >> > >> ADITION technologies AG > >> Schwarzwaldstraße 78b > >> 79117 Freiburg > >> > >> http://www.adition.com > >> > >> T +49 / (0)761 / 88147 - 30 > >> F +49 / (0)761 / 88147 - 77 > >> SUPPORT +49 / (0)1805 - ADITION > >> > >> (Festnetzpreis 14 ct/min; Mobilfunkpreise maximal 42 ct/min) > >> > >> Eingetragen beim Amtsgericht Düsseldorf unter HRB 54076 > >> Vorstände: Andreas Kleiser, Jörg Klekamp, Tihomir Perkovic, Marcus > Schlüter > >> Aufsichtsratsvorsitzender: Joachim Schneidmadl > >> UStIDNr.: DE 218 858 434 > >> > >> > > > > > > > > Johannes Schwenk > > -- > Softwareentwickler (Reporting) > ________________________________________________________ > > ADITION technologies AG > Schwarzwaldstraße 78b > 79117 Freiburg > > http://www.adition.com > > T +49 / (0)761 / 88147 - 30 > F +49 / (0)761 / 88147 - 77 > SUPPORT +49 / (0)1805 - ADITION > > (Festnetzpreis 14 ct/min; Mobilfunkpreise maximal 42 ct/min) > > Eingetragen beim Amtsgericht Düsseldorf unter HRB 54076 > Vorstände: Andreas Kleiser, Jörg Klekamp, Tihomir Perkovic, Marcus Schlüter > Aufsichtsratsvorsitzender: Joachim Schneidmadl > UStIDNr.: DE 218 858 434 > >
