[ https://issues.apache.org/jira/browse/PIG-2411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Daniel Dai updated PIG-2411: ---------------------------- Attachment: messages.avro PIG-2411-1.patch Attach the patch from Russell's github. > AvroStorage UDF in PiggyBank fails to STORE a bag of single-field tuples as > Avro arrays > --------------------------------------------------------------------------------------- > > Key: PIG-2411 > URL: https://issues.apache.org/jira/browse/PIG-2411 > Project: Pig > Issue Type: Bug > Components: piggybank > Affects Versions: 0.9.0, 0.9.1, 0.9.2 > Environment: Mac OS X 10.6.8 in local mode > Reporter: Russell Jurney > Labels: patch > Fix For: 0.10 > > Attachments: PIG-2411-1.patch, avrobug.patch, messages.avro > > Original Estimate: 6h > Remaining Estimate: 6h > > This patch is a fix for a bug for persisting bags of tuples via AvroStorage. > The script that alerted me to a bug is: > messages = LOAD '/tmp/messages.avro' USING AvroStorage(); > user_groups = GROUP messages by user_id; > per_user = FOREACH user_groups { sorted = ORDER messages BY message_id DESC; > GENERATE group AS user_id, sorted AS messages; } > DESCRIBE per_user > > per_user: {user_id: int,messages: {(message_id: int,topic: > > chararray,user_id: int)}} > STORE per_user INTO '/tmp/per_user.avro' USING AvroStorage(); > The error is: > Pig Stack Trace > --------------- > ERROR 1002: Unable to store alias per_user > org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1002: Unable to > store alias per_user > at org.apache.pig.PigServer$Graph.registerQuery(PigServer.java:1596) > at org.apache.pig.PigServer.registerQuery(PigServer.java:584) > at org.apache.pig.tools.grunt.GruntParser.processPig(GruntParser.java:942) > at > org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:386) > at > org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:188) > at > org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:164) > at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:67) > at org.apache.pig.Main.run(Main.java:487) > at org.apache.pig.Main.main(Main.java:108) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at org.apache.hadoop.util.RunJar.main(RunJar.java:156) > Caused by: java.lang.NullPointerException > at > org.apache.pig.piggybank.storage.avro.AvroStorageUtils.isTupleWrapper(AvroStorageUtils.java:327) > at > org.apache.pig.piggybank.storage.avro.PigSchema2Avro.convert(PigSchema2Avro.java:82) > at > org.apache.pig.piggybank.storage.avro.PigSchema2Avro.convert(PigSchema2Avro.java:105) > at > org.apache.pig.piggybank.storage.avro.PigSchema2Avro.convertRecord(PigSchema2Avro.java:151) > at > org.apache.pig.piggybank.storage.avro.PigSchema2Avro.convert(PigSchema2Avro.java:62) > at > org.apache.pig.piggybank.storage.avro.AvroStorage.checkSchema(AvroStorage.java:502) > at > org.apache.pig.newplan.logical.rules.InputOutputFileValidator$InputOutputFileVisitor.visit(InputOutputFileValidator.java:65) > at org.apache.pig.newplan.logical.relational.LOStore.accept(LOStore.java:77) > at > org.apache.pig.newplan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:64) > at > org.apache.pig.newplan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:66) > at > org.apache.pig.newplan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:66) > at > org.apache.pig.newplan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:66) > at > org.apache.pig.newplan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:66) > at org.apache.pig.newplan.DepthFirstWalker.walk(DepthFirstWalker.java:53) > at org.apache.pig.newplan.PlanVisitor.visit(PlanVisitor.java:50) > at > org.apache.pig.newplan.logical.rules.InputOutputFileValidator.validate(InputOutputFileValidator.java:45) > at > org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.compile(HExecutionEngine.java:292) > at org.apache.pig.PigServer.compilePp(PigServer.java:1360) > at org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1297) > at org.apache.pig.PigServer.execute(PigServer.java:1286) > at org.apache.pig.PigServer.access$400(PigServer.java:125) > at org.apache.pig.PigServer$Graph.registerQuery(PigServer.java:1591) > ... 13 more -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira