[ https://issues.apache.org/jira/browse/PIG-1339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12860606#action_12860606 ]
Ashutosh Chauhan commented on PIG-1339: --------------------------------------- This works fine on grunt. {code} grunt> a = load '1-3.txt' using PigStorage() as (あいうえお); grunt> dump a; {code} gives expected result. Problem is if it is fed as script to Pig {code} bin/pig myscript.pig {code} gives the exception as you shown above. This looks like a bug in PigScriptParser.jj where it should read the stream from script file as UTF-8. > International characters in column names not supported > ------------------------------------------------------ > > Key: PIG-1339 > URL: https://issues.apache.org/jira/browse/PIG-1339 > Project: Pig > Issue Type: Bug > Components: impl > Affects Versions: 0.6.0, 0.7.0, 0.8.0 > Reporter: Viraj Bhat > > There is a particular use-case in which someone specifies a column name to be > in International characters. > {code} > inputdata = load '/user/viraj/inputdata.txt' using PigStorage() as (あいうえお); > describe inputdata; > dump inputdata; > {code} > ====================================================================================================== > Pig Stack Trace > --------------- > ERROR 1000: Error during parsing. Lexical error at line 1, column 64. > Encountered: "\u3042" (12354), after : "" > org.apache.pig.impl.logicalLayer.parser.TokenMgrError: Lexical error at line > 1, column 64. Encountered: "\u3042" (12354), after : "" > at > org.apache.pig.impl.logicalLayer.parser.QueryParserTokenManager.getNextToken(QueryParserTokenManager.java:1791) > at > org.apache.pig.impl.logicalLayer.parser.QueryParser.jj_scan_token(QueryParser.java:8959) > at > org.apache.pig.impl.logicalLayer.parser.QueryParser.jj_3R_51(QueryParser.java:7462) > at > org.apache.pig.impl.logicalLayer.parser.QueryParser.jj_3R_120(QueryParser.java:7769) > at > org.apache.pig.impl.logicalLayer.parser.QueryParser.jj_3R_106(QueryParser.java:7787) > at > org.apache.pig.impl.logicalLayer.parser.QueryParser.jj_3R_63(QueryParser.java:8609) > at > org.apache.pig.impl.logicalLayer.parser.QueryParser.jj_3R_32(QueryParser.java:8621) > at > org.apache.pig.impl.logicalLayer.parser.QueryParser.jj_3_4(QueryParser.java:8354) > at > org.apache.pig.impl.logicalLayer.parser.QueryParser.jj_2_4(QueryParser.java:6903) > at > org.apache.pig.impl.logicalLayer.parser.QueryParser.BaseExpr(QueryParser.java:1249) > at > org.apache.pig.impl.logicalLayer.parser.QueryParser.Expr(QueryParser.java:911) > at > org.apache.pig.impl.logicalLayer.parser.QueryParser.Parse(QueryParser.java:700) > at > org.apache.pig.impl.logicalLayer.LogicalPlanBuilder.parse(LogicalPlanBuilder.java:63) > at org.apache.pig.PigServer$Graph.parseQuery(PigServer.java:1164) > at org.apache.pig.PigServer$Graph.registerQuery(PigServer.java:1114) > at org.apache.pig.PigServer.registerQuery(PigServer.java:425) > at > org.apache.pig.tools.grunt.GruntParser.processPig(GruntParser.java:737) > at > org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:324) > at > org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:162) > at > org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:138) > at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:89) > at org.apache.pig.Main.main(Main.java:391) > ====================================================================================================== > Thanks Viraj -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.