Heredocs do not appear to propagate file encoding
-------------------------------------------------
Key: JRUBY-5300
URL: http://jira.codehaus.org/browse/JRUBY-5300
Project: JRuby
Issue Type: Bug
Components: Parser
Reporter: Charles Oliver Nutter
Assignee: Charles Oliver Nutter
Fix For: JRuby 1.6
Recent parser work exposed this problem. Heredocs within an encoded file do not
appear to propagate the encoding.
File:
{noformat}
# encoding: utf-8
x = <<EOD
él piDió Un bUen café
EOD
{noformat}
Error:
{noformat}
~/projects/rails/activesupport ➔ jruby --1.9 blah.rb
: blah.rb:3: ASCII-8BIT mixed within UTF-8 (SyntaxError)
~/projects/rails/activesupport ➔ jruby -d -Xbacktrace.style=raw --1.9
blah.rb
org.jruby.lexer.yacc.SyntaxException: ASCII-8BIT mixed within UTF-8
at org.jruby.lexer.yacc.StringTerm.mixedEscape(StringTerm.java:214)
at
org.jruby.lexer.yacc.StringTerm.parseStringIntoBuffer(StringTerm.java:301)
at org.jruby.lexer.yacc.HeredocTerm.parseString(HeredocTerm.java:116)
at org.jruby.lexer.yacc.RubyYaccLexer.yylex(RubyYaccLexer.java:941)
at org.jruby.lexer.yacc.RubyYaccLexer.nextToken(RubyYaccLexer.java:311)
at org.jruby.parser.Ruby19Parser.yyparse(Ruby19Parser.java:1417)
at org.jruby.parser.Ruby19Parser.yyparse(Ruby19Parser.java:1368)
at org.jruby.parser.Ruby19Parser.parse(Ruby19Parser.java:4237)
at org.jruby.parser.Parser.parse(Parser.java:112)
at org.jruby.parser.Parser.parse(Parser.java:94)
at org.jruby.Ruby.parseFile(Ruby.java:2290)
at org.jruby.Ruby.parseFile(Ruby.java:2295)
at org.jruby.Ruby.parseFromMain(Ruby.java:444)
at org.jruby.Ruby.runFromMain(Ruby.java:403)
at org.jruby.Main.run(Main.java:304)
at org.jruby.Main.run(Main.java:144)
at org.jruby.Main.main(Main.java:113)
Thread.java:1503:in `getStackTrace': blah.rb:3: ASCII-8BIT mixed within UTF-8
(SyntaxError)
from RubyException.java:154:in `prepareBacktrace'
from RaiseException.java:156:in `preRaise'
from Ruby.java:3405:in `newRaiseException'
from Ruby.java:3234:in `newSyntaxError'
from Parser.java:140:in `parse'
from Parser.java:94:in `parse'
from Ruby.java:2290:in `parseFile'
from Ruby.java:2295:in `parseFile'
from Ruby.java:444:in `parseFromMain'
from Ruby.java:403:in `runFromMain'
from Main.java:304:in `run'
from Main.java:144:in `run'
from Main.java:113:in `main'
{noformat}
Possible fix:
{noformat}
diff --git a/src/org/jruby/lexer/yacc/HeredocTerm.java
b/src/org/jruby/lexer/yacc/HeredocTerm.java
index 81093ef..96cdf3e 100644
--- a/src/org/jruby/lexer/yacc/HeredocTerm.java
+++ b/src/org/jruby/lexer/yacc/HeredocTerm.java
@@ -82,6 +82,7 @@ public class HeredocTerm extends StrTerm {
}
ByteList str = new ByteList();
+ str.setEncoding(lexer.getEncoding());
ISourcePosition position;
if ((flags & RubyYaccLexer.STR_FUNC_EXPAND) == 0) {
{noformat}
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://jira.codehaus.org/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe from this list, please visit:
http://xircles.codehaus.org/manage_email
