[Ironruby-core] Unicode Source Files
Is the DLR going to be fixed so that it properly supports Unicode source files or is this an issue with IronRuby? If you attempt to create a new Code File with Visual Studio 2008 and call it test.rb and then execute it with: ScriptRuntime runtime = IronRuby.Ruby.CreateRuntime(); runtime.ExecuteFile( "test.rb" ); it blows up on the Unicode byte-order marker with: Unhandled Exception: Microsoft.Scripting.SyntaxErrorException: Invalid character 'ï' in expression at Microsoft.Scripting.ErrorSink.Add(SourceUnit source, String message, SourceSpan span, Int32 errorCode, Severity severity) in C:\Users\ted\Desktop\IronRuby\src\Microsoft.Scripting\ErrorSink.cs:line 34 at Microsoft.Scripting.ErrorCounter.Add(SourceUnit source, String message, SourceSpan span, Int32 errorCode, Severity severity) in C:\Users\ted\Desktop\IronRuby\src\Microsoft.Scripting\ErrorSink.cs:line 92 at IronRuby.Compiler.Tokenizer.Report(String message, Int32 errorCode, SourceSpan location, Severity severity) in C:\Users\ted\Desktop\IronRuby\src\ironruby\Compiler\Parser\Tokenizer.cs:line 430 at IronRuby.Compiler.Tokenizer.ReportError(ErrorInfo info, Object[] args) in C:\Users\ted\Desktop\IronRuby\src\ironruby\Compiler\Parser\Tokenizer.cs:line 442 at IronRuby.Compiler.Tokenizer.Tokenize(Boolean whitespaceSeen, Boolean cmdState) in C:\Users\ted\Desktop\IronRuby\src\ironruby\Compiler\Parser\Tokenizer.cs:line 966 at IronRuby.Compiler.Tokenizer.Tokenize() in C:\Users\ted\Desktop\IronRuby\src\ironruby\Compiler\Parser\Tokenizer.cs:line 739 at IronRuby.Compiler.Tokenizer.GetNextToken() in C:\Users\ted\Desktop\IronRuby\src\ironruby\Compiler\Parser\Tokenizer.cs:line 711 at IronRuby.Compiler.Parser.GetNextToken() in C:\Users\ted\Desktop\IronRuby\src\ironruby\Compiler\Parser\Parser.cs:line 99 at IronRuby.Compiler.ShiftReduceParser`2.Parse() in C:\Users\ted\Desktop\IronRuby\src\ironruby\Compiler\Parser\GPPG.cs:line 310 at IronRuby.Compiler.Parser.Parse(SourceUnit sourceUnit, RubyCompilerOptions options, ErrorSink errorSink) in C:\Users\ted\Desktop\IronRuby\src\ironruby\Compiler\Parser\Parser.cs:line 158 at IronRuby.Runtime.RubyContext.ParseSourceCode(SourceUnit sourceUnit, RubyCompilerOptions options, ErrorSink errorSink) in C:\Users\ted\Desktop\IronRuby\src\ironruby\Runtime\RubyContext.cs:line 203 at IronRuby.Runtime.RubyContext.CompileSourceCode(SourceUnit sourceUnit, CompilerOptions options, ErrorSink errorSink) in C:\Users\ted\Desktop\IronRuby\src\ironruby\Runtime\RubyContext.cs:line 179 at Microsoft.Scripting.SourceUnit.Compile(CompilerOptions options, ErrorSink errorSink) in C:\Users\ted\Desktop\IronRuby\src\Microsoft.Scripting\SourceUnit.cs:line 215 at Microsoft.Scripting.SourceUnit.Execute(Scope scope, ErrorSink errorSink) in C:\Users\ted\Desktop\IronRuby\src\Microsoft.Scripting\SourceUnit.cs:line 225 at Microsoft.Scripting.Hosting.ScriptSource.Execute(ScriptScope scope) in C:\Users\ted\Desktop\IronRuby\src\Microsoft.Scripting\Hosting\ScriptSource.cs:line 129 at Microsoft.Scripting.Hosting.ScriptEngine.ExecuteFile(String path, ScriptScope scope) in C:\Users\ted\Desktop\IronRuby\src\Microsoft.Scripting\Hosting\ScriptEngine.cs:line 159 at Microsoft.Scripting.Hosting.ScriptEngine.ExecuteFile(String path) in C:\Users\ted\Desktop\IronRuby\src\Microsoft.Scripting\Hosting\ScriptEngine.cs:line 148 at Microsoft.Scripting.Hosting.ScriptRuntime.ExecuteFile(String path) in C:\Users\ted\Desktop\IronRuby\src\Microsoft.Scripting\Hosting\ScriptRuntime.cs:line 257 at HostingDLRConsole.Program.Main(String[] args) in C:\Users\ted\Documents\Visual Studio 2008\Projects\Books\IronRuby in Action\HostingDLRConsole\HostingDLRConsole\Program.cs:line 14 Press any key to continue . . . I know I can fix this by using the Advanced Save Options but the DLR spec talks about Unicode support, so I assume this means that ScriptRuntime.ExecuteFile() should also support Unicode source files. ___ Ironruby-core mailing list Ironruby-core@rubyforge.org http://rubyforge.org/mailman/listinfo/ironruby-core
Re: [Ironruby-core] Unicode Source Files
We do this for compatibility with Ruby 1.8.6, though as you can see, we don't have the error message quite right: PS F:\> C:\ruby\bin\ruby.exe x.rb x.rb:1: Invalid char `\377' in expression x.rb:1: Invalid char `\376' in expression :) I believe you'll need to save as UTF-8 and then manually strip the BOM in order to use Unicode source files -- hopefully Tomas will tell me if I'm wrong. Source encoding for Ruby is extremely tricky, and (from what I can tell) hasn't even yet been finalized for 1.9.x. We will eventually support whatever the Ruby standards are. -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Ted Milker Sent: Sunday, October 26, 2008 9:38 AM To: ironruby-core@rubyforge.org Subject: [Ironruby-core] Unicode Source Files Is the DLR going to be fixed so that it properly supports Unicode source files or is this an issue with IronRuby? If you attempt to create a new Code File with Visual Studio 2008 and call it test.rb and then execute it with: ScriptRuntime runtime = IronRuby.Ruby.CreateRuntime(); runtime.ExecuteFile( "test.rb" ); it blows up on the Unicode byte-order marker with: Unhandled Exception: Microsoft.Scripting.SyntaxErrorException: Invalid character 'ï' in expression at Microsoft.Scripting.ErrorSink.Add(SourceUnit source, String message, SourceSpan span, Int32 errorCode, Severity severity) in C:\Users\ted\Desktop\IronRuby\src\Microsoft.Scripting\ErrorSink.cs:line 34 at Microsoft.Scripting.ErrorCounter.Add(SourceUnit source, String message, SourceSpan span, Int32 errorCode, Severity severity) in C:\Users\ted\Desktop\IronRuby\src\Microsoft.Scripting\ErrorSink.cs:line 92 at IronRuby.Compiler.Tokenizer.Report(String message, Int32 errorCode, SourceSpan location, Severity severity) in C:\Users\ted\Desktop\IronRuby\src\ironruby\Compiler\Parser\Tokenizer.cs:line 430 at IronRuby.Compiler.Tokenizer.ReportError(ErrorInfo info, Object[] args) in C:\Users\ted\Desktop\IronRuby\src\ironruby\Compiler\Parser\Tokenizer.cs:line 442 at IronRuby.Compiler.Tokenizer.Tokenize(Boolean whitespaceSeen, Boolean cmdState) in C:\Users\ted\Desktop\IronRuby\src\ironruby\Compiler\Parser\Tokenizer.cs:line 966 at IronRuby.Compiler.Tokenizer.Tokenize() in C:\Users\ted\Desktop\IronRuby\src\ironruby\Compiler\Parser\Tokenizer.cs:line 739 at IronRuby.Compiler.Tokenizer.GetNextToken() in C:\Users\ted\Desktop\IronRuby\src\ironruby\Compiler\Parser\Tokenizer.cs:line 711 at IronRuby.Compiler.Parser.GetNextToken() in C:\Users\ted\Desktop\IronRuby\src\ironruby\Compiler\Parser\Parser.cs:line 99 at IronRuby.Compiler.ShiftReduceParser`2.Parse() in C:\Users\ted\Desktop\IronRuby\src\ironruby\Compiler\Parser\GPPG.cs:line 310 at IronRuby.Compiler.Parser.Parse(SourceUnit sourceUnit, RubyCompilerOptions options, ErrorSink errorSink) in C:\Users\ted\Desktop\IronRuby\src\ironruby\Compiler\Parser\Parser.cs:line 158 at IronRuby.Runtime.RubyContext.ParseSourceCode(SourceUnit sourceUnit, RubyCompilerOptions options, ErrorSink errorSink) in C:\Users\ted\Desktop\IronRuby\src\ironruby\Runtime\RubyContext.cs:line 203 at IronRuby.Runtime.RubyContext.CompileSourceCode(SourceUnit sourceUnit, CompilerOptions options, ErrorSink errorSink) in C:\Users\ted\Desktop\IronRuby\src\ironruby\Runtime\RubyContext.cs:line 179 at Microsoft.Scripting.SourceUnit.Compile(CompilerOptions options, ErrorSink errorSink) in C:\Users\ted\Desktop\IronRuby\src\Microsoft.Scripting\SourceUnit.cs:line 215 at Microsoft.Scripting.SourceUnit.Execute(Scope scope, ErrorSink errorSink) in C:\Users\ted\Desktop\IronRuby\src\Microsoft.Scripting\SourceUnit.cs:line 225 at Microsoft.Scripting.Hosting.ScriptSource.Execute(ScriptScope scope) in C:\Users\ted\Desktop\IronRuby\src\Microsoft.Scripting\Hosting\ScriptSource.cs:line 129 at Microsoft.Scripting.Hosting.ScriptEngine.ExecuteFile(String path, ScriptScope scope) in C:\Users\ted\Desktop\IronRuby\src\Microsoft.Scripting\Hosting\ScriptEngine.cs:line 159 at Microsoft.Scripting.Hosting.ScriptEngine.ExecuteFile(String path) in C:\Users\ted\Desktop\IronRuby\src\Microsoft.Scripting\Hosting\ScriptEngine.cs:line 148 at Microsoft.Scripting.Hosting.ScriptRuntime.ExecuteFile(String path) in C:\Users\ted\Desktop\IronRuby\src\Microsoft.Scripting\Hosting\ScriptRuntime.cs:line 257 at HostingDLRConsole.Program.Main(String[] args) in C:\Users\ted\Documents\Visual Studio 2008\Projects\Books\IronRuby in Action\HostingDLRConsole\HostingDLRConsole\Program.cs:line 14 Press any key to continue . . . I know I can fix this by using the Advanced Save Options but the DLR spec talks about Unicode support, so I assume this means that ScriptRuntime.ExecuteFile() should also support Unicode source files. ___ Ironruby-core mailing list Ironruby-core@rubyforge.org http://rubyforge.org/mailman/listinfo/ironruby-core ___
Re: [Ironruby-core] Unicode Source Files
Why so rigorous? I understand the need to maintain compatibility but this effectively eliminates Visual Studio as an editor for .rb files, without some kind of clunky build mechanism. I guess I will just use an extension method to get around the behavior for the time being. >From the things I have read about Ruby and UTF-8, it seems more like it is just extremely broken, rather than extremely tricky. I still cannot even get pure Ruby stuff in Windows to work properly with UTF-8, like when using the Shoes toolkit for example. On Sun, Oct 26, 2008 at 11:52 AM, Curt Hagenlocher <[EMAIL PROTECTED]> wrote: > We do this for compatibility with Ruby 1.8.6, though as you can see, we don't > have the error message quite right: > > PS F:\> C:\ruby\bin\ruby.exe x.rb > x.rb:1: Invalid char `\377' in expression > x.rb:1: Invalid char `\376' in expression > > :) > > I believe you'll need to save as UTF-8 and then manually strip the BOM in > order to use Unicode source files -- hopefully Tomas will tell me if I'm > wrong. > > Source encoding for Ruby is extremely tricky, and (from what I can tell) > hasn't even yet been finalized for 1.9.x. We will eventually support > whatever the Ruby standards are. ___ Ironruby-core mailing list Ironruby-core@rubyforge.org http://rubyforge.org/mailman/listinfo/ironruby-core
Re: [Ironruby-core] Unicode Source Files
Here is the extension method I am using if anyone else is interested:
public static object ExecuteUnicodeFile( this ScriptRuntime rt, string
filename )
{
string rbCode;
// OpenText will strip the BOM and keep the Unicode intact
using( var rdr = File.OpenText( filename ) )
{
rbCode = rdr.ReadToEnd();
}
return IronRuby.Ruby.GetEngine( rt ).Execute( rbCode );
}
It works great for using Japanese in strings in Ruby with IronRuby and WPF.
___
Ironruby-core mailing list
Ironruby-core@rubyforge.org
http://rubyforge.org/mailman/listinfo/ironruby-core
Re: [Ironruby-core] Unicode Source Files
A lot of the work in 1.9 and 2.0 has gone to better unicode support. Most
string handling functions are now codepoint aware, and there is now the ability
for the source file to have an encoding attached to it.
Like Curt said, these are in flex, but they are spec'd in RubySpec, so they are
more than just fleeting ideas.
If you are able to solve this with an extension method, then it looks likely
that any VS integration work for IRb will take care of that. As it is, I use
GVim for most of my Ruby coding these days. :)
JD
-Original Message-
From: Ted Milker <[EMAIL PROTECTED]>
Sent: October 26, 2008 12:08 PM
To: ironruby-core@rubyforge.org
Subject: Re: [Ironruby-core] Unicode Source Files
Here is the extension method I am using if anyone else is interested:
public static object ExecuteUnicodeFile( this ScriptRuntime rt, string
filename )
{
string rbCode;
// OpenText will strip the BOM and keep the Unicode intact
using( var rdr = File.OpenText( filename ) )
{
rbCode = rdr.ReadToEnd();
}
return IronRuby.Ruby.GetEngine( rt ).Execute( rbCode );
}
It works great for using Japanese in strings in Ruby with IronRuby and WPF.
___
Ironruby-core mailing list
Ironruby-core@rubyforge.org
http://rubyforge.org/mailman/listinfo/ironruby-core
___
Ironruby-core mailing list
Ironruby-core@rubyforge.org
http://rubyforge.org/mailman/listinfo/ironruby-core
Re: [Ironruby-core] Unicode Source Files
If you save in "Western European (Windows) - Codepage 1252" from within Visual Studio, you'll get the right result -- as long as you're not using any characters with a codepoint greater than 127. And if you are, you're probably better off anyway expressing this code point as an explicit set of UTF-8 compatible bytes because -- as you've noticed -- Ruby's currently a bit weird in its Unicode support. -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Ted Milker Sent: Sunday, October 26, 2008 11:34 AM To: ironruby-core@rubyforge.org Subject: Re: [Ironruby-core] Unicode Source Files Why so rigorous? I understand the need to maintain compatibility but this effectively eliminates Visual Studio as an editor for .rb files, without some kind of clunky build mechanism. I guess I will just use an extension method to get around the behavior for the time being. >From the things I have read about Ruby and UTF-8, it seems more like it is just extremely broken, rather than extremely tricky. I still cannot even get pure Ruby stuff in Windows to work properly with UTF-8, like when using the Shoes toolkit for example. On Sun, Oct 26, 2008 at 11:52 AM, Curt Hagenlocher <[EMAIL PROTECTED]> wrote: > We do this for compatibility with Ruby 1.8.6, though as you can see, we don't > have the error message quite right: > > PS F:\> C:\ruby\bin\ruby.exe x.rb > x.rb:1: Invalid char `\377' in expression > x.rb:1: Invalid char `\376' in expression > > :) > > I believe you'll need to save as UTF-8 and then manually strip the BOM in > order to use Unicode source files -- hopefully Tomas will tell me if I'm > wrong. > > Source encoding for Ruby is extremely tricky, and (from what I can tell) > hasn't even yet been finalized for 1.9.x. We will eventually support > whatever the Ruby standards are. ___ Ironruby-core mailing list Ironruby-core@rubyforge.org http://rubyforge.org/mailman/listinfo/ironruby-core ___ Ironruby-core mailing list Ironruby-core@rubyforge.org http://rubyforge.org/mailman/listinfo/ironruby-core
Re: [Ironruby-core] Unicode Source Files
On Sun, Oct 26, 2008 at 3:17 PM, Jim Deville <[EMAIL PROTECTED]> wrote: > > If you are able to solve this with an extension method, then it looks likely > that any VS integration work for IRb will take care of that. As it is, I use > GVim for most of my Ruby coding these days. :) I use ViEmu for the best of both worlds. :) ___ Ironruby-core mailing list Ironruby-core@rubyforge.org http://rubyforge.org/mailman/listinfo/ironruby-core
Re: [Ironruby-core] Unicode Source Files
You can switch to 1.9 compat mode by passing -19 argument on command line. Tomas -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Ted Milker Sent: Sunday, October 26, 2008 1:57 PM To: ironruby-core@rubyforge.org Subject: Re: [Ironruby-core] Unicode Source Files On Sun, Oct 26, 2008 at 3:17 PM, Jim Deville <[EMAIL PROTECTED]> wrote: > > If you are able to solve this with an extension method, then it looks likely > that any VS integration work for IRb will take care of that. As it is, I use > GVim for most of my Ruby coding these days. :) I use ViEmu for the best of both worlds. :) ___ Ironruby-core mailing list Ironruby-core@rubyforge.org http://rubyforge.org/mailman/listinfo/ironruby-core ___ Ironruby-core mailing list Ironruby-core@rubyforge.org http://rubyforge.org/mailman/listinfo/ironruby-core
