Re: [royale-compiler] branch develop updated: JSRoyaleEmitter: when reading plain text files for [Embed] metadata, read as UTF-8 instead of system default
Oh, that makes it clear that UTF-8 is expected, and would presumably also account for utf-8 working by default (i.e without any configuration) in Flex compiler, in terms of aiming for consistency. OK, I will look into that next week some time, to see if I can address that too. Thanks, -Greg On Fri, Apr 7, 2023 at 7:50 AM Josh Tynjala wrote: > According to the compiler docs, .properties files are assumed to be UTF-8. > > > https://github.com/apache/royale-compiler/blob/apache-royale-0.9.9/compiler/src/main/java/org/apache/royale/compiler/internal/resourcebundles/PropertiesFileParser.java#L98-L100 > > -- > Josh Tynjala > Bowler Hat LLC <https://bowlerhat.dev> > > > On Wed, Apr 5, 2023 at 7:21 PM Greg Dove wrote: > > > @Josh: > > That sounds good to me, and makes things easier compared to what I was > > considering. > > > > > > @Alex: > > I was going to compare the behavior of the most recent Flex compiler and > > the Royale compiler to understand/avoid issues, but obviously only with > swf > > vs. swf. > > Then check for any difference in royale compiler swf vs. js. > > > > For Resource properties files I had only found one very. old 'known > issue' > > mention from the Flex 2 compiler which said: > > > >- All strings in properties files must be Latin-1 or UTF-8 encoded. > > > > I don't have experience using anything other than utf-8 throughout the > Flex > > years in these cases. The Flex 2 issue comment above seems to be quite > > restrictive for properties files. However that may have been 'fixed' in > > Flex 3 or Flex 4 compilers to allow more. > > I do suspect that even in more recent compilers it ought to have been > utf-8 > > by default though, I believe that is what we were seeing for the > migration > > project I was involved with, where the utf-8 encoding setting was needed > on > > the Royale compiler but not on the old Flex one.. My intention however > was > > to try to verify that before making any changes. If we do what Josh is > > suggesting then I would skip that work and just go with the explicit (but > > default Utf8) config setting. > > > > > > > > On Thu, Apr 6, 2023 at 11:07 AM Josh Tynjala > > wrote: > > > > > Maybe a new compiler option that will let folks choose an encoding > would > > be > > > best. Default to a sane UTF-8, and let people opt into playing with > weird > > > encodings, if they need it. > > > > > > -- > > > Josh Tynjala > > > Bowler Hat LLC <https://bowlerhat.dev> > > > > > > > > > On Wed, Apr 5, 2023 at 3:58 PM Alex Harui > > > wrote: > > > > > > > There is a chance this will break someone on Windows where the file > > > system > > > > is not UTF8 and the data includes Windows-encoded characters that > > > reference > > > > a file or something like that. > > > > > > > > IIRC, that's why I decided to live with JAVA_TOOL_OPTIONS, so we > don't > > > > make a change that can't be overridden. > > > > > > > > -Alex > > > > > > > > On 4/5/23, 1:47 PM, "Greg Dove" > > > greg.d...@gmail.com>> wrote: > > > > > > > > > > > > EXTERNAL: Use caution when clicking on links or opening attachments. > > > > > > > > > > > > > > > > > > > > Nice find, Josh. > > > > > > > > > > > > I was intending to look into some similar issue with resource* > > > > .properties *files > > > > as well. > > > > We have definitely had the need to set utf8 file.encoding in > > > > JAVA_TOOL_OPTIONS to get that to work the same as flex compiler did > for > > > > royale compiler (JS in particular, but I will try to check if royale > > swf > > > is > > > > at odds also), in some work over the last 12-18 months. > > > > This is just a mention, for your awareness, not a request for you to > do > > > > anything. I hope to find some time to spend on Royale work in the 2nd > > > half > > > > of April, so I will try to look into it then. > > > > > > > > > > > > -- Forwarded message - > > > > From: mailto:joshtynj...@apache.org>> > > > > Date: Thu, Apr 6, 2023 at 8:31 AM > > > > Subject: [royale-compiler] branch develop updated: JSRoyaleEmitter: > &g
Re: [royale-compiler] branch develop updated: JSRoyaleEmitter: when reading plain text files for [Embed] metadata, read as UTF-8 instead of system default
I ended up skipping the compiler option. Instead, [Embed] metadata with
mimeType="text/plain" can now optionally specify a charset attribute.
[Embed(source="file.txt",mimeType="text/plain",charset="CP1250")]
public var file:String;
To me, this makes more sense than a compiler option, since non-utf8
encoding should be rare. The vast majority of projects will be more likely
to have almost all embedded text files encoded as utf8, and only a small
number in a different encoding, for special circumstances.
--
Josh Tynjala
Bowler Hat LLC <https://bowlerhat.dev>
On Wed, Apr 5, 2023 at 4:06 PM Josh Tynjala
wrote:
> Maybe a new compiler option that will let folks choose an encoding would
> be best. Default to a sane UTF-8, and let people opt into playing with
> weird encodings, if they need it.
>
> --
> Josh Tynjala
> Bowler Hat LLC <https://bowlerhat.dev>
>
>
> On Wed, Apr 5, 2023 at 3:58 PM Alex Harui
> wrote:
>
>> There is a chance this will break someone on Windows where the file
>> system is not UTF8 and the data includes Windows-encoded characters that
>> reference a file or something like that.
>>
>> IIRC, that's why I decided to live with JAVA_TOOL_OPTIONS, so we don't
>> make a change that can't be overridden.
>>
>> -Alex
>>
>> On 4/5/23, 1:47 PM, "Greg Dove" > greg.d...@gmail.com>> wrote:
>>
>>
>> EXTERNAL: Use caution when clicking on links or opening attachments.
>>
>>
>>
>>
>> Nice find, Josh.
>>
>>
>> I was intending to look into some similar issue with resource*
>> .properties *files
>> as well.
>> We have definitely had the need to set utf8 file.encoding in
>> JAVA_TOOL_OPTIONS to get that to work the same as flex compiler did for
>> royale compiler (JS in particular, but I will try to check if royale swf
>> is
>> at odds also), in some work over the last 12-18 months.
>> This is just a mention, for your awareness, not a request for you to do
>> anything. I hope to find some time to spend on Royale work in the 2nd half
>> of April, so I will try to look into it then.
>>
>>
>> -- Forwarded message -
>> From: mailto:joshtynj...@apache.org>>
>> Date: Thu, Apr 6, 2023 at 8:31 AM
>> Subject: [royale-compiler] branch develop updated: JSRoyaleEmitter: when
>> reading plain text files for [Embed] metadata, read as UTF-8 instead of
>> system default
>> To: comm...@royale.apache.org <mailto:comm...@royale.apache.org> <
>> comm...@royale.apache.org <mailto:comm...@royale.apache.org>>
>>
>>
>>
>>
>> This is an automated email from the ASF dual-hosted git repository.
>>
>>
>> joshtynjala pushed a commit to branch develop
>> in repository https://gitbox.apache.org/repos/asf/royale-compiler.git <
>> https://gitbox.apache.org/repos/asf/royale-compiler.git>
>>
>>
>>
>>
>> The following commit(s) were added to refs/heads/develop by this push:
>> new 5669b4714 JSRoyaleEmitter: when reading plain text files for
>> [Embed] metadata, read as UTF-8 instead of system default
>> 5669b4714 is described below
>>
>>
>> commit 5669b4714118c1e2c2edf0e01360a39002e78033
>> Author: Josh Tynjala > joshtynj...@apache.org>>
>> AuthorDate: Wed Apr 5 13:31:34 2023 -0700
>>
>>
>> JSRoyaleEmitter: when reading plain text files for [Embed] metadata,
>> read as UTF-8 instead of system default
>>
>>
>> This matches the behavior of reading other text files, and avoids the
>> need for JAVA_TOOL_OPTIONS with file.encoding
>> ---
>> .../royale/compiler/internal/codegen/js/royale/JSRoyaleEmitter.java |
>> 2 +-
>> 1 file changed, 1 insertion(+), 1 deletion(-)
>>
>>
>> diff --git
>>
>> a/compiler-jx/src/main/java/org/apache/royale/compiler/internal/codegen/js/royale/JSRoyaleEmitter.java
>>
>> b/compiler-jx/src/main/java/org/apache/royale/compiler/internal/codegen/js/royale/JSRoyaleEmitter.java
>> index 10bc04b4e..2f67e14f5 100644
>> ---
>>
>> a/compiler-jx/src/main/java/org/apache/royale/compiler/internal/codegen/js/royale/JSRoyaleEmitter.java
>> +++
>>
>> b/compiler-jx/src/main/java/org/apache/royale/compiler/internal/codegen/js/royale/JSRoyaleEmitter.java
>> @@ -1097,7 +1097,7 @@ public class JSRoyaleEmitter extends JSGoogEmitter
>> implements IJSRoyaleEmitter
>> File file = new File(FilenameNormalization.normalize(source));
>> try {
>> String newlineReplacement = "n";
>> - String s = FileUtils.readFileToString(file);
>> + String s = FileUtils.readFileToString(file,
>> "UTF-8");
>> s = s.replaceAll("\n", "__NEWLINE_PLACEHOLDER__");
>> s = s.replaceAll("\r", "__CR_PLACEHOLDER__");
>> s = s.replaceAll("\t", "__TAB_PLACEHOLDER__");
>>
>>
>>
>>
Re: [royale-compiler] branch develop updated: JSRoyaleEmitter: when reading plain text files for [Embed] metadata, read as UTF-8 instead of system default
According to the compiler docs, .properties files are assumed to be UTF-8. https://github.com/apache/royale-compiler/blob/apache-royale-0.9.9/compiler/src/main/java/org/apache/royale/compiler/internal/resourcebundles/PropertiesFileParser.java#L98-L100 -- Josh Tynjala Bowler Hat LLC <https://bowlerhat.dev> On Wed, Apr 5, 2023 at 7:21 PM Greg Dove wrote: > @Josh: > That sounds good to me, and makes things easier compared to what I was > considering. > > > @Alex: > I was going to compare the behavior of the most recent Flex compiler and > the Royale compiler to understand/avoid issues, but obviously only with swf > vs. swf. > Then check for any difference in royale compiler swf vs. js. > > For Resource properties files I had only found one very. old 'known issue' > mention from the Flex 2 compiler which said: > >- All strings in properties files must be Latin-1 or UTF-8 encoded. > > I don't have experience using anything other than utf-8 throughout the Flex > years in these cases. The Flex 2 issue comment above seems to be quite > restrictive for properties files. However that may have been 'fixed' in > Flex 3 or Flex 4 compilers to allow more. > I do suspect that even in more recent compilers it ought to have been utf-8 > by default though, I believe that is what we were seeing for the migration > project I was involved with, where the utf-8 encoding setting was needed on > the Royale compiler but not on the old Flex one.. My intention however was > to try to verify that before making any changes. If we do what Josh is > suggesting then I would skip that work and just go with the explicit (but > default Utf8) config setting. > > > > On Thu, Apr 6, 2023 at 11:07 AM Josh Tynjala > wrote: > > > Maybe a new compiler option that will let folks choose an encoding would > be > > best. Default to a sane UTF-8, and let people opt into playing with weird > > encodings, if they need it. > > > > -- > > Josh Tynjala > > Bowler Hat LLC <https://bowlerhat.dev> > > > > > > On Wed, Apr 5, 2023 at 3:58 PM Alex Harui > > wrote: > > > > > There is a chance this will break someone on Windows where the file > > system > > > is not UTF8 and the data includes Windows-encoded characters that > > reference > > > a file or something like that. > > > > > > IIRC, that's why I decided to live with JAVA_TOOL_OPTIONS, so we don't > > > make a change that can't be overridden. > > > > > > -Alex > > > > > > On 4/5/23, 1:47 PM, "Greg Dove" > > greg.d...@gmail.com>> wrote: > > > > > > > > > EXTERNAL: Use caution when clicking on links or opening attachments. > > > > > > > > > > > > > > > Nice find, Josh. > > > > > > > > > I was intending to look into some similar issue with resource* > > > .properties *files > > > as well. > > > We have definitely had the need to set utf8 file.encoding in > > > JAVA_TOOL_OPTIONS to get that to work the same as flex compiler did for > > > royale compiler (JS in particular, but I will try to check if royale > swf > > is > > > at odds also), in some work over the last 12-18 months. > > > This is just a mention, for your awareness, not a request for you to do > > > anything. I hope to find some time to spend on Royale work in the 2nd > > half > > > of April, so I will try to look into it then. > > > > > > > > > -- Forwarded message - > > > From: mailto:joshtynj...@apache.org>> > > > Date: Thu, Apr 6, 2023 at 8:31 AM > > > Subject: [royale-compiler] branch develop updated: JSRoyaleEmitter: > when > > > reading plain text files for [Embed] metadata, read as UTF-8 instead of > > > system default > > > To: comm...@royale.apache.org <mailto:comm...@royale.apache.org> < > > > comm...@royale.apache.org <mailto:comm...@royale.apache.org>> > > > > > > > > > > > > > > > This is an automated email from the ASF dual-hosted git repository. > > > > > > > > > joshtynjala pushed a commit to branch develop > > > in repository https://gitbox.apache.org/repos/asf/royale-compiler.git > < > > > https://gitbox.apache.org/repos/asf/royale-compiler.git> > > > > > > > > > > > > > > > The following commit(s) were added to refs/heads/develop by this push: > > > new 5669b4714 JSRoyaleEmitter: when reading
Re: [royale-compiler] branch develop updated: JSRoyaleEmitter: when reading plain text files for [Embed] metadata, read as UTF-8 instead of system default
; > >
> > > Nice find, Josh.
> > >
> > >
> > > I was intending to look into some similar issue with resource*
> > > .properties *files
> > > as well.
> > > We have definitely had the need to set utf8 file.encoding in
> > > JAVA_TOOL_OPTIONS to get that to work the same as flex compiler did for
> > > royale compiler (JS in particular, but I will try to check if royale
> swf
> > is
> > > at odds also), in some work over the last 12-18 months.
> > > This is just a mention, for your awareness, not a request for you to do
> > > anything. I hope to find some time to spend on Royale work in the 2nd
> > half
> > > of April, so I will try to look into it then.
> > >
> > >
> > > -- Forwarded message -
> > > From: mailto:joshtynj...@apache.org> joshtynj...@apache.org <mailto:joshtynj...@apache.org>>>
> > > Date: Thu, Apr 6, 2023 at 8:31 AM
> > > Subject: [royale-compiler] branch develop updated: JSRoyaleEmitter:
> when
> > > reading plain text files for [Embed] metadata, read as UTF-8 instead of
> > > system default
> > > To: comm...@royale.apache.org <mailto:comm...@royale.apache.org>
> <mailto:comm...@royale.apache.org <mailto:comm...@royale.apache.org>> <
> > > comm...@royale.apache.org <mailto:comm...@royale.apache.org> comm...@royale.apache.org <mailto:comm...@royale.apache.org>>>
> > >
> > >
> > >
> > >
> > > This is an automated email from the ASF dual-hosted git repository.
> > >
> > >
> > > joshtynjala pushed a commit to branch develop
> > > in repository
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgitbox.apache.org%2Frepos%2Fasf%2Froyale-compiler.git&data=05%7C01%7Caharui%40adobe.com%7Cadd14bc9c46d4e843d3308db3645ae5d%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C638163445130547736%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=v%2FPjMQ6GCV9gMZ9Y8vsQ%2BKrB5mGoff7hd7%2B27WBNED4%3D&reserved=0
> <
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgitbox.apache.org%2Frepos%2Fasf%2Froyale-compiler.git&data=05%7C01%7Caharui%40adobe.com%7Cadd14bc9c46d4e843d3308db3645ae5d%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C638163445130547736%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=v%2FPjMQ6GCV9gMZ9Y8vsQ%2BKrB5mGoff7hd7%2B27WBNED4%3D&reserved=0>
> <
> > >
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgitbox.apache.org%2Frepos%2Fasf%2Froyale-compiler.git&data=05%7C01%7Caharui%40adobe.com%7Cadd14bc9c46d4e843d3308db3645ae5d%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C638163445130547736%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=v%2FPjMQ6GCV9gMZ9Y8vsQ%2BKrB5mGoff7hd7%2B27WBNED4%3D&reserved=0>
> <
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgitbox.apache.org%2Frepos%2Fasf%2Froyale-compiler.git&data=05%7C01%7Caharui%40adobe.com%7Cadd14bc9c46d4e843d3308db3645ae5d%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C638163445130547736%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=v%2FPjMQ6GCV9gMZ9Y8vsQ%2BKrB5mGoff7hd7%2B27WBNED4%3D&reserved=0>
> ;>
> > >
> > >
> > >
> > >
> > > The following commit(s) were added to refs/heads/develop by this push:
> > > new 5669b4714 JSRoyaleEmitter: when reading plain text files for
> > > [Embed] metadata, read as UTF-8 instead of system default
> > > 5669b4714 is described below
> > >
> > >
> > > commit 5669b4714118c1e2c2edf0e01360a39002e78033
> > > Author: Josh Tynjala joshtynj...@apache.org> > > joshtynj...@apache.org <mailto:joshtynj...@apache.org>>>
> > > AuthorDate: Wed Apr 5 13:31:34 2023 -0700
> > >
> > >
> > > JSRoyaleEmitter: when reading plain text files for [Embed] metadata,
> > > read as UTF-8 instead of system default
> > >
> > >
> > > This matches the behavior of reading other text files, and avoids the
> > > need for JAVA_TOOL_OPTIONS with file.encoding
> > > ---
> > > .../royale/compiler/internal/codegen/js/royale/JSRoyaleEmitter.java |
> > > 2 +-
> > > 1 file changed, 1 insertion(+), 1 deletion(-)
> > >
> > >
> > > diff --git
> > >
> > >
> >
> a/compiler-jx/src/main/java/org/apache/royale/compiler/internal/codegen/js/royale/JSRoyaleEmitter.java
> > >
> > >
> >
> b/compiler-jx/src/main/java/org/apache/royale/compiler/internal/codegen/js/royale/JSRoyaleEmitter.java
> > > index 10bc04b4e..2f67e14f5 100644
> > > ---
> > >
> > >
> >
> a/compiler-jx/src/main/java/org/apache/royale/compiler/internal/codegen/js/royale/JSRoyaleEmitter.java
> > > +++
> > >
> > >
> >
> b/compiler-jx/src/main/java/org/apache/royale/compiler/internal/codegen/js/royale/JSRoyaleEmitter.java
> > > @@ -1097,7 +1097,7 @@ public class JSRoyaleEmitter extends
> JSGoogEmitter
> > > implements IJSRoyaleEmitter
> > > File file = new File(FilenameNormalization.normalize(source));
> > > try {
> > > String newlineReplacement = "n";
> > > - String s = FileUtils.readFileToString(file);
> > > + String s = FileUtils.readFileToString(file,
> > > "UTF-8");
> > > s = s.replaceAll("\n", "__NEWLINE_PLACEHOLDER__");
> > > s = s.replaceAll("\r", "__CR_PLACEHOLDER__");
> > > s = s.replaceAll("\t", "__TAB_PLACEHOLDER__");
> > >
> > >
> > >
> > >
> >
>
>
>
>
Re: [royale-compiler] branch develop updated: JSRoyaleEmitter: when reading plain text files for [Embed] metadata, read as UTF-8 instead of system default
Pretty sure that if you compiled the same MXML file with Flex and Royale, there's no chance the SWF will be byte-for-byte the same. You have a better chance with an AS-only project, but I think there were different AST and output strategies in each compiler. Of course, I could be wrong, -Alex On 4/5/23, 7:21 PM, "Greg Dove" mailto:greg.d...@gmail.com>> wrote: EXTERNAL: Use caution when clicking on links or opening attachments. @Josh: That sounds good to me, and makes things easier compared to what I was considering. @Alex: I was going to compare the behavior of the most recent Flex compiler and the Royale compiler to understand/avoid issues, but obviously only with swf vs. swf. Then check for any difference in royale compiler swf vs. js. For Resource properties files I had only found one very. old 'known issue' mention from the Flex 2 compiler which said: - All strings in properties files must be Latin-1 or UTF-8 encoded. I don't have experience using anything other than utf-8 throughout the Flex years in these cases. The Flex 2 issue comment above seems to be quite restrictive for properties files. However that may have been 'fixed' in Flex 3 or Flex 4 compilers to allow more. I do suspect that even in more recent compilers it ought to have been utf-8 by default though, I believe that is what we were seeing for the migration project I was involved with, where the utf-8 encoding setting was needed on the Royale compiler but not on the old Flex one.. My intention however was to try to verify that before making any changes. If we do what Josh is suggesting then I would skip that work and just go with the explicit (but default Utf8) config setting. On Thu, Apr 6, 2023 at 11:07 AM Josh Tynjala mailto:joshtynj...@bowlerhat.dev>> wrote: > Maybe a new compiler option that will let folks choose an encoding would be > best. Default to a sane UTF-8, and let people opt into playing with weird > encodings, if they need it. > > -- > Josh Tynjala > Bowler Hat LLC > <https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbowlerhat.dev%2F&data=05%7C01%7Caharui%40adobe.com%7Cadd14bc9c46d4e843d3308db3645ae5d%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C638163445130547736%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=G81YOIv0fhkDO29mlR%2FZbNqcpy38abo61LK18azzooA%3D&reserved=0> > > <https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbowlerhat.dev%2F&data=05%7C01%7Caharui%40adobe.com%7Cadd14bc9c46d4e843d3308db3645ae5d%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C638163445130547736%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=G81YOIv0fhkDO29mlR%2FZbNqcpy38abo61LK18azzooA%3D&reserved=0>;> > > > On Wed, Apr 5, 2023 at 3:58 PM Alex Harui <mailto:aha...@adobe.com.inva>lid> > wrote: > > > There is a chance this will break someone on Windows where the file > system > > is not UTF8 and the data includes Windows-encoded characters that > reference > > a file or something like that. > > > > IIRC, that's why I decided to live with JAVA_TOOL_OPTIONS, so we don't > > make a change that can't be overridden. > > > > -Alex > > > > On 4/5/23, 1:47 PM, "Greg Dove" > <mailto:greg.d...@gmail.com> > greg.d...@gmail.com <mailto:greg.d...@gmail.com>>> wrote: > > > > > > EXTERNAL: Use caution when clicking on links or opening attachments. > > > > > > > > > > Nice find, Josh. > > > > > > I was intending to look into some similar issue with resource* > > .properties *files > > as well. > > We have definitely had the need to set utf8 file.encoding in > > JAVA_TOOL_OPTIONS to get that to work the same as flex compiler did for > > royale compiler (JS in particular, but I will try to check if royale swf > is > > at odds also), in some work over the last 12-18 months. > > This is just a mention, for your awareness, not a request for you to do > > anything. I hope to find some time to spend on Royale work in the 2nd > half > > of April, so I will try to look into it then. > > > > > > -- Forwarded message - > > From: mailto:joshtynj...@apache.org> > > <mailto:joshtynj...@apache.org <mailto:joshtynj...@apache.org>>> > > Date: Thu, Apr 6, 2023 at 8:31 AM > > Subject: [royale-compiler] branch develop updated: JSRoyaleEmitter: when > > reading plain text files for [Embed] metadata, read as UTF-8 instead of > > system default > > To: comm...@royale.apache.org <mailto:comm...@ro
Re: [royale-compiler] branch develop updated: JSRoyaleEmitter: when reading plain text files for [Embed] metadata, read as UTF-8 instead of system default
@Josh: That sounds good to me, and makes things easier compared to what I was considering. @Alex: I was going to compare the behavior of the most recent Flex compiler and the Royale compiler to understand/avoid issues, but obviously only with swf vs. swf. Then check for any difference in royale compiler swf vs. js. For Resource properties files I had only found one very. old 'known issue' mention from the Flex 2 compiler which said: - All strings in properties files must be Latin-1 or UTF-8 encoded. I don't have experience using anything other than utf-8 throughout the Flex years in these cases. The Flex 2 issue comment above seems to be quite restrictive for properties files. However that may have been 'fixed' in Flex 3 or Flex 4 compilers to allow more. I do suspect that even in more recent compilers it ought to have been utf-8 by default though, I believe that is what we were seeing for the migration project I was involved with, where the utf-8 encoding setting was needed on the Royale compiler but not on the old Flex one.. My intention however was to try to verify that before making any changes. If we do what Josh is suggesting then I would skip that work and just go with the explicit (but default Utf8) config setting. On Thu, Apr 6, 2023 at 11:07 AM Josh Tynjala wrote: > Maybe a new compiler option that will let folks choose an encoding would be > best. Default to a sane UTF-8, and let people opt into playing with weird > encodings, if they need it. > > -- > Josh Tynjala > Bowler Hat LLC <https://bowlerhat.dev> > > > On Wed, Apr 5, 2023 at 3:58 PM Alex Harui > wrote: > > > There is a chance this will break someone on Windows where the file > system > > is not UTF8 and the data includes Windows-encoded characters that > reference > > a file or something like that. > > > > IIRC, that's why I decided to live with JAVA_TOOL_OPTIONS, so we don't > > make a change that can't be overridden. > > > > -Alex > > > > On 4/5/23, 1:47 PM, "Greg Dove" > greg.d...@gmail.com>> wrote: > > > > > > EXTERNAL: Use caution when clicking on links or opening attachments. > > > > > > > > > > Nice find, Josh. > > > > > > I was intending to look into some similar issue with resource* > > .properties *files > > as well. > > We have definitely had the need to set utf8 file.encoding in > > JAVA_TOOL_OPTIONS to get that to work the same as flex compiler did for > > royale compiler (JS in particular, but I will try to check if royale swf > is > > at odds also), in some work over the last 12-18 months. > > This is just a mention, for your awareness, not a request for you to do > > anything. I hope to find some time to spend on Royale work in the 2nd > half > > of April, so I will try to look into it then. > > > > > > -- Forwarded message - > > From: mailto:joshtynj...@apache.org>> > > Date: Thu, Apr 6, 2023 at 8:31 AM > > Subject: [royale-compiler] branch develop updated: JSRoyaleEmitter: when > > reading plain text files for [Embed] metadata, read as UTF-8 instead of > > system default > > To: comm...@royale.apache.org <mailto:comm...@royale.apache.org> < > > comm...@royale.apache.org <mailto:comm...@royale.apache.org>> > > > > > > > > > > This is an automated email from the ASF dual-hosted git repository. > > > > > > joshtynjala pushed a commit to branch develop > > in repository https://gitbox.apache.org/repos/asf/royale-compiler.git < > > https://gitbox.apache.org/repos/asf/royale-compiler.git> > > > > > > > > > > The following commit(s) were added to refs/heads/develop by this push: > > new 5669b4714 JSRoyaleEmitter: when reading plain text files for > > [Embed] metadata, read as UTF-8 instead of system default > > 5669b4714 is described below > > > > > > commit 5669b4714118c1e2c2edf0e01360a39002e78033 > > Author: Josh Tynjala > joshtynj...@apache.org>> > > AuthorDate: Wed Apr 5 13:31:34 2023 -0700 > > > > > > JSRoyaleEmitter: when reading plain text files for [Embed] metadata, > > read as UTF-8 instead of system default > > > > > > This matches the behavior of reading other text files, and avoids the > > need for JAVA_TOOL_OPTIONS with file.encoding > > --- > > .../royale/compiler/internal/codegen/js/royale/JSRoyaleEmitter.java | > > 2 +- > > 1 file changed, 1 insertion(+), 1 deletion(-) > > > > > > diff --git > > > > > a/compiler-jx/src/main/java/org/apache/royal
Re: [royale-compiler] branch develop updated: JSRoyaleEmitter: when reading plain text files for [Embed] metadata, read as UTF-8 instead of system default
Maybe a new compiler option that will let folks choose an encoding would be
best. Default to a sane UTF-8, and let people opt into playing with weird
encodings, if they need it.
--
Josh Tynjala
Bowler Hat LLC <https://bowlerhat.dev>
On Wed, Apr 5, 2023 at 3:58 PM Alex Harui wrote:
> There is a chance this will break someone on Windows where the file system
> is not UTF8 and the data includes Windows-encoded characters that reference
> a file or something like that.
>
> IIRC, that's why I decided to live with JAVA_TOOL_OPTIONS, so we don't
> make a change that can't be overridden.
>
> -Alex
>
> On 4/5/23, 1:47 PM, "Greg Dove" greg.d...@gmail.com>> wrote:
>
>
> EXTERNAL: Use caution when clicking on links or opening attachments.
>
>
>
>
> Nice find, Josh.
>
>
> I was intending to look into some similar issue with resource*
> .properties *files
> as well.
> We have definitely had the need to set utf8 file.encoding in
> JAVA_TOOL_OPTIONS to get that to work the same as flex compiler did for
> royale compiler (JS in particular, but I will try to check if royale swf is
> at odds also), in some work over the last 12-18 months.
> This is just a mention, for your awareness, not a request for you to do
> anything. I hope to find some time to spend on Royale work in the 2nd half
> of April, so I will try to look into it then.
>
>
> ------ Forwarded message -----
> From: mailto:joshtynj...@apache.org>>
> Date: Thu, Apr 6, 2023 at 8:31 AM
> Subject: [royale-compiler] branch develop updated: JSRoyaleEmitter: when
> reading plain text files for [Embed] metadata, read as UTF-8 instead of
> system default
> To: comm...@royale.apache.org <mailto:comm...@royale.apache.org> <
> comm...@royale.apache.org <mailto:comm...@royale.apache.org>>
>
>
>
>
> This is an automated email from the ASF dual-hosted git repository.
>
>
> joshtynjala pushed a commit to branch develop
> in repository https://gitbox.apache.org/repos/asf/royale-compiler.git <
> https://gitbox.apache.org/repos/asf/royale-compiler.git>
>
>
>
>
> The following commit(s) were added to refs/heads/develop by this push:
> new 5669b4714 JSRoyaleEmitter: when reading plain text files for
> [Embed] metadata, read as UTF-8 instead of system default
> 5669b4714 is described below
>
>
> commit 5669b4714118c1e2c2edf0e01360a39002e78033
> Author: Josh Tynjala joshtynj...@apache.org>>
> AuthorDate: Wed Apr 5 13:31:34 2023 -0700
>
>
> JSRoyaleEmitter: when reading plain text files for [Embed] metadata,
> read as UTF-8 instead of system default
>
>
> This matches the behavior of reading other text files, and avoids the
> need for JAVA_TOOL_OPTIONS with file.encoding
> ---
> .../royale/compiler/internal/codegen/js/royale/JSRoyaleEmitter.java |
> 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
>
>
> diff --git
>
> a/compiler-jx/src/main/java/org/apache/royale/compiler/internal/codegen/js/royale/JSRoyaleEmitter.java
>
> b/compiler-jx/src/main/java/org/apache/royale/compiler/internal/codegen/js/royale/JSRoyaleEmitter.java
> index 10bc04b4e..2f67e14f5 100644
> ---
>
> a/compiler-jx/src/main/java/org/apache/royale/compiler/internal/codegen/js/royale/JSRoyaleEmitter.java
> +++
>
> b/compiler-jx/src/main/java/org/apache/royale/compiler/internal/codegen/js/royale/JSRoyaleEmitter.java
> @@ -1097,7 +1097,7 @@ public class JSRoyaleEmitter extends JSGoogEmitter
> implements IJSRoyaleEmitter
> File file = new File(FilenameNormalization.normalize(source));
> try {
> String newlineReplacement = "n";
> - String s = FileUtils.readFileToString(file);
> + String s = FileUtils.readFileToString(file,
> "UTF-8");
> s = s.replaceAll("\n", "__NEWLINE_PLACEHOLDER__");
> s = s.replaceAll("\r", "__CR_PLACEHOLDER__");
> s = s.replaceAll("\t", "__TAB_PLACEHOLDER__");
>
>
>
>
Re: [royale-compiler] branch develop updated: JSRoyaleEmitter: when reading plain text files for [Embed] metadata, read as UTF-8 instead of system default
There is a chance this will break someone on Windows where the file system is
not UTF8 and the data includes Windows-encoded characters that reference a file
or something like that.
IIRC, that's why I decided to live with JAVA_TOOL_OPTIONS, so we don't make a
change that can't be overridden.
-Alex
On 4/5/23, 1:47 PM, "Greg Dove" mailto:greg.d...@gmail.com>> wrote:
EXTERNAL: Use caution when clicking on links or opening attachments.
Nice find, Josh.
I was intending to look into some similar issue with resource*
.properties *files
as well.
We have definitely had the need to set utf8 file.encoding in
JAVA_TOOL_OPTIONS to get that to work the same as flex compiler did for
royale compiler (JS in particular, but I will try to check if royale swf is
at odds also), in some work over the last 12-18 months.
This is just a mention, for your awareness, not a request for you to do
anything. I hope to find some time to spend on Royale work in the 2nd half
of April, so I will try to look into it then.
-- Forwarded message -
From: mailto:joshtynj...@apache.org>>
Date: Thu, Apr 6, 2023 at 8:31 AM
Subject: [royale-compiler] branch develop updated: JSRoyaleEmitter: when
reading plain text files for [Embed] metadata, read as UTF-8 instead of
system default
To: comm...@royale.apache.org <mailto:comm...@royale.apache.org>
mailto:comm...@royale.apache.org>>
This is an automated email from the ASF dual-hosted git repository.
joshtynjala pushed a commit to branch develop
in repository https://gitbox.apache.org/repos/asf/royale-compiler.git
<https://gitbox.apache.org/repos/asf/royale-compiler.git>
The following commit(s) were added to refs/heads/develop by this push:
new 5669b4714 JSRoyaleEmitter: when reading plain text files for
[Embed] metadata, read as UTF-8 instead of system default
5669b4714 is described below
commit 5669b4714118c1e2c2edf0e01360a39002e78033
Author: Josh Tynjala mailto:joshtynj...@apache.org>>
AuthorDate: Wed Apr 5 13:31:34 2023 -0700
JSRoyaleEmitter: when reading plain text files for [Embed] metadata,
read as UTF-8 instead of system default
This matches the behavior of reading other text files, and avoids the
need for JAVA_TOOL_OPTIONS with file.encoding
---
.../royale/compiler/internal/codegen/js/royale/JSRoyaleEmitter.java |
2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git
a/compiler-jx/src/main/java/org/apache/royale/compiler/internal/codegen/js/royale/JSRoyaleEmitter.java
b/compiler-jx/src/main/java/org/apache/royale/compiler/internal/codegen/js/royale/JSRoyaleEmitter.java
index 10bc04b4e..2f67e14f5 100644
---
a/compiler-jx/src/main/java/org/apache/royale/compiler/internal/codegen/js/royale/JSRoyaleEmitter.java
+++
b/compiler-jx/src/main/java/org/apache/royale/compiler/internal/codegen/js/royale/JSRoyaleEmitter.java
@@ -1097,7 +1097,7 @@ public class JSRoyaleEmitter extends JSGoogEmitter
implements IJSRoyaleEmitter
File file = new File(FilenameNormalization.normalize(source));
try {
String newlineReplacement = "n";
- String s = FileUtils.readFileToString(file);
+ String s = FileUtils.readFileToString(file,
"UTF-8");
s = s.replaceAll("\n", "__NEWLINE_PLACEHOLDER__");
s = s.replaceAll("\r", "__CR_PLACEHOLDER__");
s = s.replaceAll("\t", "__TAB_PLACEHOLDER__");
Fwd: [royale-compiler] branch develop updated: JSRoyaleEmitter: when reading plain text files for [Embed] metadata, read as UTF-8 instead of system default
Nice find, Josh.
I was intending to look into some similar issue with resource*
.properties *files
as well.
We have definitely had the need to set utf8 file.encoding in
JAVA_TOOL_OPTIONS to get that to work the same as flex compiler did for
royale compiler (JS in particular, but I will try to check if royale swf is
at odds also), in some work over the last 12-18 months.
This is just a mention, for your awareness, not a request for you to do
anything. I hope to find some time to spend on Royale work in the 2nd half
of April, so I will try to look into it then.
-- Forwarded message -
From:
Date: Thu, Apr 6, 2023 at 8:31 AM
Subject: [royale-compiler] branch develop updated: JSRoyaleEmitter: when
reading plain text files for [Embed] metadata, read as UTF-8 instead of
system default
To: comm...@royale.apache.org
This is an automated email from the ASF dual-hosted git repository.
joshtynjala pushed a commit to branch develop
in repository https://gitbox.apache.org/repos/asf/royale-compiler.git
The following commit(s) were added to refs/heads/develop by this push:
new 5669b4714 JSRoyaleEmitter: when reading plain text files for
[Embed] metadata, read as UTF-8 instead of system default
5669b4714 is described below
commit 5669b4714118c1e2c2edf0e01360a39002e78033
Author: Josh Tynjala
AuthorDate: Wed Apr 5 13:31:34 2023 -0700
JSRoyaleEmitter: when reading plain text files for [Embed] metadata,
read as UTF-8 instead of system default
This matches the behavior of reading other text files, and avoids the
need for JAVA_TOOL_OPTIONS with file.encoding
---
.../royale/compiler/internal/codegen/js/royale/JSRoyaleEmitter.java |
2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git
a/compiler-jx/src/main/java/org/apache/royale/compiler/internal/codegen/js/royale/JSRoyaleEmitter.java
b/compiler-jx/src/main/java/org/apache/royale/compiler/internal/codegen/js/royale/JSRoyaleEmitter.java
index 10bc04b4e..2f67e14f5 100644
---
a/compiler-jx/src/main/java/org/apache/royale/compiler/internal/codegen/js/royale/JSRoyaleEmitter.java
+++
b/compiler-jx/src/main/java/org/apache/royale/compiler/internal/codegen/js/royale/JSRoyaleEmitter.java
@@ -1097,7 +1097,7 @@ public class JSRoyaleEmitter extends JSGoogEmitter
implements IJSRoyaleEmitter
File file = new File(FilenameNormalization.normalize(source));
try {
String newlineReplacement = "n";
- String s = FileUtils.readFileToString(file);
+ String s = FileUtils.readFileToString(file,
"UTF-8");
s = s.replaceAll("\n", "__NEWLINE_PLACEHOLDER__");
s = s.replaceAll("\r", "__CR_PLACEHOLDER__");
s = s.replaceAll("\t", "__TAB_PLACEHOLDER__");
