Update on current status: Thanks to Josh's fixes to compiler and typedefs we have reproducible binary artifacts in maven build. The main release archives, however, still have some issues:
a) Line ending inconsistency in source files. I think this is fixed by changing git global settings on CI to core.autocrlf to 'input'. b) differences in PNG, gif, ico, mp3, mp4, ttf for source files. I suspect ant zip task is compressing these, so I'll have to change build script to disable compression for those files. c) Encoding differences in txt, as, DEPENDENCIES source files. I will need to figure out where this encoding happens, I suspect it's a zip task issue too. ________________________________ From: Yishay Weiss <yishayj...@hotmail.com> Sent: Thursday, April 27, 2023 3:25 PM To: dev@royale.apache.org <dev@royale.apache.org> Subject: Re: 0.9.10 Release Thanks, that worked. ________________________________ From: Josh Tynjala <joshtynj...@bowlerhat.dev> Sent: Tuesday, April 25, 2023 9:15 PM To: dev@royale.apache.org <dev@royale.apache.org> Subject: Re: 0.9.10 Release It looks like the jquery-1.9.js externs download was being cached by download-maven-plugin, and one machine had an older version than the other. We were skipping the cache for other .js externs downloads already, so I updated them all to skip the cache. Hopefully, this will be the last difference. Give it another try! -- Josh Tynjala Bowler Hat LLC <https://bowlerhat.dev> On Thu, Apr 20, 2023 at 12:17 AM Yishay Weiss <yishayj...@hotmail.com> wrote: > I started another release candidate. Somehow, I'm getting a > reproducibility problem which did not exist before in type-defs. > > If somebody wants to confirm you can try to reproduce the problem with > the following step: > > > From the royale-typedefs repo: > 1. Run ant -f releasesteps.xml Release_Step_007 -Drelease.version=0.9.10 > -DskipTests=true > > I had to locally modify releasesteps.xml (line 122) to add CI > crendentials [1]. I still need to fix the permissions in CI. > > What I'm seeing is that locally built > royale-typedefs-jquery-0.9.10-typedefs.swc does not match archived one. > > [1] username="****" password="***" > ________________________________ > From: Josh Tynjala <joshtynj...@bowlerhat.dev> > Sent: Monday, April 17, 2023 11:52 PM > To: dev@royale.apache.org <dev@royale.apache.org> > Subject: Re: 0.9.10 Release > > Alright, my commit today seems to do the trick. All SWC files in the > js/libs, frameworks/libs, and frameworks/js/libs directories are a binary > match when I compare the Maven distribution as built on macOS vs built on > Windows. > > You need to make sure to pull the latest develop branch for both > royale-compiler and royale-asjs. > > -- > Josh Tynjala > Bowler Hat LLC <https://bowlerhat.dev> > > > On Sun, Apr 16, 2023 at 5:59 AM Yishay Weiss <yishayj...@hotmail.com> > wrote: > > > >I still need to figure out the duplicate hoisted variables in MXRoyale. > > > > I can confirm it's still happening. It does not happen on my machine with > > a simple "mvn clean install". > > ________________________________ > > From: Josh Tynjala <joshtynj...@bowlerhat.dev> > > Sent: Friday, April 7, 2023 2:01 AM > > To: dev@royale.apache.org <dev@royale.apache.org> > > Subject: Re: 0.9.10 Release > > > > My latest commits make the Jewel and JewelJS SWC files match between my > > macOS and Windows machines. I ended up having to switch to a different > > Maven Sass plugin because I realized the one we were using was no longer > > maintained, and it was misbehaving. This new plugin is based on the newer > > Sass that doesn't support CRLF output at all, so the changes I talked > about > > in my previous email ended up being a strict requirement. > > > > I still need to figure out the duplicate hoisted variables in MXRoyale. > > I'll need to dive into that on another day. > > > > -- > > Josh Tynjala > > Bowler Hat LLC <https://bowlerhat.dev> > > > > > > On Thu, Apr 6, 2023 at 9:57 AM Josh Tynjala <joshtynj...@bowlerhat.dev> > > wrote: > > > > > Some more findings: > > > > > > The difference in Jewel appears to be caused by defaults.css. The Maven > > > build at some point was updated to force Sass to output CRLF on > Windows, > > to > > > keep Git from saying that the file is modified after regenerating it. > > > Unfortunately, this breaks the ability to do a cross-platform > > reproducible > > > build because every other platform uses LF. I'm probably going to undo > > this > > > change (sorry, Greg!). However, it looks like adding a .gitattributes > > file > > > to the repo allows us to force line endings for specific files or > > > extensions, regardless of autocrlf behavior. It should provide the same > > > benefit as Greg's commit, while restoring reproducible builds. Side > > note: I > > > also found that modern Sass doesn't even provide the option to use > CRLF, > > so > > > this change is probably for the best because we may need to upgrade > Sass > > in > > > the future. > > > > > > The situation with duplicates of hoisted variables in > > > MXRoyaleBase/MXRoyale in JDK 17 might be a threading conflict (and > > probably > > > could happen with JDK 11 too, if conditions are correct). When the > issue > > > reproduces, I see that the function scope actually contains duplicate > > > definitions of every local variable, instead of just one of each. > > > Interestingly, if I add enough logging to help myself debug this issue, > > the > > > scope ends up containing one of each local variable, and it doesn't > > > duplicate the hoisted variables anymore. That makes me think multiple > > > threads may be manipulating the scope. > > > > > > -- > > > Josh Tynjala > > > Bowler Hat LLC <https://bowlerhat.dev> > > > > > > > > > On Wed, Apr 5, 2023 at 2:42 PM Josh Tynjala <joshtynj...@bowlerhat.dev > > > > > wrote: > > > > > >> Some preliminary findings: > > >> > > >> Jewel: > > >> - Some plaintext [Embed] content was not matching up between macOS and > > >> Windows. It was clearly a text encoding issue. I fixed a compiler bug > > >> where it was using the system default encoding instead of UTF-8. A > > >> workaround would be to use JAVA_TOOL_OPTIONS=-Dfile.encoding=UTF8. > > >> - The SWF files inside the Jewel SWCs still do not match between my > > >> machines. I still need to figure out why. Probably a missing compiler > > >> option in the Jewel pom.xml file. > > >> > > >> MXRoyale and MXRoyaleBase: > > >> - On Windows, I was seeing hoisted variables appear in the JS output > > >> twice. I noticed that I was using JDK 11 on macOS, and JDK 17 on > > Windows. > > >> When I switched to JDK 11 on Windows, the problem went away. It looks > > like > > >> there's a bug in the compiler that needs to be fixed to support JDK > 17. > > >> Until then, I recommend building with JDK 11 or older (8 should be > fine > > >> too). > > >> > > >> -- > > >> Josh Tynjala > > >> Bowler Hat LLC <https://bowlerhat.dev> > > >> > > >> > > >> On Wed, Apr 5, 2023 at 10:11 AM Josh Tynjala < > joshtynj...@bowlerhat.dev > > > > > >> wrote: > > >> > > >>> Alright, I built the distribution locally with Maven on two different > > >>> machines, and I can see that the following SWCs don't match: Jewel, > > >>> MXRoyale, and MXRoyaleBase > > >>> > > >>> It affects both the JS and SWF versions of the SWCs. > > >>> > > >>> I'll see what I can do. > > >>> > > >>> -- > > >>> Josh Tynjala > > >>> Bowler Hat LLC <https://bowlerhat.dev> > > >>> > > >>> > > >>> On Mon, Apr 3, 2023 at 8:04 AM Josh Tynjala < > joshtynj...@bowlerhat.dev > > > > > >>> wrote: > > >>> > > >>>> Yes, it's on by default. > > >>>> > > >>>> I guess that there must be some other reason why the builds don't > > match > > >>>> up. I just happened to notice this, in particular, was different > > between > > >>>> the Ant build and the Maven build. > > >>>> > > >>>> You may need to extract the .swc files and manually compare the > > >>>> contents to see what is different. > > >>>> > > >>>> -- > > >>>> Josh Tynjala > > >>>> Bowler Hat LLC <https://bowlerhat.dev> > > >>>> > > >>>> > > >>>> On Sun, Apr 2, 2023 at 4:01 AM Yishay Weiss <yishayj...@hotmail.com > > > > >>>> wrote: > > >>>> > > >>>>> Is that option on by default? I just tried building both on release > > >>>>> machine and locally and am still getting the mismatch. > > >>>>> > > >>>>> <exec executable="${mvn}" dir="${artifactfolder}/sources" > > >>>>> failonerror="true" > > > >>>>> <arg value="-X" /> > > >>>>> <arg value="-P" /> > > >>>>> <arg > > >>>>> > > > value="royale-release,apache-release,_reproducible-build,with-distribution,with-examples,with-manualtests,option-with-swf" > > >>>>> /> > > >>>>> <arg value="clean" /> > > >>>>> <arg value="install" /> > > >>>>> </exec> > > >>>>> > > >>>>> Is how it gets built now locally. Intuitively debug info should not > > be > > >>>>> part of the release but I need to look into it further. > > >>>>> ________________________________ > > >>>>> From: Josh Tynjala <joshtynj...@bowlerhat.dev> > > >>>>> Sent: Tuesday, March 28, 2023 6:17 PM > > >>>>> To: dev@royale.apache.org <dev@royale.apache.org> > > >>>>> Subject: Re: 0.9.10 Release > > >>>>> > > >>>>> Hey Yishay, > > >>>>> > > >>>>> I recently made a commit that adds the swf-debugfile-alias compiler > > >>>>> option > > >>>>> for SWC files built by Maven. This compiler option was being used > by > > >>>>> Ant > > >>>>> already, but it never got added to the Maven build. And I think > that > > >>>>> binaries would fail comparison without it. This may be why Release > > >>>>> Step 11 > > >>>>> was failing for you. > > >>>>> > > >>>>> Basically, swf-debugfile-alias ensures that the debugging > information > > >>>>> in > > >>>>> the SWC doesn't include the local path to the .as/.mxml source > files > > >>>>> (starting with something like C:\Users\joshtynjala\ or > > >>>>> /Users/CoolGuy19/). > > >>>>> Instead, it sets a generic path that will always match among builds > > >>>>> from > > >>>>> different machines. Considering this was missing, it sure sounds > > like a > > >>>>> reason why binary comparison would fail. > > >>>>> > > >>>>> -- > > >>>>> Josh Tynjala > > >>>>> Bowler Hat LLC <https://bowlerhat.dev> > > >>>>> > > >>>>> > > >>>>> On Wed, Feb 22, 2023 at 10:38 AM Yishay Weiss < > > yishayj...@hotmail.com> > > >>>>> wrote: > > >>>>> > > >>>>> > FYI I am put this on hold about 2 weeks ago after failing to > debug > > a > > >>>>> > binary reproducibility issue in as-js repo. > > >>>>> > > > >>>>> > @Harbs<mailto:harbs.li...@gmail.com>, can you see if the same > > issue > > >>>>> > exists for you? > > >>>>> > > > >>>>> > You can follow instructions [1] and tell me if step 2 succeeded > or > > >>>>> failed > > >>>>> > for you. > > >>>>> > > > >>>>> > Thanks. > > >>>>> > > > >>>>> > [1] > > >>>>> > > > >>>>> > > > apacheroyale.francecentral.cloudapp.azure.com:8080/job/Royale_Release_Step_011/ws/Success-5.txt > > >>>>> > < > > >>>>> > > > >>>>> > > > http://apacheroyale.francecentral.cloudapp.azure.com:8080/job/Royale_Release_Step_011/ws/Success-5.txt > > >>>>> > > > > >>>>> > ________________________________ > > >>>>> > From: Josh Tynjala <joshtynj...@bowlerhat.dev> > > >>>>> > Sent: Tuesday, January 17, 2023 10:08 PM > > >>>>> > To: dev@royale.apache.org <dev@royale.apache.org> > > >>>>> > Subject: Re: 0.9.10 Release > > >>>>> > > > >>>>> > Oops! I meant NO further changes. Sorry about that. You're good > to > > >>>>> go on my > > >>>>> > end. > > >>>>> > > > >>>>> > -- > > >>>>> > Josh Tynjala > > >>>>> > Bowler Hat LLC <https://bowlerhat.dev> > > >>>>> > > > >>>>> > > > >>>>> > On Tue, Jan 17, 2023 at 11:32 AM Yishay Weiss < > > >>>>> yishayj...@hotmail.com> > > >>>>> > wrote: > > >>>>> > > > >>>>> > > Thanks for the update, let me know when to pull the trigger. > > >>>>> > > > > >>>>> > > Get Outlook for Android<https://aka.ms/AAb9ysg> > > >>>>> > > ________________________________ > > >>>>> > > From: Josh Tynjala <joshtynj...@bowlerhat.dev> > > >>>>> > > Sent: Tuesday, January 17, 2023 7:05:29 PM > > >>>>> > > To: dev@royale.apache.org <dev@royale.apache.org> > > >>>>> > > Subject: Re: 0.9.10 Release > > >>>>> > > > > >>>>> > > I've updated the release notes with my changes. I have further > > >>>>> changes > > >>>>> > > planned before the release. > > >>>>> > > > > >>>>> > > -- > > >>>>> > > Josh Tynjala > > >>>>> > > Bowler Hat LLC <https://bowlerhat.dev> > > >>>>> > > > > >>>>> > > > > >>>>> > > On Mon, Jan 16, 2023 at 12:40 AM Yishay Weiss < > > >>>>> yishayj...@hotmail.com> > > >>>>> > > wrote: > > >>>>> > > > > >>>>> > > > Hi Guys, > > >>>>> > > > > > >>>>> > > > We’re starting a new release soon. As it’s been a while and I > > >>>>> expect > > >>>>> > some > > >>>>> > > > issues I have agreed with Harbs to do this release. 0.9.11 > > >>>>> should be > > >>>>> > > easier > > >>>>> > > > if we don’t wait too long, so the goal is for a different > > >>>>> volunteer to > > >>>>> > do > > >>>>> > > > the next one. > > >>>>> > > > > > >>>>> > > > If anybody wants to commit changes they are welcome to do so > > >>>>> now, and > > >>>>> > > > update the release notes. > > >>>>> > > > > > >>>>> > > > Please make sure the integration tests are working post > > commits. > > >>>>> > > > > > >>>>> > > > Thanks, > > >>>>> > > > Yishay > > >>>>> > > > > > >>>>> > > > > >>>>> > > > >>>>> > > >>>> > > >
