I'm a bit stumped here. The royale-compiler is failing with this error:
BUILD FAILEDc:\jenkins\workspace\royale-compiler\jenkins.xml:55: Execute failed: java.io.IOException: Cannot run program "git" (in directory "c:\jenkins\workspace\royale-compiler"): CreateProcess error=2, The system cannot find the file specified at java.lang.ProcessBuilder.start(Unknown Source) It is from this part of the jenkins.xml: <target name="closure-lib-download" description="Copies Closure Library from GitHub"> <echo message="Be patient. This takes a few minutes..." /> <exec executable="git" failifexecutionfails="true"> <arg value="clone" /> <arg value="https://github.com/google/closure-library" /> <arg value="${closure.root}" /> </exec> <echo>Cloned fresh copy of Google Closure Library</echo> </target> I am not sure how this executable call works without the full path. I checked the env variables, I do have both the git/bin and git/cmd directory in the PATH: [image: image.png] What am I missing? Thanks, Om On Sat, Apr 18, 2020 at 11:54 PM OmPrakash Muppirala <bigosma...@gmail.com> wrote: > Update - I have successfully attached my VM as a slave agent of our > Jenkins instance. > I am going to pull the job - /royale-asjs_jsonly and tie it to my agent so > I can test it. > > It will fail for a sometime while I go set up my VM with all the > dependencies. > > Once we have the job successfully building on my VM, we can figure out how > to divide the load between the master and agent. > > Thanks, > Om > > On Sat, Apr 18, 2020 at 8:59 AM OmPrakash Muppirala <bigosma...@gmail.com> > wrote: > >> We could achieve the same by creating a new agent (separate vm) and >> attaching it as a slave to the current Jenkins server. >> >> Then we can use labels/tags to tie release jobs to one agent and non >> release jobs to another agent. >> >> That way we can maintain everything in one UI, one login while achieving >> the concurrent building capabilities we want. >> >> Jenkins is built for this kind of stuff and I think it would be best we >> followed this practice. >> >> Thanks, >> Om >> >> On Sat, Apr 18, 2020, 8:18 AM Alex Harui <aha...@adobe.com.invalid> >> wrote: >> >>> Om, I thought you were going to get the image from my VM and run it on >>> another VM. Then we'd remove all non-release jobs from your VM and all of >>> the release jobs from my VM. That won't solve the problem of how long it >>> takes for a commit to be built and tested, but will allow the RM to get a >>> release out without getting stuck behind the commit jobs. Is that >>> currently the plan? >>> >>> -Alex >>> >>> On 4/18/20, 12:43 AM, "OmPrakash Muppirala" <bigosma...@gmail.com> >>> wrote: >>> >>> Makes sense to me. I will set parallel executors back to 1. >>> >>> And start working on a new Jenkins instance. >>> >>> Thanks, >>> Om >>> >>> >>> On Sat, Apr 18, 2020, 12:28 AM Alex Harui <aha...@adobe.com.invalid> >>> wrote: >>> >>> > Are you saying that Java, Ant and Maven do not know how to >>> distribute work >>> > to different cores? >>> > >>> > To repeat my main concern: our Jenkins jobs often use shared >>> resources. >>> > If you want to take the time to fix that, feel free, but I do not >>> know how >>> > you are going to keep two jobs from both trying to launch Flash >>> Player >>> > Debugger at the same time and open the same socket to communicate >>> with it, >>> > or try to parse the flashlog.txt that they have both written to. >>> > >>> > The royale-asjs job will use the royale-compiler workspace to get >>> the >>> > compiler. If the compiler is building, the jars are not there and >>> the >>> > royale-asjs build fails. >>> > >>> > There is also only so much memory and some of our jobs use all of >>> it and >>> > I'm not sure if jobs slowed down due to paging memory and >>> writing/reading >>> > from disk, even if it is an SSD. >>> > >>> > I'm done for tonight, but that's what I was seeing last time I >>> tried to >>> > have both master and an agent running jobs in parallel. >>> > >>> > Thanks, >>> > -Alex >>> > >>> > On 4/17/20, 11:52 PM, "OmPrakash Muppirala" <bigosma...@gmail.com> >>> wrote: >>> > >>> > A typical setup would be to have 0 executors on master and at >>> least n >>> > number of executors on each agent, where n = number of cpu >>> cores. >>> > I see that the current VM has 2 cores. So, I started with >>> setting the >>> > num >>> > executors on the current agent to 2. >>> > >>> > This should improve things quite a bit. If this doesn't work, >>> I will >>> > create a new VM and make it the slave agent of the current >>> Jenkins >>> > master. >>> > >>> > >>> > Thanks, >>> > Om >>> > >>> > On Fri, Apr 17, 2020 at 11:31 PM Alex Harui >>> <aha...@adobe.com.invalid> >>> > wrote: >>> > >>> > > On your VM or mine? I don't think mine can handle it. I >>> used to >>> > run jobs >>> > > of master and agent and we got lots of failures and jobs took >>> > forever. If >>> > > two jobs are trying to run tests with the shared Flash Player >>> > Debugger, for >>> > > example, you won't get consistent results. >>> > > >>> > > -Alex >>> > > >>> > > On 4/17/20, 11:23 PM, "OmPrakash Muppirala" < >>> bigosma...@gmail.com> >>> > wrote: >>> > > >>> > > I just realized that we have only one Agent (node) >>> listed under >>> > > Jenkins. I >>> > > am going to add at least one more node to start with. >>> > > >>> > > Thanks, >>> > > Om >>> > > >>> > > On Thu, Apr 16, 2020 at 9:58 AM Alex Harui >>> > <aha...@adobe.com.invalid> >>> > > wrote: >>> > > >>> > > > I think I followed the instructions and enabled SSH. >>> > > > >>> > > > On 4/16/20, 9:37 AM, "Yishay Weiss" < >>> yishayj...@hotmail.com> >>> > wrote: >>> > > > >>> > > > I haven’t gotten it to work. Could be ssh needs to >>> be >>> > enabled on >>> > > the >>> > > > VM [1]. This is nice to have but not essential. So >>> Alex, let >>> > me know >>> > > if you >>> > > > want to try that, otherwise I’ll suspend my efforts on >>> this. >>> > > > >>> > > > [1] >>> > > > >>> > > >>> > >>> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmedium.com%2Ftechinpieces%2Fpractical-azure-how-to-enable-ssh-on-azure-vm-84d8fba8103e&data=02%7C01%7Caharui%40adobe.com%7C15fe635dba674d69308108d7e36c2ea2%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C637227926092990578&sdata=%2FdPPdqLaeWZYyshv14%2BhhL0ffSznS%2BAUOGQ3xNnwido%3D&reserved=0 >>> > > > >>> > > > From: Yishay Weiss<mailto:yishayj...@hotmail.com> >>> > > > Sent: Wednesday, April 15, 2020 5:09 PM >>> > > > To: dev@royale.apache.org<mailto: >>> dev@royale.apache.org> >>> > > > Subject: RE: Prioritize Release Jobs on CI >>> > > > >>> > > > I’d like to try and install an ssh server. Can >>> anyone >>> > think of a >>> > > > reason not to do that? I’m asking in terms of >>> security, and if >>> > there >>> > > might >>> > > > be a problem because it’s a VM. >>> > > > >>> > > > Thanks. >>> > > > >>> > > > From: Yishay Weiss<mailto:yishayj...@hotmail.com> >>> > > > Sent: Monday, April 13, 2020 10:02 AM >>> > > > To: dev@royale.apache.org<mailto: >>> dev@royale.apache.org> >>> > > > Subject: RE: Prioritize Release Jobs on CI >>> > > > >>> > > > Related, as most (all?) of the instructions don’t >>> require >>> > > anything but >>> > > > a shell wouldn’t it be easier and faster to access the >>> machine >>> > with >>> > > ssh >>> > > > [1], rather than using remote desktop? >>> > > > >>> > > > [1] >>> > > > >>> > > >>> > >>> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fwindows-server%2Fadministration%2Fopenssh%2Fopenssh_install_firstuse&data=02%7C01%7Caharui%40adobe.com%7C15fe635dba674d69308108d7e36c2ea2%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C637227926092990578&sdata=pKNVfpAa0zGTTCz2jnsF9Uley6b%2BY%2Bg1ZlSWJ0%2FiYcY%3D&reserved=0 >>> > > > >>> > > > From: Yishay Weiss<mailto:yishayj...@hotmail.com> >>> > > > Sent: Monday, April 13, 2020 9:55 AM >>> > > > To: dev@royale.apache.org<mailto: >>> dev@royale.apache.org> >>> > > > Subject: RE: Prioritize Release Jobs on CI >>> > > > >>> > > > Can’t you do that with remote desktop? >>> > > > >>> > > > ________________________________ >>> > > > From: OmPrakash Muppirala <bigosma...@gmail.com> >>> > > > Sent: Monday, April 13, 2020 9:51:25 AM >>> > > > To: Apache Royale Development < >>> dev@royale.apache.org> >>> > > > Subject: Re: Prioritize Release Jobs on CI >>> > > > >>> > > > Alex, >>> > > > >>> > > > I believe this is your personal Azure subscription >>> right? >>> > Any >>> > > chance >>> > > > I can >>> > > > get access to it so I can try to copy the image >>> out the >>> > machine >>> > > and >>> > > > redeploy it somewhere else? >>> > > > >>> > > > Thanks, >>> > > > Om >>> > > > >>> > > > On Sun, Apr 12, 2020 at 6:18 PM OmPrakash >>> Muppirala < >>> > > > bigosma...@gmail.com> >>> > > > wrote: >>> > > > >>> > > > > I can help set this up on Azure. Give me some >>> time to >>> > work >>> > > out the >>> > > > > details? >>> > > > > >>> > > > > Thanks, >>> > > > > Om >>> > > > > >>> > > > > On Sun, Apr 12, 2020, 11:49 AM Harbs < >>> > harbs.li...@gmail.com> >>> > > wrote: >>> > > > > >>> > > > >> OK. Good to know. >>> > > > >> >>> > > > >> > On Apr 12, 2020, at 9:45 PM, Alex Harui >>> > > <aha...@adobe.com.INVALID >>> > > > > >>> > > > >> wrote: >>> > > > >> > >>> > > > >> > FWIW, I just looked and the longest job in >>> the release >>> > > steps so >>> > > > far is >>> > > > >> 8 minutes. Most are under 2 minutes. There >>> might be >>> > jobs >>> > > later >>> > > > that take >>> > > > >> longer that we haven't run yet. IMO, the issue >>> isn't >>> > speed >>> > > of the >>> > > > machine, >>> > > > >> it is just that we are sharing the machine with >>> longer >>> > jobs >>> > > (1 hour >>> > > > for >>> > > > >> TourDeFlexMigration). And again, the machine >>> will be >>> > idle for >>> > > > stretches of >>> > > > >> time while the RM verifies artifacts after each >>> step. >>> > > > >> > >>> > > > >> > -Alex >>> > > > >> > >>> > > > >> > On 4/12/20, 11:32 AM, "Harbs" < >>> harbs.li...@gmail.com >>> > > <mailto: >>> > > > >> harbs.li...@gmail.com>> wrote: >>> > > > >> > >>> > > > >> > Fair enough. >>> > > > >> > >>> > > > >> > I expect builds to be somewhere between 10 >>> and 20 >>> > times >>> > > faster >>> > > > on a >>> > > > >> powerful machine. >>> > > > >> > >>> > > > >> > Yeah. It’s probably going to be a bit of >>> work >>> > changing >>> > > the >>> > > > server, >>> > > > >> but probably worth it in the long run. >>> > > > >> > >>> > > > >> > I think I’ll try this when I do the next >>> release >>> > unless >>> > > Yishay >>> > > > wants >>> > > > >> to work with me on this for this release — but >>> I’m not >>> > going >>> > > to be >>> > > > able to >>> > > > >> help until after Passover (i.e. next week). >>> > > > >> > >>> > > > >> > Thanks, >>> > > > >> > Harbs >>> > > > >> > >>> > > > >> >> On Apr 12, 2020, at 9:22 PM, Alex Harui >>> > > <aha...@adobe.com.INVALID >>> > > > > >>> > > > >> wrote: >>> > > > >> >> >>> > > > >> >> The Azure portal says: Standard F2s_v2 (2 >>> vcpus, 4 >>> > GiB >>> > > memory) >>> > > > >> >> >>> > > > >> >> I think I am reading changes to the build >>> process in >>> > your >>> > > > >> suggestions. I do not really want to spend >>> more of my >>> > time >>> > > on this >>> > > > >> process. But if you want to do the work, >>> that's fine >>> > with me. >>> > > > >> >> >>> > > > >> >> -Alex >>> > > > >> >> >>> > > > >> >> On 4/12/20, 10:57 AM, "Harbs" < >>> harbs.li...@gmail.com >>> > > <mailto: >>> > > > >> harbs.li...@gmail.com> <mailto: >>> harbs.li...@gmail.com >>> > <mailto: >>> > > > >> harbs.li...@gmail.com>>> wrote: >>> > > > >> >> >>> > > > >> >> What kind of horsepower is included in the >>> free >>> > Azure >>> > > account? >>> > > > >> >> >>> > > > >> >> The server I mentioned builds >>> (considerably) >>> > faster than >>> > > my own >>> > > > >> local machine. The ci server seems to build >>> many times >>> > slower. >>> > > > >> >> >>> > > > >> >> One thing we can do to minimize running >>> server time >>> > > would be to >>> > > > >> transfer the artifacts to storage instead of >>> keeping >>> > them on >>> > > the >>> > > > server. On >>> > > > >> AWS, I’d probably use S3. Not sure what the >>> similar >>> > service on >>> > > > Azure is >>> > > > >> called. >>> > > > >> >> >>> > > > >> >>> On Apr 12, 2020, at 8:26 PM, Alex Harui >>> > > > <aha...@adobe.com.INVALID >>> > > > >> <mailto:aha...@adobe.com.INVALID>> wrote: >>> > > > >> >>> >>> > > > >> >>> OK, that's pretty much how I understand >>> Azure as >>> > well. >>> > > The key >>> > > > thing >>> > > > >> is that "running" includes time where the CI >>> server is >>> > not >>> > > running >>> > > > any >>> > > > >> Jenkins jobs. The CI Server steps might take >>> only a few >>> > > hours of >>> > > > actual >>> > > > >> server time, but there is time where the RM is >>> verifying >>> > > artifacts >>> > > > locally >>> > > > >> so you'd be paying for that or the RM would >>> have to keep >>> > > shutting >>> > > > down and >>> > > > >> restarting. >>> > > > >> >>> >>> > > > >> >>> Seems like it would be cheaper/simpler to >>> get the >>> > free >>> > > MSDN >>> > > > account >>> > > > >> and leave it running. >>> > > > >> >>> >>> > > > >> >>> -Alex >>> > > > >> >>> >>> > > > >> >>> On 4/12/20, 10:15 AM, "Harbs" < >>> > harbs.li...@gmail.com >>> > > <mailto: >>> > > > >> harbs.li...@gmail.com>> wrote: >>> > > > >> >>> >>> > > > >> >>> My experience is with AWS. >>> > > > >> >>> >>> > > > >> >>> I assume Microsoft has similar offerings, >>> but I >>> > don’t >>> > > have >>> > > > >> experience with Azure. >>> > > > >> >>> >>> > > > >> >>> AWS has on-demand EC2 instances which you >>> pay for >>> > only >>> > > the >>> > > > actual >>> > > > >> time that they are running.[1] >>> > > > >> >>> >>> > > > >> >>> Instances can be started and stopped via >>> command >>> > line >>> > > (or via >>> > > > the >>> > > > >> web interface) as long as you have valid >>> credentials to >>> > do so. >>> > > > >> >>> >>> > > > >> >>> For example: an m5.4xlarge instance has 16 >>> cores >>> > and >>> > > costs >>> > > > about >>> > > > >> $1.5 per hour. On a machine like that, a full >>> build >>> > would >>> > > probably >>> > > > take >>> > > > >> less than 10 minutes. It’s probably possible to >>> do a >>> > full >>> > > release >>> > > > with only >>> > > > >> a few hours of server time. >>> > > > >> >>> >>> > > > >> >>> Leaving a server like that running all the >>> time >>> > would get >>> > > > expensive, >>> > > > >> but if it’s just spun up for releases, you’d >>> get very >>> > fast >>> > > builds >>> > > > at a >>> > > > >> reasonable price. >>> > > > >> >>> >>> > > > >> >>> I’d be happy to pay $10-$50 (and possibly >>> more) per >>> > > release to >>> > > > make >>> > > > >> the release process painless for the RM. >>> > > > >> >>> >>> > > > >> >>> [1] >>> > > > >> >>> > > > >>> > > >>> > >>> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Faws.amazon.com%2Fec2%2Fpricing%2Fon-demand%2F&data=02%7C01%7Caharui%40adobe.com%7C15fe635dba674d69308108d7e36c2ea2%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C637227926092990578&sdata=AlpLmsJYbBpN%2BLTXIwn2ESklg5aU04KGAfWe2vhJxyc%3D&reserved=0 >>> > > > >> < >>> > > > >> >>> > > > >>> > > >>> > >>> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Faws.amazon.com%2Fec2%2Fpricing%2Fon-demand%2F&data=02%7C01%7Caharui%40adobe.com%7C15fe635dba674d69308108d7e36c2ea2%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C637227926092990578&sdata=AlpLmsJYbBpN%2BLTXIwn2ESklg5aU04KGAfWe2vhJxyc%3D&reserved=0 >>> > > > >> >< >>> > > > >> >>> > > > >>> > > >>> > >>> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Faws.amazon.com%2Fec2%2Fpricing%2Fon-demand%2F&data=02%7C01%7Caharui%40adobe.com%7C15fe635dba674d69308108d7e36c2ea2%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C637227926092990578&sdata=AlpLmsJYbBpN%2BLTXIwn2ESklg5aU04KGAfWe2vhJxyc%3D&reserved=0 >>> > > > >> < >>> > > > >> >>> > > > >>> > > >>> > >>> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Faws.amazon.com%2Fec2%2Fpricing%2Fon-demand%2F&data=02%7C01%7Caharui%40adobe.com%7C15fe635dba674d69308108d7e36c2ea2%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C637227926092990578&sdata=AlpLmsJYbBpN%2BLTXIwn2ESklg5aU04KGAfWe2vhJxyc%3D&reserved=0 >>> > > > >> >>< >>> > > > >> >>> > > > >>> > > >>> > >>> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Faws.amazon.com%2Fec2%2Fpricing%2Fon-demand%2F&data=02%7C01%7Caharui%40adobe.com%7C15fe635dba674d69308108d7e36c2ea2%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C637227926092990578&sdata=AlpLmsJYbBpN%2BLTXIwn2ESklg5aU04KGAfWe2vhJxyc%3D&reserved=0 >>> > > > >> < >>> > > > >> >>> > > > >>> > > >>> > >>> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Faws.amazon.com%2Fec2%2Fpricing%2Fon-demand%2F&data=02%7C01%7Caharui%40adobe.com%7C15fe635dba674d69308108d7e36c2ea2%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C637227926093000574&sdata=YZE52sCQqRXb6CAMtU4DW3RKXjg%2FffHhiGy5b0Yz2I0%3D&reserved=0 >>> > > > >> >< >>> > > > >> >>> > > > >>> > > >>> > >>> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Faws.amazon.com%2Fec2%2Fpricing%2Fon-demand%2F&data=02%7C01%7Caharui%40adobe.com%7C15fe635dba674d69308108d7e36c2ea2%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C637227926093000574&sdata=YZE52sCQqRXb6CAMtU4DW3RKXjg%2FffHhiGy5b0Yz2I0%3D&reserved=0 >>> > > > >> < >>> > > > >> >>> > > > >>> > > >>> > >>> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Faws.amazon.com%2Fec2%2Fpricing%2Fon-demand%2F&data=02%7C01%7Caharui%40adobe.com%7C15fe635dba674d69308108d7e36c2ea2%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C637227926093000574&sdata=YZE52sCQqRXb6CAMtU4DW3RKXjg%2FffHhiGy5b0Yz2I0%3D&reserved=0 >>> > > > >> >>> >>> > > > >> >>> >>> > > > >> >>>> On Apr 12, 2020, at 7:45 PM, Alex Harui >>> > > > <aha...@adobe.com.INVALID> >>> > > > >> wrote: >>> > > > >> >>>> >>> > > > >> >>>> I'm not very experienced with spinning up >>> > servers. The >>> > > CI >>> > > > server we >>> > > > >> are using is effectively free, based on a >>> generous >>> > donation >>> > > from >>> > > > Microsoft >>> > > > >> of MSDN accounts to ASF committers. So I leave >>> it up >>> > 24/7, >>> > > and >>> > > > share the >>> > > > >> RDP access on private@. I think any other ASF >>> > committer >>> > > could do >>> > > > the >>> > > > >> same. IIRC, if that server actually is >>> stopped, I have >>> > to >>> > > use my >>> > > > personal >>> > > > >> (unshared) MSDN credentials to start it again. >>> AIUI, >>> > if I >>> > > > actually paid >>> > > > >> for the server, it would cost me to leave it >>> running >>> > even if >>> > > it >>> > > > didn't run >>> > > > >> jobs between releases. >>> > > > >> >>>> >>> > > > >> >>>> Is that what you are basically saying? I >>> think it >>> > might >>> > > be >>> > > > best if >>> > > > >> another committer got a CI server going via the >>> MS >>> > donation >>> > > and >>> > > > could leave >>> > > > >> it up 24/7. >>> > > > >> >>>> >>> > > > >> >>>> -Alex >>> > > > >> >>>> >>> > > > >> >>>> On 4/12/20, 9:28 AM, "Harbs" < >>> > harbs.li...@gmail.com> >>> > > wrote: >>> > > > >> >>>> >>> > > > >> >>>> I’m willing to do this. >>> > > > >> >>>> >>> > > > >> >>>> Considering that the release will be run >>> > infrequently, it >>> > > > should be >>> > > > >> doable to have a relatively powerful server >>> that could >>> > be >>> > > spun up on >>> > > > >> demand. This is something I have setup for my >>> own >>> > releases. >>> > > > >> >>>> >>> > > > >> >>>> The only complication would be that each >>> RM would >>> > need >>> > > valid >>> > > > >> credentials to spin up the server. >>> > > > >> >>>> >>> > > > >> >>>> Harbs >>> > > > >> >>>> >>> > > > >> >>>>> On Apr 12, 2020, at 7:10 PM, Alex Harui >>> > > > <aha...@adobe.com.INVALID> >>> > > > >> wrote: >>> > > > >> >>>>> >>> > > > >> >>>>> A better solution, IMO, is for someone >>> else to >>> > offer up >>> > > a CI >>> > > > server >>> > > > >> only for release jobs. >>> > > > >> >>> > > > >> >>> > > > >>> > > > >>> > > > >>> > > > >>> > > >>> > > >>> > > >>> > >>> > >>> > >>> >>> >>>