Om, I thought you were going to get the image from my VM and run it on another
VM. Then we'd remove all non-release jobs from your VM and all of the release
jobs from my VM. That won't solve the problem of how long it takes for a
commit to be built and tested, but will allow the RM to get a release out
without getting stuck behind the commit jobs. Is that currently the plan?
-Alex
On 4/18/20, 12:43 AM, "OmPrakash Muppirala" <bigosma...@gmail.com> wrote:
Makes sense to me. I will set parallel executors back to 1.
And start working on a new Jenkins instance.
Thanks,
Om
On Sat, Apr 18, 2020, 12:28 AM Alex Harui <aha...@adobe.com.invalid> wrote:
> Are you saying that Java, Ant and Maven do not know how to distribute work
> to different cores?
>
> To repeat my main concern: our Jenkins jobs often use shared resources.
> If you want to take the time to fix that, feel free, but I do not know how
> you are going to keep two jobs from both trying to launch Flash Player
> Debugger at the same time and open the same socket to communicate with it,
> or try to parse the flashlog.txt that they have both written to.
>
> The royale-asjs job will use the royale-compiler workspace to get the
> compiler. If the compiler is building, the jars are not there and the
> royale-asjs build fails.
>
> There is also only so much memory and some of our jobs use all of it and
> I'm not sure if jobs slowed down due to paging memory and writing/reading
> from disk, even if it is an SSD.
>
> I'm done for tonight, but that's what I was seeing last time I tried to
> have both master and an agent running jobs in parallel.
>
> Thanks,
> -Alex
>
> On 4/17/20, 11:52 PM, "OmPrakash Muppirala" <bigosma...@gmail.com> wrote:
>
> A typical setup would be to have 0 executors on master and at least n
> number of executors on each agent, where n = number of cpu cores.
> I see that the current VM has 2 cores. So, I started with setting the
> num
> executors on the current agent to 2.
>
> This should improve things quite a bit. If this doesn't work, I will
> create a new VM and make it the slave agent of the current Jenkins
> master.
>
>
> Thanks,
> Om
>
> On Fri, Apr 17, 2020 at 11:31 PM Alex Harui <aha...@adobe.com.invalid>
> wrote:
>
> > On your VM or mine? I don't think mine can handle it. I used to
> run jobs
> > of master and agent and we got lots of failures and jobs took
> forever. If
> > two jobs are trying to run tests with the shared Flash Player
> Debugger, for
> > example, you won't get consistent results.
> >
> > -Alex
> >
> > On 4/17/20, 11:23 PM, "OmPrakash Muppirala" <bigosma...@gmail.com>
> wrote:
> >
> > I just realized that we have only one Agent (node) listed under
> > Jenkins. I
> > am going to add at least one more node to start with.
> >
> > Thanks,
> > Om
> >
> > On Thu, Apr 16, 2020 at 9:58 AM Alex Harui
> <aha...@adobe.com.invalid>
> > wrote:
> >
> > > I think I followed the instructions and enabled SSH.
> > >
> > > On 4/16/20, 9:37 AM, "Yishay Weiss" <yishayj...@hotmail.com>
> wrote:
> > >
> > > I haven’t gotten it to work. Could be ssh needs to be
> enabled on
> > the
> > > VM [1]. This is nice to have but not essential. So Alex, let
> me know
> > if you
> > > want to try that, otherwise I’ll suspend my efforts on this.
> > >
> > > [1]
> > >
> >
>
https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmedium.com%2Ftechinpieces%2Fpractical-azure-how-to-enable-ssh-on-azure-vm-84d8fba8103e&data=02%7C01%7Caharui%40adobe.com%7C15fe635dba674d69308108d7e36c2ea2%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C637227926092990578&sdata=%2FdPPdqLaeWZYyshv14%2BhhL0ffSznS%2BAUOGQ3xNnwido%3D&reserved=0
> > >
> > > From: Yishay Weiss<mailto:yishayj...@hotmail.com>
> > > Sent: Wednesday, April 15, 2020 5:09 PM
> > > To: dev@royale.apache.org<mailto:dev@royale.apache.org>
> > > Subject: RE: Prioritize Release Jobs on CI
> > >
> > > I’d like to try and install an ssh server. Can anyone
> think of a
> > > reason not to do that? I’m asking in terms of security, and if
> there
> > might
> > > be a problem because it’s a VM.
> > >
> > > Thanks.
> > >
> > > From: Yishay Weiss<mailto:yishayj...@hotmail.com>
> > > Sent: Monday, April 13, 2020 10:02 AM
> > > To: dev@royale.apache.org<mailto:dev@royale.apache.org>
> > > Subject: RE: Prioritize Release Jobs on CI
> > >
> > > Related, as most (all?) of the instructions don’t require
> > anything but
> > > a shell wouldn’t it be easier and faster to access the machine
> with
> > ssh
> > > [1], rather than using remote desktop?
> > >
> > > [1]
> > >
> >
>
https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fwindows-server%2Fadministration%2Fopenssh%2Fopenssh_install_firstuse&data=02%7C01%7Caharui%40adobe.com%7C15fe635dba674d69308108d7e36c2ea2%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C637227926092990578&sdata=pKNVfpAa0zGTTCz2jnsF9Uley6b%2BY%2Bg1ZlSWJ0%2FiYcY%3D&reserved=0
> > >
> > > From: Yishay Weiss<mailto:yishayj...@hotmail.com>
> > > Sent: Monday, April 13, 2020 9:55 AM
> > > To: dev@royale.apache.org<mailto:dev@royale.apache.org>
> > > Subject: RE: Prioritize Release Jobs on CI
> > >
> > > Can’t you do that with remote desktop?
> > >
> > > ________________________________
> > > From: OmPrakash Muppirala <bigosma...@gmail.com>
> > > Sent: Monday, April 13, 2020 9:51:25 AM
> > > To: Apache Royale Development <dev@royale.apache.org>
> > > Subject: Re: Prioritize Release Jobs on CI
> > >
> > > Alex,
> > >
> > > I believe this is your personal Azure subscription right?
> Any
> > chance
> > > I can
> > > get access to it so I can try to copy the image out the
> machine
> > and
> > > redeploy it somewhere else?
> > >
> > > Thanks,
> > > Om
> > >
> > > On Sun, Apr 12, 2020 at 6:18 PM OmPrakash Muppirala <
> > > bigosma...@gmail.com>
> > > wrote:
> > >
> > > > I can help set this up on Azure. Give me some time to
> work
> > out the
> > > > details?
> > > >
> > > > Thanks,
> > > > Om
> > > >
> > > > On Sun, Apr 12, 2020, 11:49 AM Harbs <
> harbs.li...@gmail.com>
> > wrote:
> > > >
> > > >> OK. Good to know.
> > > >>
> > > >> > On Apr 12, 2020, at 9:45 PM, Alex Harui
> > <aha...@adobe.com.INVALID
> > > >
> > > >> wrote:
> > > >> >
> > > >> > FWIW, I just looked and the longest job in the
release
> > steps so
> > > far is
> > > >> 8 minutes. Most are under 2 minutes. There might be
> jobs
> > later
> > > that take
> > > >> longer that we haven't run yet. IMO, the issue isn't
> speed
> > of the
> > > machine,
> > > >> it is just that we are sharing the machine with longer
> jobs
> > (1 hour
> > > for
> > > >> TourDeFlexMigration). And again, the machine will be
> idle for
> > > stretches of
> > > >> time while the RM verifies artifacts after each step.
> > > >> >
> > > >> > -Alex
> > > >> >
> > > >> > On 4/12/20, 11:32 AM, "Harbs" <harbs.li...@gmail.com
> > <mailto:
> > > >> harbs.li...@gmail.com>> wrote:
> > > >> >
> > > >> > Fair enough.
> > > >> >
> > > >> > I expect builds to be somewhere between 10 and 20
> times
> > faster
> > > on a
> > > >> powerful machine.
> > > >> >
> > > >> > Yeah. It’s probably going to be a bit of work
> changing
> > the
> > > server,
> > > >> but probably worth it in the long run.
> > > >> >
> > > >> > I think I’ll try this when I do the next release
> unless
> > Yishay
> > > wants
> > > >> to work with me on this for this release — but I’m not
> going
> > to be
> > > able to
> > > >> help until after Passover (i.e. next week).
> > > >> >
> > > >> > Thanks,
> > > >> > Harbs
> > > >> >
> > > >> >> On Apr 12, 2020, at 9:22 PM, Alex Harui
> > <aha...@adobe.com.INVALID
> > > >
> > > >> wrote:
> > > >> >>
> > > >> >> The Azure portal says: Standard F2s_v2 (2 vcpus, 4
> GiB
> > memory)
> > > >> >>
> > > >> >> I think I am reading changes to the build process in
> your
> > > >> suggestions. I do not really want to spend more of my
> time
> > on this
> > > >> process. But if you want to do the work, that's fine
> with me.
> > > >> >>
> > > >> >> -Alex
> > > >> >>
> > > >> >> On 4/12/20, 10:57 AM, "Harbs" <harbs.li...@gmail.com
> > <mailto:
> > > >> harbs.li...@gmail.com> <mailto:harbs.li...@gmail.com
> <mailto:
> > > >> harbs.li...@gmail.com>>> wrote:
> > > >> >>
> > > >> >> What kind of horsepower is included in the free
> Azure
> > account?
> > > >> >>
> > > >> >> The server I mentioned builds (considerably)
> faster than
> > my own
> > > >> local machine. The ci server seems to build many times
> slower.
> > > >> >>
> > > >> >> One thing we can do to minimize running server
time
> > would be to
> > > >> transfer the artifacts to storage instead of keeping
> them on
> > the
> > > server. On
> > > >> AWS, I’d probably use S3. Not sure what the similar
> service on
> > > Azure is
> > > >> called.
> > > >> >>
> > > >> >>> On Apr 12, 2020, at 8:26 PM, Alex Harui
> > > <aha...@adobe.com.INVALID
> > > >> <mailto:aha...@adobe.com.INVALID>> wrote:
> > > >> >>>
> > > >> >>> OK, that's pretty much how I understand Azure as
> well.
> > The key
> > > thing
> > > >> is that "running" includes time where the CI server is
> not
> > running
> > > any
> > > >> Jenkins jobs. The CI Server steps might take only a
few
> > hours of
> > > actual
> > > >> server time, but there is time where the RM is
verifying
> > artifacts
> > > locally
> > > >> so you'd be paying for that or the RM would have to
keep
> > shutting
> > > down and
> > > >> restarting.
> > > >> >>>
> > > >> >>> Seems like it would be cheaper/simpler to get the
> free
> > MSDN
> > > account
> > > >> and leave it running.
> > > >> >>>
> > > >> >>> -Alex
> > > >> >>>
> > > >> >>> On 4/12/20, 10:15 AM, "Harbs" <
> harbs.li...@gmail.com
> > <mailto:
> > > >> harbs.li...@gmail.com>> wrote:
> > > >> >>>
> > > >> >>> My experience is with AWS.
> > > >> >>>
> > > >> >>> I assume Microsoft has similar offerings, but I
> don’t
> > have
> > > >> experience with Azure.
> > > >> >>>
> > > >> >>> AWS has on-demand EC2 instances which you pay for
> only
> > the
> > > actual
> > > >> time that they are running.[1]
> > > >> >>>
> > > >> >>> Instances can be started and stopped via command
> line
> > (or via
> > > the
> > > >> web interface) as long as you have valid credentials to
> do so.
> > > >> >>>
> > > >> >>> For example: an m5.4xlarge instance has 16 cores
> and
> > costs
> > > about
> > > >> $1.5 per hour. On a machine like that, a full build
> would
> > probably
> > > take
> > > >> less than 10 minutes. It’s probably possible to do a
> full
> > release
> > > with only
> > > >> a few hours of server time.
> > > >> >>>
> > > >> >>> Leaving a server like that running all the time
> would get
> > > expensive,
> > > >> but if it’s just spun up for releases, you’d get very
> fast
> > builds
> > > at a
> > > >> reasonable price.
> > > >> >>>
> > > >> >>> I’d be happy to pay $10-$50 (and possibly more)
per
> > release to
> > > make
> > > >> the release process painless for the RM.
> > > >> >>>
> > > >> >>> [1]
> > > >>
> > >
> >
>
https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Faws.amazon.com%2Fec2%2Fpricing%2Fon-demand%2F&data=02%7C01%7Caharui%40adobe.com%7C15fe635dba674d69308108d7e36c2ea2%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C637227926092990578&sdata=AlpLmsJYbBpN%2BLTXIwn2ESklg5aU04KGAfWe2vhJxyc%3D&reserved=0
> > > >> <
> > > >>
> > >
> >
>
https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Faws.amazon.com%2Fec2%2Fpricing%2Fon-demand%2F&data=02%7C01%7Caharui%40adobe.com%7C15fe635dba674d69308108d7e36c2ea2%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C637227926092990578&sdata=AlpLmsJYbBpN%2BLTXIwn2ESklg5aU04KGAfWe2vhJxyc%3D&reserved=0
> > > >> ><
> > > >>
> > >
> >
>
https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Faws.amazon.com%2Fec2%2Fpricing%2Fon-demand%2F&data=02%7C01%7Caharui%40adobe.com%7C15fe635dba674d69308108d7e36c2ea2%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C637227926092990578&sdata=AlpLmsJYbBpN%2BLTXIwn2ESklg5aU04KGAfWe2vhJxyc%3D&reserved=0
> > > >> <
> > > >>
> > >
> >
>
https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Faws.amazon.com%2Fec2%2Fpricing%2Fon-demand%2F&data=02%7C01%7Caharui%40adobe.com%7C15fe635dba674d69308108d7e36c2ea2%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C637227926092990578&sdata=AlpLmsJYbBpN%2BLTXIwn2ESklg5aU04KGAfWe2vhJxyc%3D&reserved=0
> > > >> >><
> > > >>
> > >
> >
>
https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Faws.amazon.com%2Fec2%2Fpricing%2Fon-demand%2F&data=02%7C01%7Caharui%40adobe.com%7C15fe635dba674d69308108d7e36c2ea2%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C637227926092990578&sdata=AlpLmsJYbBpN%2BLTXIwn2ESklg5aU04KGAfWe2vhJxyc%3D&reserved=0
> > > >> <
> > > >>
> > >
> >
>
https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Faws.amazon.com%2Fec2%2Fpricing%2Fon-demand%2F&data=02%7C01%7Caharui%40adobe.com%7C15fe635dba674d69308108d7e36c2ea2%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C637227926093000574&sdata=YZE52sCQqRXb6CAMtU4DW3RKXjg%2FffHhiGy5b0Yz2I0%3D&reserved=0
> > > >> ><
> > > >>
> > >
> >
>
https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Faws.amazon.com%2Fec2%2Fpricing%2Fon-demand%2F&data=02%7C01%7Caharui%40adobe.com%7C15fe635dba674d69308108d7e36c2ea2%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C637227926093000574&sdata=YZE52sCQqRXb6CAMtU4DW3RKXjg%2FffHhiGy5b0Yz2I0%3D&reserved=0
> > > >> <
> > > >>
> > >
> >
>
https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Faws.amazon.com%2Fec2%2Fpricing%2Fon-demand%2F&data=02%7C01%7Caharui%40adobe.com%7C15fe635dba674d69308108d7e36c2ea2%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C637227926093000574&sdata=YZE52sCQqRXb6CAMtU4DW3RKXjg%2FffHhiGy5b0Yz2I0%3D&reserved=0
> > > >> >>>
> > > >> >>>
> > > >> >>>> On Apr 12, 2020, at 7:45 PM, Alex Harui
> > > <aha...@adobe.com.INVALID>
> > > >> wrote:
> > > >> >>>>
> > > >> >>>> I'm not very experienced with spinning up
> servers. The
> > CI
> > > server we
> > > >> are using is effectively free, based on a generous
> donation
> > from
> > > Microsoft
> > > >> of MSDN accounts to ASF committers. So I leave it up
> 24/7,
> > and
> > > share the
> > > >> RDP access on private@. I think any other ASF
> committer
> > could do
> > > the
> > > >> same. IIRC, if that server actually is stopped, I have
> to
> > use my
> > > personal
> > > >> (unshared) MSDN credentials to start it again. AIUI,
> if I
> > > actually paid
> > > >> for the server, it would cost me to leave it running
> even if
> > it
> > > didn't run
> > > >> jobs between releases.
> > > >> >>>>
> > > >> >>>> Is that what you are basically saying? I think it
> might
> > be
> > > best if
> > > >> another committer got a CI server going via the MS
> donation
> > and
> > > could leave
> > > >> it up 24/7.
> > > >> >>>>
> > > >> >>>> -Alex
> > > >> >>>>
> > > >> >>>> On 4/12/20, 9:28 AM, "Harbs" <
> harbs.li...@gmail.com>
> > wrote:
> > > >> >>>>
> > > >> >>>> I’m willing to do this.
> > > >> >>>>
> > > >> >>>> Considering that the release will be run
> infrequently, it
> > > should be
> > > >> doable to have a relatively powerful server that could
> be
> > spun up on
> > > >> demand. This is something I have setup for my own
> releases.
> > > >> >>>>
> > > >> >>>> The only complication would be that each RM would
> need
> > valid
> > > >> credentials to spin up the server.
> > > >> >>>>
> > > >> >>>> Harbs
> > > >> >>>>
> > > >> >>>>> On Apr 12, 2020, at 7:10 PM, Alex Harui
> > > <aha...@adobe.com.INVALID>
> > > >> wrote:
> > > >> >>>>>
> > > >> >>>>> A better solution, IMO, is for someone else to
> offer up
> > a CI
> > > server
> > > >> only for release jobs.
> > > >>
> > > >>
> > >
> > >
> > >
> > >
> >
> >
> >
>
>
>
