OK, I've now seen Jenkins perform the steps to build the release artifacts.
Folks interested in Docker-izing the steps are welcome to look at the jobs on
the "Royale Release" tab on the CI server.
http://apacheroyaleci.westus2.cloudapp.azure.com:8080/view/Royale%20Release/
These steps assume that the RM can run the basic Maven and Ant build on the
RM's computer. I think that's a fair requirement since all of us on the PMC
need to able to do that to build the RC in order to vote on it. Jenkins does
other tasks like run the Maven release plugin steps.
Currently that results in binaries on Jenkins that are downloaded to the RM's
computer. These binaries need to be verified by the RM which is the next phase
I will be starting on now. The RM verifies the bits and then PGP signs them.
And then the bits are uploaded off the RM's computer to Maven Staging or
dist.a.o/dev. If that uploading turns out to be a point of failure, we have
the option of having Jenkins upload the big files and have the RM only upload
PGP signature files. Or finding a way for Jenkins to get the signature files
from the RM. The uploads worked fine for me, but then again, so did the old
script's uploads.
Therefore, once I get the binary verification phase completed, I think someone
other than me should be the RM and try to use these steps to generate the
release and help debug the process for the next RM. So, please try to carve
out some time to be the RM. One advantage of doing most of the work on Jenkins
is that it frees up my computer to do other things while Jenkins is cranking
away.
I think we're at least a week away from binary verification, maybe two, so it
is time to start thinking about what is going in this release.
Thanks,
-Alex
On 3/7/19, 4:15 PM, "Alex Harui" <aha...@adobe.com.INVALID> wrote:
In case you haven't guessed, I'm testing out Jenkins and its ability to
create the artifacts and send emails. Please ignore any email that looks like
a vote or discuss thread.
Thanks,
-Alex
On 2/10/19, 8:44 PM, "Alex Harui" <aha...@adobe.com.INVALID> wrote:
Om,
Well, that's what I used for the last release. I'm sure there is
probably some inaccuracy in it.
That said, I'm abandoning that document and taking a new angle because
that document presumed that the release manager was trying to create a release
on his/her computer. I've given up on that and working on making releases from
a shared computer for the reasons I've stated upthread. I hope to make some
progress on that this week.
The key difference is that the new angle does not presume that you have
Git SSH and PGP signatures all set up on some computer. I have not looked into
how Docker would handle that. You certainly wouldn't want the Docker image to
contain your SSH or PGP keys/creds. And if the Docker image doesn't, then
that is another stumbling block for future RMs.
The other key difference is that the old script presumed you could
create the 3 release in 3 huge "easy" steps. We've seen that is only true for
me. So the new angle creates many discrete steps managed by Jenkins.
So, it is up to you to decide what you want to "Docker-ize". You can
try to Docker-ize the current 3 big steps, so RMs can try to run it on their
systems, but I'd bet they will just faiI due to network issues. I would be
interested in using Docker to make each of these many discrete steps portable
to another server. I'm not going to involve Docker at this point. My main
goal is just to see if I can create a workflow of many discrete steps that
isn't horribly painful. Once we see what these steps turn out to be, then we
can worry about server portability of those steps.
Either way, we want to know about running Browser+Selenium for sure.
And maybe FlashPlayerDebugger or AIR. I would want to know, for example, how
you debug a failing checkintest in a Docker container.
Thanks,
-Alex
On 2/10/19, 5:18 PM, "OmPrakash Muppirala" <bigosma...@gmail.com> wrote:
Alex,
Just to be clear, I am following the steps from here to try and
setup a
docker container.
https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Froyale-asjs%2Fwiki%2FRelease-Manager-Notes&data=02%7C01%7Caharui%40adobe.com%7C9a4f659f80694687c3cd08d6a35b378d%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C636876009505592396&sdata=rJsHtgUQ%2BH5iQOXlZc%2FCEvtxH0Lm25fGn0wlAe6oqSs%3D&reserved=0
Is this doc up to date?
Thanks,
Om
On Wed, Feb 6, 2019 at 12:02 PM OmPrakash Muppirala
<bigosma...@gmail.com>
wrote:
> One approach is to have each step in the process spin up a docker
image.
> And use docker-compose to run each step. All the images can be
made to
> share a common volume where all the artifacts are stored across
steps.
>
> You are right about the networking issue though. Any network
related
> failure that occurs on the host machine will most likely occur in
the
> docker container. Although, I am not clear how we can guarantee
that the
> same issues will not occur on the Jenkins server. I mean, what
is special
> about the Jenkins server that makes it immune to these networking
issues?
>
> Thanks,
> Om
>
> On Wed, Feb 6, 2019 at 11:48 AM Alex Harui
<aha...@adobe.com.invalid>
> wrote:
>
>> Again, I don't know anything about Docker, but before we spend a
lot of
>> time on Docker, I also want to point out that the process to
create a
>> release cannot really be thought of as "one application". It
will be a
>> series of "steps" to run. How many steps depends on whether
we think we
>> can isolate enough stuff via Docker to be able to run Docker on
the RM's
>> computer instead of some shared computer. On a shared computer
there will
>> be dozens of steps because the RM will need to enter passwords
to commit
>> stuff. On a local computer I guess the RM can supply passwords
but I think
>> there will be stopping points where the Maven artifacts are
deployed and
>> the staging repo is closed, and another stopping point for the
vote. It
>> seems like Docker works by downloading dependencies. Given that
the
>> problem the RMs had last time involved downloads and uploads,
why do we
>> think Docker will really solve this for creating releases on
local machines?
>>
>> The reason to do this on a shared machine is so that new RMs
don't have
>> to do as much setup. But then I wonder about the efficiency of
kicking off
>> that many Docker images. Jenkins can manage that already. Does
Docker
>> have some sort of similar Dashboard or would we use Jenkins to
kick off
>> Docker steps? I can't quite picture what is the outermost
>> control/dashboard.
>>
>> -Alex
>>
>> On 2/6/19, 11:03 AM, "Harbs" <harbs.li...@gmail.com> wrote:
>>
>> A quick search turns up this:
>>
>>
>>
https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcodingsans.com%2Fblog%2Fselenium-with-docker-testing&data=02%7C01%7Caharui%40adobe.com%7C9a4f659f80694687c3cd08d6a35b378d%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C636876009505592396&sdata=fA7bthRUAGXzqJdmK4raheZMEHQ4qyy3NHXjufAF6jc%3D&reserved=0
>> <
>>
https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcodingsans.com%2Fblog%2Fselenium-with-docker-testing&data=02%7C01%7Caharui%40adobe.com%7C9a4f659f80694687c3cd08d6a35b378d%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C636876009505592396&sdata=fA7bthRUAGXzqJdmK4raheZMEHQ4qyy3NHXjufAF6jc%3D&reserved=0
>> >
>>
>>
https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.blazemeter.com%2Fblog%2Fhow-to-run-selenium-tests-in-docker&data=02%7C01%7Caharui%40adobe.com%7C9a4f659f80694687c3cd08d6a35b378d%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C636876009505592396&sdata=QeJ5h98H2ghmY%2BMoGWKVgYaS6OKiv9BY7Cbu%2FRYh9K0%3D&reserved=0
>> <
>>
https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.blazemeter.com%2Fblog%2Fhow-to-run-selenium-tests-in-docker&data=02%7C01%7Caharui%40adobe.com%7C9a4f659f80694687c3cd08d6a35b378d%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C636876009505592396&sdata=QeJ5h98H2ghmY%2BMoGWKVgYaS6OKiv9BY7Cbu%2FRYh9K0%3D&reserved=0
>> >
>>
>>
https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FSeleniumHQ%2Fdocker-selenium&data=02%7C01%7Caharui%40adobe.com%7C9a4f659f80694687c3cd08d6a35b378d%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C636876009505592396&sdata=pvNt%2B4Spevu9uXaZeX1bV4AKi3p1KVnwND4XZvqlxt0%3D&reserved=0
>> <
>>
https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FSeleniumHQ%2Fdocker-selenium&data=02%7C01%7Caharui%40adobe.com%7C9a4f659f80694687c3cd08d6a35b378d%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C636876009505592396&sdata=pvNt%2B4Spevu9uXaZeX1bV4AKi3p1KVnwND4XZvqlxt0%3D&reserved=0
>> >
>>
>> > On Feb 6, 2019, at 8:59 PM, Alex Harui
<aha...@adobe.com.INVALID>
>> wrote:
>> >
>> > Thanks for volunteering to try it Om. IMO, even more
important
>> than Firefox+Flash is SomeBrowser+Selenium. We may also need to
run Adobe
>> AIR's adb. We could probably turn off the Flash tests or
replace Flash
>> with AIR.
>> >
>> > Thanks,
>> > -Alex
>> >
>> > On 2/6/19, 10:54 AM, "Carlos Rovira"
<carlosrov...@apache.org>
>> wrote:
>> >
>> > Hi Om, that would be great! waiting for your experience
with a
>> that! :)
>> >
>> > El mié., 6 feb. 2019 a las 19:31, OmPrakash Muppirala (<
>> bigosma...@gmail.com>)
>> > escribió:
>> >
>> >> Yes, I agree with Alex about not being able to run UI out
of docker
>> >> containers. I never thought of the checkintests when I
made the
>> >> suggestion.
>> >>
>> >> The firefox-flash image that Yishay pointed out looks
promising.
>> I will
>> >> poke around with it and see if that works for us.
>> >>
>> >> Thanks,
>> >> Om
>> >>
>> >> On Wed, Feb 6, 2019 at 12:41 AM Yishay Weiss <
>> yishayj...@hotmail.com>
>> >> wrote:
>> >>
>> >>> I’ll let Om or someone else with docker experience tell
us if
>> this [1] is
>> >>> relevant.
>> >>>
>> >>>
>> >>>
>> >>> [1]
>>
https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fhub.docker.com%2Fr%2Fbeli%2Ffirefox-flash%2F&data=02%7C01%7Caharui%40adobe.com%7C9a4f659f80694687c3cd08d6a35b378d%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C636876009505592396&sdata=O5WCCm7fcFGjaXpoDum5hMAJXl3JWscFMRMywpBAN1w%3D&reserved=0
>> >>>
>> >>>
>> >>>
>> >>> ________________________________
>> >>> From: Alex Harui <aha...@adobe.com.INVALID>
>> >>> Sent: Wednesday, February 6, 2019 10:05:54 AM
>> >>> To: dev@royale.apache.org
>> >>> Subject: Re: 0.9.6 Release
>> >>>
>> >>> I don't anything about docker, but in 15 minutes of
reading I ran
>> into
>> >>> this:
>> >>>
>> >>>
>> >>
>>
https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.channelfutures.com%2Fopen-source%2Fwhen-not-to-use-docker-understanding-the-limitations-of-containers&data=02%7C01%7Caharui%40adobe.com%7C9a4f659f80694687c3cd08d6a35b378d%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C636876009505602405&sdata=bjIQhi1pOBfBd8jQAc0XmbrsWzlDwSqieUiUoSovnIE%3D&reserved=0
>> >>>
>> >>> Which says: Docker can't " Run applications with
graphical
>> interfaces".
>> >>> If you want Royale to use Docker for releases, show that
it can
>> run
>> >>> checkintests with Flash and the Browser. Then I will
look into
>> it more.
>> >>>
>> >>> The highest level goal is to make it as easy as possible
for
>> someone to
>> >>> volunteer to be an RM. Any requirement of "install this
(Docker,
>> etc) on
>> >>> your computer" is, IMO, another barrier to entry. Yeah,
RMs will
>> have to
>> >>> have Maven installed and maybe Ant, but you should
already have
>> those
>> >>> installed to be a committer/PMC member.
>> >>>
>> >>> That said, a good takeaway from the Docker idea is to
try to find
>> a way
>> >> to
>> >>> make an "Image" of whatever we end up with on whatever
server we
>> end up
>> >>> using so if the image can be copied and used on other
servers.
>> I'm not
>> >>> exactly sure how to do that with Azure, which hosts my CI
>> server. I will
>> >>> spend a few more minutes researching that.
>> >>>
>> >>> I could not quickly find any way to get a free VM on
Azure or AWS
>> that
>> >>> isn't a free-trial-start-paying-after-a-year. So,
unless someone
>> comes
>> >> up
>> >>> with a free server we can use "forever", I'm going to
just start
>> with my
>> >>> Azure VM.
>> >>>
>> >>> -Alex
>> >>>
>> >>> On 2/5/19, 10:59 PM, "Carlos Rovira"
<carlosrov...@apache.org>
>> wrote:
>> >>>
>> >>> Hi.
>> >>>
>> >>> the plan sounds very good to me. Just my 2 thoughts
on this:
>> >>>
>> >>> 1.- As I was reading I was thinking as well on
something like
>> Docker
>> >>> and
>> >>> see Om as well thinking on the same. Maybe is the way
to this
>> with
>> >> the
>> >>> actual technology. Seems VMs are stepping out a bit
this days
>> in
>> >> favor
>> >>> of
>> >>> things like Docker. Maybe the same did Git over Svn,
and today
>> Svn is
>> >>> an
>> >>> old remembrance. I must say that I have no experience
with
>> Docker, so
>> >>> doing
>> >>> that will require acquire that knowledge, but seems
it could
>> be worth
>> >>> it.
>> >>>
>> >>> 2.- Maybe is not possible, but I want to propose to
do this
>> work I a
>> >>> separate branch, so it could be in parallel to other
>> developments. I
>> >>> think
>> >>> work over develop is practical if there's something
tiny that
>> could
>> >> be
>> >>> done
>> >>> in a commit. But as we need more than one, or is a
long
>> process (like
>> >>> this), chances are to make develop branch unstable
and even
>> for some
>> >>> days.
>> >>> I think we should try to avoid that scenario, and
branches are
>> the
>> >> best
>> >>> way. If we do this way, we'll benefit of more
reliable develop
>> >> branch.
>> >>>
>> >>> Thanks and good to know of this plan :)
>> >>>
>> >>> Carlos
>> >>>
>> >>>
>> >>> El mar., 5 feb. 2019 a las 23:19, Harbs (<
>> harbs.li...@gmail.com>)
>> >>> escribió:
>> >>>
>> >>>> I’ve never used Docker myself, but that might be a good
plan.
>> >>>>
>> >>>>> On Feb 6, 2019, at 12:07 AM, OmPrakash Muppirala <
>> >>> bigosma...@gmail.com>
>> >>>> wrote:
>> >>>>>
>> >>>>> I was wondering if we can use docker images to setup
and seal
>> the
>> >>> RM
>> >>>>> environment. Then other RMs simply need to run the
image
>> locally
>> >>> and run
>> >>>>> the release scripts. Might be easier. If folks like
this plan,
>> >> I
>> >>> can
>> >>>> try
>> >>>>> to put something together.
>> >>>>>
>> >>>>> Thanks,
>> >>>>> Om
>> >>>>>
>> >>>>> On Tue, Feb 5, 2019 at 1:40 PM Harbs
<harbs.li...@gmail.com>
>> >>> wrote:
>> >>>>>
>> >>>>>> In a recent discussion, it looks like other projects
have
>> gotten
>> >>>> resources
>> >>>>>> from AWS.
>> >>>>>>
>> >>>>>> Whatever service we use, could setup a “shared”
Royale account
>> >>> that all
>> >>>>>> PMC members could have access to.
>> >>>>>>
>> >>>>>> I don’t know if there’s some way we could leverage
Gitlab’s
>> >>> integration
>> >>>>>> pipelines
>> >>>
>> >>
>>
https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.gitlab.com%2Fee%2Fci%2FREADME.html&data=02%7C01%7Caharui%40adobe.com%7C9a4f659f80694687c3cd08d6a35b378d%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C636876009505602405&sdata=ZrqmL9%2B0oXasDm4aDW2q8K%2F%2BEkSc17zQerzJ4m%2BM7lU%3D&reserved=0
>> >>> <
>> >>>>>>
>> >>>
>> >>
>>
https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.gitlab.com%2Fee%2Fci%2FREADME.html&data=02%7C01%7Caharui%40adobe.com%7C9a4f659f80694687c3cd08d6a35b378d%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C636876009505602405&sdata=ZrqmL9%2B0oXasDm4aDW2q8K%2F%2BEkSc17zQerzJ4m%2BM7lU%3D&reserved=0
>> >>>>
>> >>>>>>
>> >>>>>>> On Feb 5, 2019, at 11:33 PM, Alex Harui
>> >> <aha...@adobe.com.INVALID
>> >>>>
>> >>>>>> wrote:
>> >>>>>>>
>> >>>>>>> Well, the big hole in this plan is that I think we
have to use
>> >>>> someone's
>> >>>>>> personal VM account (in this case, mine). I can't
think of a
>> >> way
>> >>> we can
>> >>>>>> run interactive commands like git push on builds@.
But that
>> >>> reminds me
>> >>>>>> to go see what are current options are for free/cheap
compute
>> >>> servers.
>> >>>>>>>
>> >>>>>>> On 2/5/19, 1:20 PM, "Piotr Zarzycki" <
>> >> piotrzarzyck...@gmail.com>
>> >>>> wrote:
>> >>>>>>>
>> >>>>>>> Sounds like best plan ever. Using the same PC by
everyone is
>> >>> awesome!
>> >>>>>>>
>> >>>>>>> On Tue, Feb 5, 2019, 8:39 PM Harbs
<harbs.li...@gmail.com
>> >>> wrote:
>> >>>>>>>
>> >>>>>>>> A big +1 from me!
>> >>>>>>>>
>> >>>>>>>> Looking forward!
>> >>>>>>>>
>> >>>>>>>>> On Feb 5, 2019, at 9:34 PM, Alex Harui
>> >>> <aha...@adobe.com.INVALID>
>> >>>>>> wrote:
>> >>>>>>>>>
>> >>>>>>>>> Hi,
>> >>>>>>>>>
>> >>>>>>>>> We are coming up on 3 months since 0.9.4. I have
finished
>> >> the
>> >>>> changes
>> >>>>>>>> to get production Royale modules to work in Tour De
Flex.
>> >> Lots
>> >>> of
>> >>>> other
>> >>>>>>>> good changes have been contributed.
>> >>>>>>>>>
>> >>>>>>>>> There were emails around the 0.9.4 release about
others
>> >>> stepping up
>> >>>> to
>> >>>>>>>> cut the next release, but that hasn't happened. I
tried and
>> >>> failed to
>> >>>>>> get
>> >>>>>>>> Apache Infra to allow us to run our release
packaging on the
>> >>> Jenkins
>> >>>>>>>> servers. They felt there were too many security
concerns
>> with
>> >>> having
>> >>>>>> the
>> >>>>>>>> servers push changes to Git and PGP sign artifacts.
>> >>>>>>>>>
>> >>>>>>>>> However, we MUST find a way for other RMs to be
successful.
>> >>> There is
>> >>>>>> no
>> >>>>>>>> way I should or want to be the only RM. But I have
an idea
>> >> that
>> >>>>>> involves
>> >>>>>>>> creating a long list of Jenkins jobs on my CI
server that add
>> >>> up to
>> >>>> the
>> >>>>>>>> release. The RM would log into Jenkins and run
some job
>> >> titled
>> >>>> "Apache
>> >>>>>>>> Royale Release Step 1", then wait for an email
indicating
>> that
>> >>> it
>> >>>>>> completed
>> >>>>>>>> and follow instructions in the email, such as
logging into
>> the
>> >>> CI
>> >>>> server
>> >>>>>>>> via Remote Desktop, opening a command prompt and
running "git
>> >>> push"
>> >>>> and
>> >>>>>>>> entering his/her username and password. Then run
the next
>> job
>> >>> and so
>> >>>>>> on.
>> >>>>>>>> There will be a point where the RM has to download
the build
>> >>>> artifacts,
>> >>>>>>>> verify them, then PGP sign them, and upload them.
That will
>> >> be
>> >>> a
>> >>>> likely
>> >>>>>>>> point of failure, but that step should be a single
Maven
>> >>> command and
>> >>>>>> thus
>> >>>>>>>> can be restarted until it finally succeeds. Then
more
>> Jenkins
>> >>> jobs
>> >>>>>> will be
>> >>>>>>>> run. But if this works then folks won't have to
setup their
>> >>> computers
>> >>>>>> to
>> >>>>>>>> be an RM.
>> >>>>>>>>>
>> >>>>>>>>> So, prepare for lots of commits and reverts as I
try to put
>> >>> this
>> >>>>>>>> together. I'm guessing it won't actually work
until the last
>> >>> week of
>> >>>>>>>> February at the earliest so there is no big rush to
get other
>> >>> stuff in
>> >>>>>> for
>> >>>>>>>> this release, and if we get it to work, hopefully
we'll
>> >> release
>> >>> more
>> >>>>>> often
>> >>>>>>>> with other folks being the RM.
>> >>>>>>>>>
>> >>>>>>>>> -Alex
>> >>>>>>>>>
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>
>> >>>>>>>
>> >>>>>>
>> >>>>>>
>> >>>>
>> >>>>
>> >>>
>> >>> --
>> >>> Carlos Rovira
>> >>>
>> >>>
>> >>
>>
https://nam04.safelinks.protection.outlook.com/?url=http%3A%2F%2Fabout.me%2Fcarlosrovira&data=02%7C01%7Caharui%40adobe.com%7C9a4f659f80694687c3cd08d6a35b378d%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C636876009505602405&sdata=egeH1lqxrifO9sWYd0PO67pKuQP6Riv0N2tEDf9rriA%3D&reserved=0
>> >>>
>> >>>
>> >>>
>> >>
>> >
>> >
>> > --
>> > Carlos Rovira
>> >
>>
https://nam04.safelinks.protection.outlook.com/?url=http%3A%2F%2Fabout.me%2Fcarlosrovira&data=02%7C01%7Caharui%40adobe.com%7C9a4f659f80694687c3cd08d6a35b378d%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C636876009505602405&sdata=egeH1lqxrifO9sWYd0PO67pKuQP6Riv0N2tEDf9rriA%3D&reserved=0
>> >
>> >
>>
>>
>>
>>
