Hope everyone had a good break. Just wanted to check if there were further thoughts on OSX builds. Chris, did you have time to look into virtualizing Mac OS? Would it make sense for us to put something in place in the interim e.g. the clang solution?
On Tue, Dec 12, 2017 at 7:59 PM, de Abreu, Marco <mab...@amazon.com> wrote: > Thanks for looking into this, Chris! No hurries on that one, we’ll look > into it next stage when we add new system- and build-configurations to the > CI. > > On 12.12.17, 19:12, "Chris Olivier" <cjolivie...@gmail.com> wrote: > > I am on vacation starting Thursday. > > On Tue, Dec 12, 2017 at 9:49 AM kellen sunderland < > kellen.sunderl...@gmail.com> wrote: > > > Absolutely, let's do an investigation and see if it's possible to > > virtualize. Would you have time to look into it a bit further? > > > > On Tue, Dec 12, 2017 at 6:47 PM, Chris Olivier < > cjolivie...@gmail.com> > > wrote: > > > > > Don’t get me wrong, I’m not saying this Mac OS Jenkins solution is > doable > > > but I feel like we should investigate because the payoff would be > large. > > > > > > > > > On Tue, Dec 12, 2017 at 9:38 AM Chris Olivier < > cjolivie...@gmail.com> > > > wrote: > > > > > > > Apple’s Darwin OS Is recently open-sourced. > > > > https://github.com/PureDarwin/PureDarwin > > > > > > > > How to convert this into a non-GUI VM I am not sure but I am > willing to > > > > bet that people have done it already. > > > > > > > > On Tue, Dec 12, 2017 at 9:16 AM kellen sunderland < > > > > kellen.sunderl...@gmail.com> wrote: > > > > > > > >> It might be technically possible, but I think it would violate > the > > MacOS > > > >> license: http://store.apple.com/Catalog/US/Images/MacOSX.htm > > > >> > > > >> "2. Permitted License Uses and Restrictions. > > > >> A. This License allows you to install and use one copy of the > Apple > > > >> Software on a single Apple-labeled computer at a time. This > License > > does > > > >> not allow the Apple Software to exist on more than one computer > at a > > > >> time,and you may not make the Apple Software available over a > network > > > >> where > > > >> it could be used by multiple computers at the same time. You > may make > > > one > > > >> copy of the Apple Software (excluding the Boot ROM code) in > > > >> machine-readable form for backup purposes only; provided that > the > > backup > > > >> copy must include all copyright or other proprietary notices > contained > > > on > > > >> the original. " > > > >> > > > >> I could be wrong though, does anyone know the details of MacOS > > > licensing / > > > >> virtualization? > > > >> > > > >> On Tue, Dec 12, 2017 at 6:10 PM, Chris Olivier < > cjolivie...@gmail.com > > > > > > >> wrote: > > > >> > > > >> > googling seems to be full of running OSX (and even > open-sourced > > > >> PureDarwin) > > > >> > in VMs. One could conceivably run a VM on an EC2 instance, > right? > > > >> > > > > >> > On Tue, Dec 12, 2017 at 9:01 AM kellen sunderland < > > > >> > kellen.sunderl...@gmail.com> wrote: > > > >> > > > > >> > > It would be ideal if we could cover OSX in Jenkins, but the > only > > > >> solution > > > >> > > that I'm aware of would require physical machines to be the > > workers. > > > >> I > > > >> > > would be weakly opposed to having physical servers running > on PRs. > > > >> The > > > >> > > downsides that I see in order of importance: > > > >> > > > > > >> > > - We can't autoscale physical hardware. If we find that > the > > load > > > is > > > >> > too > > > >> > > high we have to buy more machines. > > > >> > > - Security would be tricky, as they'd have to be connected > to the > > > >> > internet > > > >> > > and then to our Jekins master instance. Connecting via a > wired > > > >> network > > > >> > > would probably not be possible on most corporate networks > as these > > > >> > machines > > > >> > > are by definition running arbitrary code from the > internet. Many > > > >> > corporate > > > >> > > sites have public wifi that this machine could potentially > connect > > > to, > > > >> > but > > > >> > > then our PRs start failing if the wifi disconnects > temporarily. > > To > > > >> > connect > > > >> > > to the master we would need to setup a vpn solution with > endpoints > > > in > > > >> our > > > >> > > vpc on AWS. This is possible but would probably require a > lot of > > > >> > security > > > >> > > work. > > > >> > > - We can't just create a simple startup script or yaml > file that > > is > > > >> > > checked into GitHub to manage the machine. Someone will > actually > > > >> have to > > > >> > > physically administer the machine, apply updates, etc. > which will > > > make > > > >> > > community ownership difficult. > > > >> > > > > > >> > > Specific to an OSX build: > > > >> > > - We can't virtualize OSX which means we'd only be able to > cover > > > one > > > >> OSX > > > >> > > build environment per physical device. We couldn't target a > > matrix > > > of > > > >> > OSX > > > >> > > and Xcode versions as in Travis. > > > >> > > > > > >> > > -Kellen > > > >> > > > > > >> > > On Tue, Dec 12, 2017 at 5:46 PM, Chris Olivier < > > > cjolivie...@gmail.com > > > >> > > > > >> > > wrote: > > > >> > > > > > >> > > > So why Travis when we could possibly use Jenkins? > > > >> > > > > > > >> > > > On Tue, Dec 12, 2017 at 7:59 AM Marco de Abreu < > > > >> > > > marco.g.ab...@googlemail.com> > > > >> > > > wrote: > > > >> > > > > > > >> > > > > Yes that's correct, Chris. > > > >> > > > > > > > >> > > > > Am 12.12.2017 4:46 nachm. schrieb "Chris Olivier" < > > > >> > > cjolivie...@gmail.com > > > >> > > > >: > > > >> > > > > > > > >> > > > > > A quick google search seems to indicate that Mac can > be used > > > as > > > >> a > > > >> > > > Jenkins > > > >> > > > > > slave. Is this correct? > > > >> > > > > > > > > >> > > > > > On Tue, Dec 12, 2017 at 7:42 AM Steffen Rochel < > > > >> > > > steffenroc...@gmail.com> > > > >> > > > > > wrote: > > > >> > > > > > > > > >> > > > > > > +1 for #1 and #2 > > > >> > > > > > > > > > >> > > > > > > I’m working on getting a MacPro to add to CI system. > > > >> > > > > > > On Tue, Dec 12, 2017 at 1:43 AM kellen sunderland < > > > >> > > > > > > kellen.sunderl...@gmail.com> wrote: > > > >> > > > > > > > > > >> > > > > > > > Background: TravisCI is a startup providing > managed > > > >> continuous > > > >> > > > > > > > integration services with GitHub integration and > YAML > > > based > > > >> > > > > > > configuration. > > > >> > > > > > > > TravisCI is one of the few CI providers that will > build > > a > > > >> > variety > > > >> > > > of > > > >> > > > > > > > OSX/MacOS builds for software projects. Their > pricing > > > >> ranges > > > >> > > from > > > >> > > > > Free > > > >> > > > > > > > (for open source, 1 concurrent job, to $489 > monthly for > > 10 > > > >> > > > concurrent > > > >> > > > > > > jobs). > > > >> > > > > > > > > > > >> > > > > > > > Problem: We’ve had a few OSX build issues slip > into > > MXNet > > > >> > master > > > >> > > in > > > >> > > > > the > > > >> > > > > > > > past few weeks. We’ve previously had a Travis CI > based > > > >> testing > > > >> > > > > system > > > >> > > > > > > that > > > >> > > > > > > > would have caught these issues. > > > >> > > > > > > > > > > >> > > > > > > > Proposals so far: > > > >> > > > > > > > > > > >> > > > > > > > 1) Use TravisCI in it’s free mode for a very > minimal > > > sanity > > > >> > check > > > >> > > > on > > > >> > > > > > OSX. > > > >> > > > > > > > If we compile the program, and for example run > C++ unit > > > >> tests > > > >> > > we’re > > > >> > > > > > > > unlikely to run into problems with queued > builds. The > > > total > > > >> > > build > > > >> > > > > time > > > >> > > > > > > > here should be less than 15 minutes. > Configuration > > should > > > >> be > > > >> > > quite > > > >> > > > > > > simple > > > >> > > > > > > > and easy to maintain. Error messages should also > be > > > >> obvious to > > > >> > > > > > > > contributors. > > > >> > > > > > > > 2) Run clang in Linux with our current CI. > Building > > with > > > >> clang > > > >> > > > > should > > > >> > > > > > > > take less than 10 minutes, should flush out a > large > > subset > > > >> of > > > >> > the > > > >> > > > > > issues > > > >> > > > > > > > we’ve seen with OSX, and be quite easy to > maintain. > > > >> > > > > > > > 3) Run full test-suites in TravisCI, equaling the > level > > of > > > >> > > coverage > > > >> > > > > we > > > >> > > > > > > > provide to Linux in Jenkins. This could require > us to > > > >> > subscribe > > > >> > > > to a > > > >> > > > > > > > monthly package with Travis to ensure our build > queue > > > >> doesn’t > > > >> > > grow > > > >> > > > to > > > >> > > > > > an > > > >> > > > > > > > unacceptable length. It may also require a > volunteer to > > > >> setup > > > >> > > and > > > >> > > > > > > maintain > > > >> > > > > > > > long-term. > > > >> > > > > > > > > > > >> > > > > > > > I’d +1 #1 and #2 as I think those should be > low-cost, > > > >> > > low-maintence > > > >> > > > > > > > solutions that should catch the majority of the > problems > > > >> we’ve > > > >> > > seen > > > >> > > > > > thus > > > >> > > > > > > > far. > > > >> > > > > > > > > > > >> > > > > > > > -Kellen > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > > > > > > > > > >
