Kellen has fixed the one bug in our build system and thus, there are no outstanding tests :)
Exactly, it will run on branch and PR validation. Best regards, Marco sandeep krishnamurthy <sandeep.krishn...@gmail.com> schrieb am Di., 18. Sep. 2018, 19:32: > This is awesome. Thanks a lot Kellen and Marco. With this work complete, we > will have MXNet Python tests running for Mac on Travis CI, for PR and > Branch builds? > Thank you for working on fixing the tests and making it run as part of > Travis CI for Mac platform. Is there any Github issue or Jira where we can > see disabled / tests that needs to be fixed for Mac? This might be useful > if we can call for contributions. > > Best, > Sandeep > > > On Tue, Sep 18, 2018 at 9:51 AM Marco de Abreu > <marco.g.ab...@googlemail.com.invalid> wrote: > > > Hey everyone, > > > > we are about to enable Python tests for Mac. The outstanding bugs have > been > > fixed by Kellen and we're just waiting for the PRs to pass. We'll send a > > separate email as soon as they are enabled. > > > > Additionally, we had a small problem that Travis runs got aborted if > > multiple commits were done in a short timeframe. While this is acceptable > > for PRs, this causes our branch jobs to also fail. An examples is > available > > at [1]. In order to cope with this, I have asked Apache Infra to disable > > cancellation of concurrent jobs. They agreed to this, but reminded us > that > > they might turn it back on if we consume too many resources. > > > > The dashboard to review the Travis resource utilization is available at > > [2]. Just log in as Guest. > > > > Best regards, > > Marco > > > > [1]: > > > > > https://travis-ci.org/apache/incubator-mxnet/builds/430135867?utm_source=github_status&utm_medium=notification > > [2]: > > > > > https://demo.kibble.apache.org/dashboard.html?page=ci&source=e0ce4eee89a77ec231eee1fdbbc647cb3de2f6ecfc3cef8d8c11dc2d&interval=hour > > > > > > On Thu, Sep 13, 2018 at 1:06 AM kellen sunderland < > > kellen.sunderl...@gmail.com> wrote: > > > > > We've got fairly limited ability to change what's reported by Travis. > > Most > > > administration is done by the ASF Infra crew, so it's tough for us to > > > experiment with settings. It'd be great if you could bear with us for > a > > > few days. It shouldn't take too long to either (1) get happy-feeling > > green > > > checks back, or (2) decide we don't care as much as we thought we did > > about > > > MacOS support. > > > > > > On Wed, Sep 12, 2018 at 9:53 PM Aaron Markham < > aaron.s.mark...@gmail.com > > > > > > wrote: > > > > > > > Is there any way to make it not show a red X failure in the GitHub UI > > > when > > > > TravisCI fails? I keep going back to check what flakey test failed > this > > > > time and realizing that Jenkins is still running and it was the "not > > > > required" Travis fail. The green checkmark makes me happy and it's > > easier > > > > to keep an eye on what's going on. If Travis times out a lot of the > > time, > > > > then most of our PRs will look red/bad/sad when they're not. > > > > > > > > What about no failure flag set, but add a label that Travis > failed.... > > or > > > > if we can't control the flag, auto-set labels for each Travis and > > Jenkins > > > > pass/fail so we still get the benefit of at-a-glance status checks. > > > > > > > > On Wed, Sep 12, 2018 at 6:04 AM Marco de Abreu > > > > <marco.g.ab...@googlemail.com.invalid> wrote: > > > > > > > > > Hello, > > > > > > > > > > Travis CI has successfully been enabled just now. This means you > will > > > now > > > > > see a new status under your PR which is called > > > > > "continuous-integration/travis-ci/pr". > > > > > > > > > > The job only compiles MXNet on Mac and currently does not run unit > > > tests > > > > - > > > > > we expect the overall execution duration to be around 6 minutes and > > > thus > > > > > faster than the full Jenkins pipeline. The status is set to "not > > > > required" > > > > > which means that it does not block merging if that job fails since > > the > > > > > pipeline is still in beta. But in general, it would be good if > > > committers > > > > > review the results in case the job shows a failure. Our last known > > > state > > > > is > > > > > that the pipeline works properly, but we will keep everybody up to > > date > > > > in > > > > > case we get aware of any problems. > > > > > > > > > > The next step will be integration of Python CPU unit tests. There > > will > > > > be a > > > > > separate email if we got an update on that manner. > > > > > > > > > > Special thanks to Kellen Sunderland for the contribution of this > > Travis > > > > CI > > > > > pipeline. > > > > > > > > > > Best regards, > > > > > Marco > > > > > > > > > > On Wed, Sep 5, 2018 at 8:19 PM Tianqi Chen < > tqc...@cs.washington.edu > > > > > > > > wrote: > > > > > > > > > > > Alrite, then I think it is fine as long as we can kept up with > > build > > > > > speed > > > > > > without timeout. > > > > > > > > > > > > > > > > > > Tianqi > > > > > > > > > > > > On Wed, Sep 5, 2018 at 9:14 AM kellen sunderland < > > > > > > kellen.sunderl...@gmail.com> wrote: > > > > > > > > > > > > > Travis actually has explicit support for ccache, it's a > platform > > > > > feature. > > > > > > > I've run it and it seems to work quite well. See for example > > this > > > > > build: > > > > > > > > > > > > > > > > https://travis-ci.org/KellenSunderland/incubator-mxnet/builds/424768656 > > > > > > > > > > > > > > On Wed, Sep 5, 2018 at 7:10 PM Tianqi Chen < > > > tqc...@cs.washington.edu > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > Travis it self is stateless, which means ccache is not likely > > > going > > > > > to > > > > > > > > work. As far as I understand, if jenkins master is in the > > public > > > > > > domain, > > > > > > > > you do not need to setup a vpn to the subset of the master. > > > > > > > > > > > > > > > > As for versions of MacOS, we are likely going to be fine with > > one > > > > > > > version, > > > > > > > > as usually the problems exhibits on mac are similar > > > > > > > > > > > > > > > > Tianqi > > > > > > > > On Wed, Sep 5, 2018 at 9:04 AM kellen sunderland < > > > > > > > > kellen.sunderl...@gmail.com> wrote: > > > > > > > > > > > > > > > > > @Tianqi: Yeah there's going to be a lot of trade-offs to > > using > > > > > > > Travis. I > > > > > > > > > hope we can get it running fast enough with ccache that it > > > won't > > > > > > > timeout > > > > > > > > > when running tests, but even that is questionable. In my > > > private > > > > > > > testing > > > > > > > > > it was running in about 35 minutes and the global timeout > for > > > > > Travis > > > > > > > jobs > > > > > > > > > is 45 minutes. I'd say let's run it for a few builds and > see > > > how > > > > > it > > > > > > > > goes. > > > > > > > > > It won't be enabled in a mode that blocks PRs any time > soon. > > > > > > > > > > > > > > > > > > I don't think physical hardware is a great solution. We > > would > > > > have > > > > > > to > > > > > > > > > purchase the hardware, then maintain security updates, > > install > > > > > > > different > > > > > > > > > versions of XCode / MacOS, setup a vpn to our jenkins > master, > > > > > etc. I > > > > > > > > would > > > > > > > > > also worry that if the machine goes down for whatever > reason > > it > > > > > would > > > > > > > > block > > > > > > > > > PRs, and someone would have to be physically present to > turn > > it > > > > > back > > > > > > > on. > > > > > > > > > Even assuming we set all the hardware up it's still not > > > scalable > > > > so > > > > > > > we'd > > > > > > > > > have to over-provision. > > > > > > > > > > > > > > > > > > I'm hoping the Travis solution works for the time being. If > > it > > > > > > doesn't > > > > > > > > > we'll have to take a look at a few other options, but I've > > > spent > > > > a > > > > > > fair > > > > > > > > > amount of time thinking about this and I don't think there > > are > > > > any > > > > > > good > > > > > > > > > options that don't have trade-offs. > > > > > > > > > > > > > > > > > > @Lin: Great! Thanks for the offer. There'll be a few > > features > > > > we > > > > > > want > > > > > > > > to > > > > > > > > > re-enable once the Job gets hooked up again. I'll ping you > > > when > > > > > it's > > > > > > > > ready > > > > > > > > > and see if there's anything you think would be interesting > to > > > > help > > > > > > > with. > > > > > > > > > > > > > > > > > > -Kellen > > > > > > > > > > > > > > > > > > On Wed, Sep 5, 2018 at 6:58 PM Lin Yuan < > apefor...@gmail.com > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > Hi Kellen, > > > > > > > > > > > > > > > > > > > > I would love to contribute. Please let me know if you > have > > > any > > > > > > > > particular > > > > > > > > > > work item that I can help. > > > > > > > > > > > > > > > > > > > > Best, > > > > > > > > > > > > > > > > > > > > Lin > > > > > > > > > > > > > > > > > > > > On Wed, Sep 5, 2018 at 9:51 AM Tianqi Chen < > > > > > > tqc...@cs.washington.edu > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > is it possible for us to get a MacBook and hook it to > the > > > > > current > > > > > > > > > Jenkins > > > > > > > > > > > CI? Travis OSX usually build from scratch and that was > > > pretty > > > > > > slow > > > > > > > > > > > > > > > > > > > > > > Tianqi > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Sep 5, 2018 at 8:49 AM kellen sunderland < > > > > > > > > > > > kellen.sunderl...@gmail.com> wrote: > > > > > > > > > > > > > > > > > > > > > > > Great you feel that way Lin, please feel free to > > > contribute > > > > > if > > > > > > > you > > > > > > > > > have > > > > > > > > > > > any > > > > > > > > > > > > features you'd like tested. We are using the travis > > > image > > > > > > > xcode9.4 > > > > > > > > > > which > > > > > > > > > > > > is based on MacOS 10.13. > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Sep 5, 2018 at 6:40 PM Lin Yuan < > > > > apefor...@gmail.com > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > Hi Kellen, > > > > > > > > > > > > > > > > > > > > > > > > > > Many thanks for your and Marco's effort! I think > this > > > is > > > > a > > > > > > very > > > > > > > > > > crucial > > > > > > > > > > > > > piece to improve MXNet stability. > > > > > > > > > > > > > > > > > > > > > > > > > > To add some data points: > > > > > > > > > > > > > 1) Customers using CoreML to MXNet converter were > > > blocked > > > > > > for a > > > > > > > > > while > > > > > > > > > > > > > because the converter was broken and no unit test > was > > > in > > > > > > place > > > > > > > to > > > > > > > > > > > detect > > > > > > > > > > > > > that. > > > > > > > > > > > > > 2) Developers on Mac cannot verify their local > > commits > > > > > > because > > > > > > > > some > > > > > > > > > > > unit > > > > > > > > > > > > > tests on master were broken. This wasted much time > > and > > > > > > resource > > > > > > > > on > > > > > > > > > > > > jenkins > > > > > > > > > > > > > server to detect the failure. > > > > > > > > > > > > > 3) Please consider running the CI on Mac OS 10.13 > > since > > > > > this > > > > > > is > > > > > > > > the > > > > > > > > > > > > minimum > > > > > > > > > > > > > Mac OS version that supports CoreML (to support > > CoreML > > > to > > > > > > MXNet > > > > > > > > > > > > converter) > > > > > > > > > > > > > > > > > > > > > > > > > > Best Regards, > > > > > > > > > > > > > > > > > > > > > > > > > > Lin > > > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Sep 5, 2018, 3:02 AM kellen sunderland < > > > > > > > > > > > > > kellen.sunderl...@gmail.com> > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > I'm bumping this thread as we've recently had our > > > first > > > > > > > serious > > > > > > > > > bug > > > > > > > > > > > on > > > > > > > > > > > > > > MacOS that would have been caught by enabling > > Travis. > > > > > > > > > > > > > > > > > > > > > > > > > > > > I'm going to do a little experimental work > together > > > > with > > > > > > > Marco > > > > > > > > > with > > > > > > > > > > > the > > > > > > > > > > > > > > goal of enabling a minimal Travis build that will > > run > > > > > > python > > > > > > > > > tests. > > > > > > > > > > > So > > > > > > > > > > > > > far > > > > > > > > > > > > > > I've verified that Travis will in fact find a bug > > > that > > > > > > > > currently > > > > > > > > > > > exists > > > > > > > > > > > > > in > > > > > > > > > > > > > > master and has been reproduced by MacOS clients. > > > This > > > > > > > > indicates > > > > > > > > > to > > > > > > > > > > > me > > > > > > > > > > > > > that > > > > > > > > > > > > > > adding Travis will add value to our CI. > > > > > > > > > > > > > > > > > > > > > > > > > > > > My best guess is that it might take us some > > iteration > > > > > > before > > > > > > > we > > > > > > > > > > find > > > > > > > > > > > a > > > > > > > > > > > > > > scalable way to integrate Travis. Given this > we're > > > > going > > > > > > to > > > > > > > > > enable > > > > > > > > > > > > > Travis > > > > > > > > > > > > > > in non-blocking mode (i.e. failures are safe to > > > ignore > > > > > for > > > > > > > the > > > > > > > > > time > > > > > > > > > > > > > being). > > > > > > > > > > > > > > > > > > > > > > > > > > > > To help mitigate the risk of timeouts, and to > > remove > > > > > legacy > > > > > > > > code > > > > > > > > > > I'm > > > > > > > > > > > > > going > > > > > > > > > > > > > > to re-create the travis.yml file from scratch. I > > > think > > > > > > it'll > > > > > > > > be > > > > > > > > > > much > > > > > > > > > > > > > less > > > > > > > > > > > > > > confusing if we only have working code related to > > > > Travis > > > > > in > > > > > > > our > > > > > > > > > > > > codebase, > > > > > > > > > > > > > > so that contributors won't have to experiment to > > see > > > > what > > > > > > is > > > > > > > or > > > > > > > > > > isn't > > > > > > > > > > > > > > working. We've got some great, but slightly > > > > out-of-date > > > > > > > > > > > functionality > > > > > > > > > > > > in > > > > > > > > > > > > > > the legacy .travis.yml file. I hope we can work > > > > together > > > > > > to > > > > > > > > > update > > > > > > > > > > > the > > > > > > > > > > > > > > legacy features, ensure they work with the > current > > > > folder > > > > > > > > > structure > > > > > > > > > > > and > > > > > > > > > > > > > > also make sure the features run within Travis's > 45 > > > > minute > > > > > > > > global > > > > > > > > > > time > > > > > > > > > > > > > > window. > > > > > > > > > > > > > > > > > > > > > > > > > > > > I'd also like to set expectations that this is > > > > strictly a > > > > > > > > > volunteer > > > > > > > > > > > > > > effort. I'd welcome help from the community for > > > > support > > > > > > and > > > > > > > > > > > > maintenance. > > > > > > > > > > > > > > The model downloading caching work particularly > > > stands > > > > > out > > > > > > to > > > > > > > > me > > > > > > > > > as > > > > > > > > > > > > > > something I'd like to re-enable again as soon as > > > > > possible. > > > > > > > > > > > > > > > > > > > > > > > > > > > > -Kellen > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Jan 9, 2018 at 11:52 AM Marco de Abreu < > > > > > > > > > > > > > > marco.g.ab...@googlemail.com> > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Looks good! +1 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Jan 9, 2018 at 10:24 AM, kellen > > sunderland > > > < > > > > > > > > > > > > > > > kellen.sunderl...@gmail.com> wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I think most were in favour of at a minimum > > > > creating > > > > > a > > > > > > > > clang > > > > > > > > > > > build > > > > > > > > > > > > so > > > > > > > > > > > > > > > I've > > > > > > > > > > > > > > > > created a PR > > > > > > > > > > > > > > > > > > > > > > > > https://github.com/apache/incubator-mxnet/pull/9330/commits/ > > > > > > > > > > > > > > > > 84089ea14123ebe4d66cc92e82a2d529cfbd8b19. > > > > > > > > > > > > > > > > My hope is this will catch many of the issues > > > > > blocking > > > > > > > OSX > > > > > > > > > > > builds. > > > > > > > > > > > > > In > > > > > > > > > > > > > > > fact > > > > > > > > > > > > > > > > it already caught one issue. If you guys are > > in > > > > > > favour I > > > > > > > > can > > > > > > > > > > > > remove > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > WIP and ask that it be merged. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Jan 4, 2018 at 6:29 PM, Chris > Olivier < > > > > > > > > > > > > cjolivie...@gmail.com > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Nope, I have been on vacation. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Jan 4, 2018 at 9:10 AM, kellen > > > > sunderland < > > > > > > > > > > > > > > > > > kellen.sunderl...@gmail.com> wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hope everyone had a good break. Just > > wanted > > > to > > > > > > check > > > > > > > > if > > > > > > > > > > > there > > > > > > > > > > > > > were > > > > > > > > > > > > > > > > > further > > > > > > > > > > > > > > > > > > thoughts on OSX builds. Chris, did you > > have > > > > time > > > > > > to > > > > > > > > look > > > > > > > > > > > into > > > > > > > > > > > > > > > > > virtualizing > > > > > > > > > > > > > > > > > > Mac OS? Would it make sense for us to > put > > > > > > something > > > > > > > in > > > > > > > > > > place > > > > > > > > > > > > in > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > interim e.g. the clang solution? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Dec 12, 2017 at 7:59 PM, de > Abreu, > > > > Marco > > > > > < > > > > > > > > > > > > > > mab...@amazon.com> > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks for looking into this, Chris! No > > > > hurries > > > > > > on > > > > > > > > that > > > > > > > > > > > one, > > > > > > > > > > > > > > we’ll > > > > > > > > > > > > > > > > look > > > > > > > > > > > > > > > > > > > into it next stage when we add new > > system- > > > > and > > > > > > > > > > > > > > build-configurations > > > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > CI. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On 12.12.17, 19:12, "Chris Olivier" < > > > > > > > > > > cjolivie...@gmail.com > > > > > > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I am on vacation starting Thursday. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Dec 12, 2017 at 9:49 AM > > kellen > > > > > > > > sunderland < > > > > > > > > > > > > > > > > > > > kellen.sunderl...@gmail.com> > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Absolutely, let's do an > > investigation > > > > and > > > > > > see > > > > > > > > if > > > > > > > > > > it's > > > > > > > > > > > > > > > possible > > > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > > > > virtualize. Would you have time > to > > > > look > > > > > > into > > > > > > > > it > > > > > > > > > a > > > > > > > > > > > bit > > > > > > > > > > > > > > > further? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Dec 12, 2017 at 6:47 PM, > > > Chris > > > > > > > Olivier > > > > > > > > < > > > > > > > > > > > > > > > > > > > cjolivie...@gmail.com> > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Don’t get me wrong, I’m not > > saying > > > > this > > > > > > Mac > > > > > > > > OS > > > > > > > > > > > > Jenkins > > > > > > > > > > > > > > > > solution > > > > > > > > > > > > > > > > > > is > > > > > > > > > > > > > > > > > > > doable > > > > > > > > > > > > > > > > > > > > > but I feel like we should > > > investigate > > > > > > > because > > > > > > > > > the > > > > > > > > > > > > > payoff > > > > > > > > > > > > > > > > would > > > > > > > > > > > > > > > > > be > > > > > > > > > > > > > > > > > > > large. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Dec 12, 2017 at 9:38 AM > > > Chris > > > > > > > > Olivier < > > > > > > > > > > > > > > > > > > > cjolivie...@gmail.com> > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Apple’s Darwin OS Is recently > > > > > > > open-sourced. > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://github.com/PureDarwin/PureDarwin > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > How to convert this into a > > > non-GUI > > > > > VM I > > > > > > > am > > > > > > > > > not > > > > > > > > > > > sure > > > > > > > > > > > > > > but I > > > > > > > > > > > > > > > > am > > > > > > > > > > > > > > > > > > > willing to > > > > > > > > > > > > > > > > > > > > > > bet that people have done it > > > > already. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Dec 12, 2017 at 9:16 > AM > > > > > kellen > > > > > > > > > > > sunderland < > > > > > > > > > > > > > > > > > > > > > > kellen.sunderl...@gmail.com> > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> It might be technically > > > possible, > > > > > but > > > > > > I > > > > > > > > > think > > > > > > > > > > it > > > > > > > > > > > > > would > > > > > > > > > > > > > > > > > violate > > > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > MacOS > > > > > > > > > > > > > > > > > > > > > >> license: > > > http://store.apple.com/ > > > > > > > > > > > > > > > > > Catalog/US/Images/MacOSX.htm > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >> "2. Permitted License Uses > and > > > > > > > > Restrictions. > > > > > > > > > > > > > > > > > > > > > >> A. This License allows you > to > > > > > install > > > > > > > and > > > > > > > > > use > > > > > > > > > > > one > > > > > > > > > > > > > copy > > > > > > > > > > > > > > > of > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > Apple > > > > > > > > > > > > > > > > > > > > > >> Software on a single > > > Apple-labeled > > > > > > > > computer > > > > > > > > > > at a > > > > > > > > > > > > > time. > > > > > > > > > > > > > > > > This > > > > > > > > > > > > > > > > > > > License > > > > > > > > > > > > > > > > > > > > does > > > > > > > > > > > > > > > > > > > > > >> not allow the Apple Software > > to > > > > > exist > > > > > > on > > > > > > > > > more > > > > > > > > > > > than > > > > > > > > > > > > > one > > > > > > > > > > > > > > > > > > computer > > > > > > > > > > > > > > > > > > > at a > > > > > > > > > > > > > > > > > > > > > >> time,and you may not make > the > > > > Apple > > > > > > > > Software > > > > > > > > > > > > > available > > > > > > > > > > > > > > > > over > > > > > > > > > > > > > > > > > a > > > > > > > > > > > > > > > > > > > network > > > > > > > > > > > > > > > > > > > > > >> where > > > > > > > > > > > > > > > > > > > > > >> it could be used by multiple > > > > > computers > > > > > > > at > > > > > > > > > the > > > > > > > > > > > same > > > > > > > > > > > > > > time. > > > > > > > > > > > > > > > > You > > > > > > > > > > > > > > > > > > > may make > > > > > > > > > > > > > > > > > > > > > one > > > > > > > > > > > > > > > > > > > > > >> copy of the Apple Software > > > > > (excluding > > > > > > > the > > > > > > > > > Boot > > > > > > > > > > > ROM > > > > > > > > > > > > > > code) > > > > > > > > > > > > > > > > in > > > > > > > > > > > > > > > > > > > > > >> machine-readable form for > > backup > > > > > > > purposes > > > > > > > > > > only; > > > > > > > > > > > > > > provided > > > > > > > > > > > > > > > > > that > > > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > backup > > > > > > > > > > > > > > > > > > > > > >> copy must include all > > copyright > > > or > > > > > > other > > > > > > > > > > > > proprietary > > > > > > > > > > > > > > > > notices > > > > > > > > > > > > > > > > > > > contained > > > > > > > > > > > > > > > > > > > > > on > > > > > > > > > > > > > > > > > > > > > >> the original. " > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >> I could be wrong though, > does > > > > anyone > > > > > > > know > > > > > > > > > the > > > > > > > > > > > > > details > > > > > > > > > > > > > > of > > > > > > > > > > > > > > > > > MacOS > > > > > > > > > > > > > > > > > > > > > licensing / > > > > > > > > > > > > > > > > > > > > > >> virtualization? > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >> On Tue, Dec 12, 2017 at 6:10 > > PM, > > > > > Chris > > > > > > > > > > Olivier < > > > > > > > > > > > > > > > > > > > cjolivie...@gmail.com > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> wrote: > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >> > googling seems to be full > of > > > > > running > > > > > > > OSX > > > > > > > > > > (and > > > > > > > > > > > > even > > > > > > > > > > > > > > > > > > > open-sourced > > > > > > > > > > > > > > > > > > > > > >> PureDarwin) > > > > > > > > > > > > > > > > > > > > > >> > in VMs. One could > > conceivably > > > > run > > > > > a > > > > > > VM > > > > > > > > on > > > > > > > > > an > > > > > > > > > > > EC2 > > > > > > > > > > > > > > > > instance, > > > > > > > > > > > > > > > > > > > right? > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > >> > On Tue, Dec 12, 2017 at > 9:01 > > > AM > > > > > > kellen > > > > > > > > > > > > sunderland > > > > > > > > > > > > > < > > > > > > > > > > > > > > > > > > > > > >> > > kellen.sunderl...@gmail.com > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > >> > > It would be ideal if we > > > could > > > > > > cover > > > > > > > > OSX > > > > > > > > > in > > > > > > > > > > > > > > Jenkins, > > > > > > > > > > > > > > > > but > > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > only > > > > > > > > > > > > > > > > > > > > > >> solution > > > > > > > > > > > > > > > > > > > > > >> > > that I'm aware of would > > > > require > > > > > > > > physical > > > > > > > > > > > > > machines > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > be > > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > workers. > > > > > > > > > > > > > > > > > > > > > >> I > > > > > > > > > > > > > > > > > > > > > >> > > would be weakly opposed > to > > > > > having > > > > > > > > > physical > > > > > > > > > > > > > servers > > > > > > > > > > > > > > > > > running > > > > > > > > > > > > > > > > > > > on PRs. > > > > > > > > > > > > > > > > > > > > > >> The > > > > > > > > > > > > > > > > > > > > > >> > > downsides that I see in > > > order > > > > of > > > > > > > > > > importance: > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > >> > > - We can't autoscale > > > physical > > > > > > > > hardware. > > > > > > > > > > > If > > > > > > > > > > > > we > > > > > > > > > > > > > > > find > > > > > > > > > > > > > > > > > that > > > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > load > > > > > > > > > > > > > > > > > > > > > is > > > > > > > > > > > > > > > > > > > > > >> > too > > > > > > > > > > > > > > > > > > > > > >> > > high we have to buy more > > > > > machines. > > > > > > > > > > > > > > > > > > > > > >> > > - Security would be > > tricky, > > > > as > > > > > > > they'd > > > > > > > > > > have > > > > > > > > > > > to > > > > > > > > > > > > > be > > > > > > > > > > > > > > > > > > connected > > > > > > > > > > > > > > > > > > > to the > > > > > > > > > > > > > > > > > > > > > >> > internet > > > > > > > > > > > > > > > > > > > > > >> > > and then to our Jekins > > > master > > > > > > > > instance. > > > > > > > > > > > > > > Connecting > > > > > > > > > > > > > > > > via > > > > > > > > > > > > > > > > > a > > > > > > > > > > > > > > > > > > > wired > > > > > > > > > > > > > > > > > > > > > >> network > > > > > > > > > > > > > > > > > > > > > >> > > would probably not be > > > possible > > > > > on > > > > > > > most > > > > > > > > > > > > corporate > > > > > > > > > > > > > > > > > networks > > > > > > > > > > > > > > > > > > > as these > > > > > > > > > > > > > > > > > > > > > >> > machines > > > > > > > > > > > > > > > > > > > > > >> > > are by definition > running > > > > > > arbitrary > > > > > > > > code > > > > > > > > > > > from > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > internet. Many > > > > > > > > > > > > > > > > > > > > > >> > corporate > > > > > > > > > > > > > > > > > > > > > >> > > sites have public wifi > > that > > > > this > > > > > > > > machine > > > > > > > > > > > could > > > > > > > > > > > > > > > > > potentially > > > > > > > > > > > > > > > > > > > connect > > > > > > > > > > > > > > > > > > > > > to, > > > > > > > > > > > > > > > > > > > > > >> > but > > > > > > > > > > > > > > > > > > > > > >> > > then our PRs start > failing > > > if > > > > > the > > > > > > > wifi > > > > > > > > > > > > > disconnects > > > > > > > > > > > > > > > > > > > temporarily. > > > > > > > > > > > > > > > > > > > > To > > > > > > > > > > > > > > > > > > > > > >> > connect > > > > > > > > > > > > > > > > > > > > > >> > > to the master we would > > need > > > to > > > > > > > setup a > > > > > > > > > vpn > > > > > > > > > > > > > > solution > > > > > > > > > > > > > > > > with > > > > > > > > > > > > > > > > > > > endpoints > > > > > > > > > > > > > > > > > > > > > in > > > > > > > > > > > > > > > > > > > > > >> our > > > > > > > > > > > > > > > > > > > > > >> > > vpc on AWS. This is > > > possible > > > > > but > > > > > > > > would > > > > > > > > > > > > probably > > > > > > > > > > > > > > > > > require a > > > > > > > > > > > > > > > > > > > lot of > > > > > > > > > > > > > > > > > > > > > >> > security > > > > > > > > > > > > > > > > > > > > > >> > > work. > > > > > > > > > > > > > > > > > > > > > >> > > - We can't just create > a > > > > simple > > > > > > > > startup > > > > > > > > > > > > script > > > > > > > > > > > > > or > > > > > > > > > > > > > > > > yaml > > > > > > > > > > > > > > > > > > > file that > > > > > > > > > > > > > > > > > > > > is > > > > > > > > > > > > > > > > > > > > > >> > > checked into GitHub to > > > manage > > > > > the > > > > > > > > > machine. > > > > > > > > > > > > > > Someone > > > > > > > > > > > > > > > > will > > > > > > > > > > > > > > > > > > > actually > > > > > > > > > > > > > > > > > > > > > >> have to > > > > > > > > > > > > > > > > > > > > > >> > > physically administer > the > > > > > machine, > > > > > > > > apply > > > > > > > > > > > > > updates, > > > > > > > > > > > > > > > etc. > > > > > > > > > > > > > > > > > > > which will > > > > > > > > > > > > > > > > > > > > > make > > > > > > > > > > > > > > > > > > > > > >> > > community ownership > > > difficult. > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > >> > > Specific to an OSX > build: > > > > > > > > > > > > > > > > > > > > > >> > > - We can't virtualize > OSX > > > > which > > > > > > > means > > > > > > > > > > we'd > > > > > > > > > > > > only > > > > > > > > > > > > > > be > > > > > > > > > > > > > > > > able > > > > > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > > > cover > > > > > > > > > > > > > > > > > > > > > one > > > > > > > > > > > > > > > > > > > > > >> OSX > > > > > > > > > > > > > > > > > > > > > >> > > build environment per > > > physical > > > > > > > device. > > > > > > > > > We > > > > > > > > > > > > > > couldn't > > > > > > > > > > > > > > > > > > target a > > > > > > > > > > > > > > > > > > > > matrix > > > > > > > > > > > > > > > > > > > > > of > > > > > > > > > > > > > > > > > > > > > >> > OSX > > > > > > > > > > > > > > > > > > > > > >> > > and Xcode versions as in > > > > Travis. > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > >> > > -Kellen > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > >> > > On Tue, Dec 12, 2017 at > > 5:46 > > > > PM, > > > > > > > Chris > > > > > > > > > > > > Olivier < > > > > > > > > > > > > > > > > > > > > > cjolivie...@gmail.com > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > >> > > wrote: > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > >> > > > So why Travis when we > > > could > > > > > > > possibly > > > > > > > > > use > > > > > > > > > > > > > > Jenkins? > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > >> > > > On Tue, Dec 12, 2017 > at > > > 7:59 > > > > > AM > > > > > > > > Marco > > > > > > > > > de > > > > > > > > > > > > > Abreu < > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > marco.g.ab...@googlemail.com> > > > > > > > > > > > > > > > > > > > > > >> > > > wrote: > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > Yes that's correct, > > > Chris. > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > Am 12.12.2017 4:46 > > > nachm. > > > > > > > schrieb > > > > > > > > > > "Chris > > > > > > > > > > > > > > > Olivier" > > > > > > > > > > > > > > > > < > > > > > > > > > > > > > > > > > > > > > >> > > cjolivie...@gmail.com > > > > > > > > > > > > > > > > > > > > > >> > > > >: > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > A quick google > > search > > > > > seems > > > > > > to > > > > > > > > > > > indicate > > > > > > > > > > > > > that > > > > > > > > > > > > > > > Mac > > > > > > > > > > > > > > > > > can > > > > > > > > > > > > > > > > > > > be used > > > > > > > > > > > > > > > > > > > > > as > > > > > > > > > > > > > > > > > > > > > >> a > > > > > > > > > > > > > > > > > > > > > >> > > > Jenkins > > > > > > > > > > > > > > > > > > > > > >> > > > > > slave. Is this > > > correct? > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > On Tue, Dec 12, > 2017 > > > at > > > > > 7:42 > > > > > > > AM > > > > > > > > > > > Steffen > > > > > > > > > > > > > > > Rochel < > > > > > > > > > > > > > > > > > > > > > >> > > > > steffenroc...@gmail.com > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > +1 for #1 and #2 > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > I’m working on > > > > getting a > > > > > > > > MacPro > > > > > > > > > to > > > > > > > > > > > add > > > > > > > > > > > > > to > > > > > > > > > > > > > > CI > > > > > > > > > > > > > > > > > > system. > > > > > > > > > > > > > > > > > > > > > >> > > > > > > On Tue, Dec 12, > > 2017 > > > > at > > > > > > 1:43 > > > > > > > > AM > > > > > > > > > > > kellen > > > > > > > > > > > > > > > > > sunderland > > > > > > > > > > > > > > > > > > < > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > kellen.sunderl...@gmail.com > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > Background: > > > > TravisCI > > > > > > is a > > > > > > > > > > startup > > > > > > > > > > > > > > > providing > > > > > > > > > > > > > > > > > > > managed > > > > > > > > > > > > > > > > > > > > > >> continuous > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > integration > > > services > > > > > > with > > > > > > > > > GitHub > > > > > > > > > > > > > > > integration > > > > > > > > > > > > > > > > > and > > > > > > > > > > > > > > > > > > > YAML > > > > > > > > > > > > > > > > > > > > > based > > > > > > > > > > > > > > > > > > > > > >> > > > > > > configuration. > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > TravisCI is > one > > of > > > > the > > > > > > few > > > > > > > > CI > > > > > > > > > > > > > providers > > > > > > > > > > > > > > > that > > > > > > > > > > > > > > > > > > will > > > > > > > > > > > > > > > > > > > build > > > > > > > > > > > > > > > > > > > > a > > > > > > > > > > > > > > > > > > > > > >> > variety > > > > > > > > > > > > > > > > > > > > > >> > > > of > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > OSX/MacOS > builds > > > for > > > > > > > > software > > > > > > > > > > > > > projects. > > > > > > > > > > > > > > > > Their > > > > > > > > > > > > > > > > > > > pricing > > > > > > > > > > > > > > > > > > > > > >> ranges > > > > > > > > > > > > > > > > > > > > > >> > > from > > > > > > > > > > > > > > > > > > > > > >> > > > > Free > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > (for open > > source, > > > 1 > > > > > > > > concurrent > > > > > > > > > > > job, > > > > > > > > > > > > to > > > > > > > > > > > > > > > $489 > > > > > > > > > > > > > > > > > > > monthly for > > > > > > > > > > > > > > > > > > > > 10 > > > > > > > > > > > > > > > > > > > > > >> > > > concurrent > > > > > > > > > > > > > > > > > > > > > >> > > > > > > jobs). > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > Problem: We’ve > > > had a > > > > > few > > > > > > > OSX > > > > > > > > > > build > > > > > > > > > > > > > > issues > > > > > > > > > > > > > > > > slip > > > > > > > > > > > > > > > > > > > into > > > > > > > > > > > > > > > > > > > > MXNet > > > > > > > > > > > > > > > > > > > > > >> > master > > > > > > > > > > > > > > > > > > > > > >> > > in > > > > > > > > > > > > > > > > > > > > > >> > > > > the > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > past few > weeks. > > > > We’ve > > > > > > > > > > previously > > > > > > > > > > > > had > > > > > > > > > > > > > a > > > > > > > > > > > > > > > > Travis > > > > > > > > > > > > > > > > > > CI > > > > > > > > > > > > > > > > > > > based > > > > > > > > > > > > > > > > > > > > > >> testing > > > > > > > > > > > > > > > > > > > > > >> > > > > system > > > > > > > > > > > > > > > > > > > > > >> > > > > > > that > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > would have > > caught > > > > > these > > > > > > > > > issues. > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > Proposals so > > far: > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > 1) Use > TravisCI > > in > > > > > it’s > > > > > > > free > > > > > > > > > > mode > > > > > > > > > > > > for > > > > > > > > > > > > > a > > > > > > > > > > > > > > > very > > > > > > > > > > > > > > > > > > > minimal > > > > > > > > > > > > > > > > > > > > > sanity > > > > > > > > > > > > > > > > > > > > > >> > check > > > > > > > > > > > > > > > > > > > > > >> > > > on > > > > > > > > > > > > > > > > > > > > > >> > > > > > OSX. > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > If we compile > > the > > > > > > program, > > > > > > > > and > > > > > > > > > > for > > > > > > > > > > > > > > example > > > > > > > > > > > > > > > > run > > > > > > > > > > > > > > > > > > > C++ unit > > > > > > > > > > > > > > > > > > > > > >> tests > > > > > > > > > > > > > > > > > > > > > >> > > we’re > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > unlikely to > run > > > into > > > > > > > > problems > > > > > > > > > > with > > > > > > > > > > > > > > queued > > > > > > > > > > > > > > > > > > > builds. The > > > > > > > > > > > > > > > > > > > > > total > > > > > > > > > > > > > > > > > > > > > >> > > build > > > > > > > > > > > > > > > > > > > > > >> > > > > time > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > here should be > > > less > > > > > than > > > > > > > 15 > > > > > > > > > > > minutes. > > > > > > > > > > > > > > > > > > > Configuration > > > > > > > > > > > > > > > > > > > > should > > > > > > > > > > > > > > > > > > > > > >> be > > > > > > > > > > > > > > > > > > > > > >> > > quite > > > > > > > > > > > > > > > > > > > > > >> > > > > > > simple > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > and easy to > > > > maintain. > > > > > > > Error > > > > > > > > > > > > messages > > > > > > > > > > > > > > > should > > > > > > > > > > > > > > > > > > also > > > > > > > > > > > > > > > > > > > be > > > > > > > > > > > > > > > > > > > > > >> obvious to > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > contributors. > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > 2) Run clang > in > > > > Linux > > > > > > with > > > > > > > > our > > > > > > > > > > > > current > > > > > > > > > > > > > > CI. > > > > > > > > > > > > > > > > > > > Building > > > > > > > > > > > > > > > > > > > > with > > > > > > > > > > > > > > > > > > > > > >> clang > > > > > > > > > > > > > > > > > > > > > >> > > > > should > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > take less than > > 10 > > > > > > minutes, > > > > > > > > > > should > > > > > > > > > > > > > flush > > > > > > > > > > > > > > > out > > > > > > > > > > > > > > > > a > > > > > > > > > > > > > > > > > > > large > > > > > > > > > > > > > > > > > > > > subset > > > > > > > > > > > > > > > > > > > > > >> of > > > > > > > > > > > > > > > > > > > > > >> > the > > > > > > > > > > > > > > > > > > > > > >> > > > > > issues > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > we’ve seen > with > > > OSX, > > > > > and > > > > > > > be > > > > > > > > > > quite > > > > > > > > > > > > easy > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > > > maintain. > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > 3) Run full > > > > > test-suites > > > > > > in > > > > > > > > > > > TravisCI, > > > > > > > > > > > > > > > > equaling > > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > level > > > > > > > > > > > > > > > > > > > > of > > > > > > > > > > > > > > > > > > > > > >> > > coverage > > > > > > > > > > > > > > > > > > > > > >> > > > > we > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > provide to > Linux > > > in > > > > > > > Jenkins. > > > > > > > > > > This > > > > > > > > > > > > > could > > > > > > > > > > > > > > > > > require > > > > > > > > > > > > > > > > > > > us to > > > > > > > > > > > > > > > > > > > > > >> > subscribe > > > > > > > > > > > > > > > > > > > > > >> > > > to a > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > monthly > package > > > with > > > > > > > Travis > > > > > > > > to > > > > > > > > > > > > ensure > > > > > > > > > > > > > > our > > > > > > > > > > > > > > > > > build > > > > > > > > > > > > > > > > > > > queue > > > > > > > > > > > > > > > > > > > > > >> doesn’t > > > > > > > > > > > > > > > > > > > > > >> > > grow > > > > > > > > > > > > > > > > > > > > > >> > > > to > > > > > > > > > > > > > > > > > > > > > >> > > > > > an > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > unacceptable > > > length. > > > > > It > > > > > > > may > > > > > > > > > > also > > > > > > > > > > > > > > require > > > > > > > > > > > > > > > a > > > > > > > > > > > > > > > > > > > volunteer to > > > > > > > > > > > > > > > > > > > > > >> setup > > > > > > > > > > > > > > > > > > > > > >> > > and > > > > > > > > > > > > > > > > > > > > > >> > > > > > > maintain > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > long-term. > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > I’d +1 #1 and > #2 > > > as > > > > I > > > > > > > think > > > > > > > > > > those > > > > > > > > > > > > > should > > > > > > > > > > > > > > > be > > > > > > > > > > > > > > > > > > > low-cost, > > > > > > > > > > > > > > > > > > > > > >> > > low-maintence > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > solutions that > > > > should > > > > > > > catch > > > > > > > > > the > > > > > > > > > > > > > majority > > > > > > > > > > > > > > > of > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > problems > > > > > > > > > > > > > > > > > > > > > >> we’ve > > > > > > > > > > > > > > > > > > > > > >> > > seen > > > > > > > > > > > > > > > > > > > > > >> > > > > > thus > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > far. > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > -Kellen > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > Sandeep Krishnamurthy >
