So firstly let's try to keep our responses empathetic and avoid ad-hom comments. It might be beneficial to take some time to review the Apache Code of Conduct [1]. Konstantin has taken a lot of time to think about dependency management in MXNet on a volunteer basis which is commendable.
Second, Junru, I share many of your security concerns, but my understanding is that Conan.io allows you to pull dependencies as binaries, or as source using the -build option, so we're not limited to strictly pulling binaries from remote servers. Some benefits I see: * A uniform method of pulling dependencies is much easier to maintain and reason about. Need to update a package because of a security vulnerability? Go to the single place we configure dependencies and update it. * We have many dependencies that do not need to be checked out depending on the build options a user desires (so called build conditional dependencies). There's not much point in downloading / checking out these dependencies if you're not going to use them. * Subrepo sources have to have certified license reviews every release. Using a package manager would solve this issue. * We have an extra user base (Conan.io users) who get exposure to MXNet, growing our user base. Many of these benefits we'd get with other package management systems. One option I had previously proposed was Hunter, which is basically a wrapper around CMake's ExternalProject functionality. The tradeoff I see between the two is that Hunter (or ExternalProject via CMake) wold be lighter weight but would have less support, a smaller community and would be hard to use consistently across the project with the variety of collaborators it has. 1: https://www.apache.org/foundation/policies/conduct.html. 2: https://github.com/ruslo/hunter On Fri, May 3, 2019 at 9:10 AM Junru Shao <junrushao1...@gmail.com> wrote: > I am actually a bit concerned about the security issues. We are asked to > download binaries from third-party websites, which are not controlled or > validated by Apache. Although it is claimed to be “decentralized”, I am > really not convinced where the security comes from. > > In the meantime, sacrificing security doesn’t really bring us tangible > benefits. CMake does support download artifacts from a pre-defined website > very well. We may also have pre-built binaries stored in our CI docker > without having to download it them from the internet. > > Another point is that I am not convinced of any advantage of Conan over > other package managers for C++. Would you mind at least mentioning the > benefits somewhere in this thread, rather than carelessly includes tons of > irrelevant links (some of which are even wrong) and eagerly asking for a > vote? I believe that reasonable discussion would keep us within a *healthy* > discussion. > > Last but not least, as we all know, most of the dependencies you mentioned > adopts CMake, rather than Conan, the meta-generator which generates CMake. > I didn’t see your logic stands like “oh you have tons of dependencies so > you must use Conan”, why not I simply specify versions in git submidule, > which everyone understands how it behaves? Everyone know how to include a > sub-directory in cmake in one line, so why we have to write python to make > it less understandable and more complicated? > > In conclusion, we need to be reasonable in healthy discussion. I don’t > particularly want to rudely +1 or -1 for a thing that is unclear to me, but > I really want to see pros and cons, discussion about issues and concerns, > rather than broken links to nonsense. > > Looking forward to your reply! > > Thanks, > Junru > > On Fri, May 3, 2019 at 08:05 kellen sunderland < > kellen.sunderl...@gmail.com> > wrote: > > > Hey Konstantin. Thanks for starting an email thread and sorry for the > > confusion. I think the ides is that we should discuss and agree on > > Conan.io adoption first on the dev list, then start merging PRs. Release > > 1.4.1 is already in testing and the 1.5 code freeze deadline is also near > > so I think it could be difficult to make such a large change on one of > > those releases. I've looked into package management solutions for the > > project before. I was in favour of hunter, but I think Conan's adoption > > rate now makes it the best option. It's simple to use and is becoming > > industry standard, with a minor downside of requiring Python (which has > > meanwhile become the most popular dev language). I'd personally be -1 > for > > 1.4.1 or 1.5, +1 for using Conan in 1.6 or 2.0. > > > > -Kellen > > > > On Fri, May 3, 2019 at 12:59 AM Konstantin Ivlev <tomsks...@gmail.com> > > wrote: > > > > > hi Sheng Zha, > > > > > > on pull request review I was told by Anirudh anirudhacharya and Roshani > > > Nagmote to start discussion/vote on the mxnet dev list. it seems to be > a > > > vicious circle now - on GitHub I am told to use vote, and on vote I am > > told > > > to use GitHub, this doesn't help much. > > > FYI GitHub review stuck, it's already opened since November 2018, and > > it's > > > still not approved (however, there were no objections during the > review). > > > Previous discussion in e-mail thread also didn't encounter any > > objections, > > > and all questions were answered. > > > JIRA ticket has no discussion at all (except it has duplicates of > > comments > > > made on GitHub). > > > so let's process with 3-day vote for now, as other communication > channels > > > were already tried with no success. > > > > > > yours sincerely, Konstantin > > > > > > пт, 3 мая 2019 г. в 14:17, Sheng Zha <zhash...@apache.org>: > > > > > > > Hi Konstantin, > > > > > > > > While conan looks like an option that's worth exploring, given that > > your > > > > request is to merge the pull request, I'd suggest that the request > > should > > > > go through the regular pull request review and it doesn't really > need a > > > > vote (as it doesn't substitute reviews anyway) > > > > > > > > If you would like to gather more attention to it, feel free to ping > in > > a > > > > discussion thread. > > > > > > > > -sz > > > > > > > > On 2019/05/03 06:29:55, Konstantin Ivlev <tomsks...@gmail.com> > wrote: > > > > > Dear MXNet community, > > > > > > > > > > This is the 3-day vote to add conan support for Apache MXNet > > > (incubating) > > > > > version v1.4.1. > > > > > The voting on dev@ list will start May 03 23:59:59 (PST) and close > > on > > > > May > > > > > 06 23:59:59. > > > > > > > > > > Background: conan is open-source, freeware, cross-platform package > > > > manager > > > > > for C and C++ projects, written in python. it provides integration > > with > > > > > various build systems, include CMake. conan may use bintray as a > > server > > > > to > > > > > store and download pre-built packages, or packages might be always > > > built > > > > > from sources. > > > > > > > > > > Problem: currently (as for v1.4.1), Apache MXNet (incubating) is > > using > > > > > several ways to fetch 3rd-party dependencies simultaneously, for > > > > instance: > > > > > 1. download GitHub archives during the build > > > > > - OpenBLAS > > > > > - OpenCV > > > > > 2. conda (alternative way to GitHub archives) > > > > > 3. download from CMake > > > > > - Intel Math Kernel Library (MKL) > > > > > 4. Git submodules > > > > > - cub > > > > > - dlpack > > > > > - dmlc-core > > > > > - googletest > > > > > - mkldnn > > > > > - mshadow > > > > > - onnx-tensorrt > > > > > - openmp > > > > > - ps-lite > > > > > - tvm > > > > > therefore, there are multiple places to look for 3rd parties, and > its > > > > hard > > > > > to update them, as you need to remember or figure it out how to > > update > > > a > > > > > particular dependency to newer version, for instance. > > > > > current Apache MXNet (incubating) build instructions differ very > much > > > per > > > > > platform, require to download and unzip some archives manually, > > > > specifying > > > > > variables with paths to this archives, in conjunction of updating > git > > > > > submodules, > > > > > > > > > > Action: merge pull request providing an initial conan support for > > > Apache > > > > > MXNet (incubating). support conan as an alternate approach to fetch > > > > various > > > > > 3rd-party dependencies. old approaches will be still available, > > > supported > > > > > and left intact. > > > > > > > > > > Below are links to > > > > > 1) conan web-site: https://conan.io/ > > > > > 2) conan GitHub repository: https://github.com/conan-io/conan > > > > > 3) conan documentation: https://docs.conan.io/en/latest/ > > > > > 4) bintray: https://bintray.com > > > > > 5) pull request adding conan support to Apache MXNet (incubating): > > > > > https://github.com/apache/incubator-mxnet/pull/13400 > > > > > 6) JIRA issue: https://issues.apache.org/jira/browse/MXNET-1229 > > > > > 7) previous email discussion: > > > > > > > > > > > > > > > https://lists.apache.org/thread.html/301a46a637f7e3c249c475713f701bef7530c32bc92d8834c0882897@%3Cdev.mxnet.apache.org%3E > > > > > 8) MXNet build instructions: > > > > > https://mxnet-tqchen.readthedocs.io/en/latest/how_to/build.html > > > > > 9) MXNet build instructions (Windows): > > > > > > > > > > > > > > > https://mxnet.incubator.apache.org/versions/master/install/windows_setup.html > > > > > 10) MXNet build instructions (OSX): > > > > > > > > > http://mxnet.incubator.apache.org/versions/master/install/osx_setup.html > > > > > 11) MXNet build instructions (Linux): > > > > > > > > > > > > > > > http://mxnet.incubator.apache.org/versions/master/install/ubuntu_setup.html > > > > > 12) MXNet development setup (OSX): > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/MXNET/MXNet+Developer+Setup+on+Mac > > > > > > > > > > Please remember to TEST first before voting accordingly: > > > > > +1 = approve > > > > > +0 = no opinion > > > > > -1 = disapprove (provide reason) > > > > > > > > > > Best regards, > > > > > Konstantin Ivlev > > > > > > > > > > > > > > >
