Thanks Matt! The main reason we used thirdparty(as stated above) is that the TVM PPMC believes it is important to protect the Apache brand and release policy(artifact being compatible apache). Given some of the optional test packages (e.g. pytorch/tensorflow) are not what we could control, and it is hard to validate their compatibility. There are also similar issues around specific runtime environments like cuda. Note that CI images are not artifacts that we release, but merely as a way to setup the ci build env.
So to keep things simple and clean from branding/licensing concerns, we use a thirdparty and clearly distinguish them from ASF artifacts. To the development, there is no strict location requirement as long as we can point to a cache. And the cache location can be changed at any time point by any committer. TQ On Fri, Aug 28, 2020 at 7:13 PM Matt Sicker <boa...@gmail.com> wrote: > Makes sense. Do you think the recent changes to Docker Hub will affect > this? It might be a good idea to move the images into the Apache org, even > if they’re not for end users. > > On Fri, Aug 28, 2020 at 20:52 Tianqi Chen <tqc...@apache.org> wrote: > > > Thanks Matt! > > > > > > > > That was indeed the approach we used before. The main problem of this > > > > approach is > > > > - There could be certain upstream changes (say tensorflow get updated) > that > > > > may not retrigger the rebuild > > > > - The CI instance itself can quickly get crowded by the historical cached > > > > images and causes disk overflow problems. > > > > - Overall we find it is very hard to reproduce the CI error when there > is a > > > > problem because of the potential mismatch(due to stale build) > > > > > > > > So the community switched to using a stable binary tag as for more stable > > > > env without blocking the CI, while periodically updating > > > > these images when this is a dependency change. > > > > > > > > TQ > > > > > > > > On Fri, Aug 28, 2020 at 6:47 PM Matt Sicker <boa...@gmail.com> wrote: > > > > > > > > > Is there even a need to upload the Docker images for CI? Docker > > recognizes > > > > > layers by checksums, so CI build agents will cache the image layers > > > > > regardless of whether or not you upload them to Docker Hub. Layers get > > > > > rebuilt when the Docker file changes or you force an update. > > > > > > > > > > On Fri, Aug 28, 2020 at 20:12 Tianqi Chen <tqc...@apache.org> wrote: > > > > > > > > > > > Thanks Justin and Henry for the discussion thread. > > > > > > > > > > > > > > > > > > > > > > > > Please allow me to dissect and summarize again some of the discussion > > > > > > > > > > > > points:) > > > > > > > > > > > > > > > > > > > > > > > > C0: Docker Image > > > > > > > > > > > > - There was a mis-understanding that the docker image like ci-gpu > > > > > > > > > > > > contains tvm. They do not, and instead contain an environment to > > > > > > > > > > > > build and run test cases. > > > > > > > > > > > > - The thirdparty binary cache is used to speed up the test build. > > Anyone > > > > > > > > > > > > could build these binaries from a TVM source. > > > > > > > > > > > > - To avoid confusion about the branding, we have renamed the docker > hub > > > > > > > > > > > > name to > > > > > > > > > > > > a different name that does not contain tvm so that it is clear as a > > > > > > > > > > > > thirdparty. > > > > > > > > > > > > > > > > > > > > > > > > C1: Pointing to Apache Release > > > > > > > > > > > > - The PPMC 100% agree that we should produce high-quality Apache > > > > > > > > > > > > compatible releases, and refer to them in the documents. > > > > > > > > > > > > - The installation documentation points to the official release > > download > > > > > > > > > > > > page [1]. > > > > > > > > > > > > - There is a for developer section, which gives instructions to > > > > > developers > > > > > > > > > > > > about > > > > > > > > > > > > how to clone the code from the git repo. > > > > > > > > > > > > > > > > > > > > > > > > C2: Number of releases > > > > > > > > > > > > - The PPMC wants to produce high quality releases according to > feature > > > > > > > > > > > > plan coordinated with the community, rather than creating more > > > > > releases. > > > > > > > > > > > > - The release process is well documented[2], > > > > > > > > > > > > - Per incubator policy[2], number of releases is not a hard bar for > > > > > > > > > > > > graduation. > > > > > > > > > > > > "Podlings do not need to actually publish a release...". Instead > the > > > > > most > > > > > > > > > > > > important > > > > > > > > > > > > thing is the demonstration of being able to cut apache release. > > > > > > > > > > > > - Besides the release process being well documented and there are > > already > > > > > > > > > > > > two high-quality releases(without WIP). The PMC contains several > > Apache > > > > > > > > > > > > members > > > > > > > > > > > > who have previous experiences in cutting releases in TLP projects. > > > > > > > > > > > > We are very confident that the PMC can continue to do so. > > > > > > > > > > > > > > > > > > > > > > > > C3: The discourse forum > > > > > > > > > > > > - The discourse forum, like the use of github, is mainly a way for > user > > > > > > > > > > > > community to > > > > > > > > > > > > engage with the project. > > > > > > > > > > > > - The usage of discourse forum is voted by the community[3] > > > > > > > > > > > > - The community follows the public archive principle, which means > > > > > > > > > > > > everything is archived > > > > > > > > > > > > to mail-lists. Again, everything happens (also) happens on a > > mail-list. > > > > > > > > > > > > - The discourse forum is currently maintained by a few PMC members as > > > > > > > > > > > > volunteers, > > > > > > > > > > > > as a thirdparty service to the community. > > > > > > > > > > > > - Again, the development happens in github, everything mirrored in > dev@ > > . > > > > > > > > > > > > The existence of the forum > > > > > > > > > > > > would not block the development. > > > > > > > > > > > > > > > > > > > > > > > > Finally, to summarize. I believe one of the main reasons behind these > > > > > > > > > > > > issues are trust rather than procedural. > > > > > > > > > > > > As a ASF member, I am certainly strive to make sure the related > > volunteer > > > > > > > > > > > > maintained resources > > > > > > > > > > > > will continue beyond a certain individual by many ways including: > > > > > > > > > > > > > > > > > > > > > > > > - Introduce multiple volunteers from different organizations in the > PMC > > > > > to > > > > > > > > > > > > do the work. > > > > > > > > > > > > - Make sure the messages are backed up to mail-list. > > > > > > > > > > > > - Help INFRA to manage certain services instead if they are willing. > > > > > > > > > > > > However, note in cases like CI > > > > > > > > > > > > INFRA is also happy to let the PMC make specific choices. > > > > > > > > > > > > > > > > > > > > > > > > While many of the arguments in the above points boils down to "what > if > > > > > the > > > > > > > > > > > > few people did something evil via loophole X". > > > > > > > > > > > > And similar arguments can be applied to committer nominations in > > general. > > > > > > > > > > > > As I may quote from our committer > > > > > > > > > > > > nomination message "We like to work on trust rather than unnecessary > > > > > > > > > > > > constraints". > > > > > > > > > > > > This is the message that keeps bringing me back and makes me proud > as a > > > > > ASF > > > > > > > > > > > > member. > > > > > > > > > > > > > > > > > > > > > > > > There is no "one way" to the Apache way[3]. IMHO, the best thing I > love > > > > > > > > > > > > about the ASF is indeed the people, and the > > > > > > > > > > > > trust we give to each PMC as long as things are operated under the > > Apache > > > > > > > > > > > > Way. The TVM PPMC has been working hard to > > > > > > > > > > > > build a healthy, diverse and independent community. And strives to > > > > > protect > > > > > > > > > > > > the Apache brand and uphold apache release policy. > > > > > > > > > > > > Again, we welcome constructive feedback for continued improvement. > > > > > > > > > > > > > > > > > > > > > > > > Thank you! > > > > > > > > > > > > > > > > > > > > > > > > TQ > > > > > > > > > > > > > > > > > > > > > > > > ----- > > > > > > > > > > > > - [1] > > > > > > > > > > > > > > https://tvm.apache.org/docs/install/from_source.html#install-from-source > > > > > > > > > > > > - [2] https://incubator.apache.org/guides/graduation.html > > > > > > > > > > > > - [3] > > > > > > > > > > > > > > > > > > > > > > > > > > https://lists.apache.org/thread.html/c34b728f01d1030146594e47e0706cd1990ed731d06e3c179b7d501a%40%3Cdev.tvm.apache.org%3E > > > > > > > > > > > > - [4] https://www.apache.org/theapacheway/ > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Aug 28, 2020 at 5:15 PM Henry Saputra < > henry.sapu...@gmail.com > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > Sure, that is easy to fix and good suggestion. But, hope it is not > a > > > > > > > > > > > > > blocker issue. > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Aug 28, 2020, 5:09 PM Justin Mclean < > > jus...@classsoftware.com> > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > > > > > > > > > > > > > > > The "install from source” page should probably point people to > the > > > > > > source > > > > > > > > > > > > > > releases rather than the latest code in GitHub. > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > Justin > > > > > > > > > > > > > > > > --------------------------------------------------------------------- > > > > > > > > > > > > > > To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org > > > > > > > > > > > > > > For additional commands, e-mail: > general-h...@incubator.apache.org > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > Matt Sicker <boa...@gmail.com> > > > > > > > > > -- > Matt Sicker <boa...@gmail.com> >
