Iblis, Thank you for your work in developing a release process for the Julia package. Thanks especially for explaining it in way that is approachable for others not familiar with the Julia dependency management system.
I know there are some conditions under which Apache can distribute releases downstream. >From this conversation https://issues.apache.org/jira/browse/LEGAL-270 It seems that "it is appropriate to distribute official releases through downstream channels". Another github repo seems like it could be a downstream channel for ease with Julia deps. I don't know whether the DMLC repo would be a problem with the naming or as long as it specified it was a distribution of the Apache (incubating) MXNet project it would be ok. Do you know of any other Julia Apache projects? Maybe we could look and see how they are doing it. Best, Carin On Sat, Jan 5, 2019 at 11:00 AM iblis <ib...@hs.ntnu.edu.tw> wrote: > Hello MXNet community, > > After the Julia package has been imported into the main repo, > the next unresolved issue is establishing the releasing process for Julia. > > The PR #12845, which is for upgrading Julia package to work with Julia > v0.7+, > is ready for review at this moment. > I want to release the Julia part along with the next mxnet major release > (v1.5, I think). > > We imported the Julia package to make it part of the CI validation chain. > And merging code into main repo makes Julia release process different from > normal. > > At first, let me describe a usual release process for a Julia package. > I will try to explain it with corresponding ideas in Python. > > 1.Package manager in Julia: > There are a module from stdlib named `Pkg` served as the package > manager in Julia, > like Python's `pip`. > > 2. The package registry and package source: > The package manager will download the desired packages from a default > registry. > In Python, pip will conduct two actions before installing packags: > (1) Search the metadata (like pkg name, version, ...etc) > from the CheeseShop (a.k.a PyPI) for users. > (2) Fetch them from the CheeseShop if matched. > > In Julia, there is a default registry to hold the packages metadata, > hosted on GitHub: > https://github.com/JuliaLang/METADATA.jl > > And where to fetch the package? The aren't a center for download > packages. > It's unlike Python's idea, we don't upload package to a center server > in case of Julia. > > The package manager will fetch stuffs from the URL listed in the > metadata. > It should be a valid git URL for git-clone. > e.g. For the package `Distributions`: > > https://github.com/JuliaLang/METADATA.jl/blob/metadata-v2/Distributions/url > Most of the packages use GitHub for hosting code. > > 3. A valid package directory structure > > In the process of package installation, > Python's pip will run `./setup.py` from unpacked package dir. > This is kind of protocol between package developer and package > manager. > > In Julia, the package manager run `./deps/build.jl` from the > git-cloned dir. > > Here comes an issue of mxnet's main repo. > The Julia package is collected under the subdir `./julia/`, so the > setup script > in our case is `./julia/deps/build.jl`. > This breaks the protocol. > > Also, in runtime, there is another protocol for phase of module > loading. > The entry point of a module must be `./src/PackageName.jl`. > In our repo, it's `./julia/src/MXNet.jl`. > So, the directory structure of mxnet main repo is invalid for Julia > package manager. > Putting the url of main repo as package metadata will not work. > > The proposed solution: > > To provide a valid dir structure, it need a standalone git repo. > We can just treat this standalone repo as the place to "upload" package > code, > just like uploading Python package to PyPI in every release. > > I want to discuss about the home of this standalone repo. > 1) Reuse the old repo: https://github.com/dmlc/MXNet.jl > It's under DMLC. I have the committer bit of this repo. > 2) A new repo under ASF's organization? > 3) Anything else? > > I will vote for option (1). > > ---- > The actual process in detail: > 1) Split the julia dir out via git-subtree > ``` > cd /path/to/mxnet/ > git subtree split -P julia -b MXNet_jl > ``` > > 2) Push the branch `MXNet_jl` to the standalone git repo mentioned in > previous section. > > 3) Push a new tag for MXNet.jl. The version number is same as the > mxnet main repo. > > 4) Make a release page via GitHub release button. > > 5) Then, updating metadata to Julia package registry will be > automated by the bot. > https://github.com/attobot/attobot > > 6) Relax! > > -- > Iblis Lin >
