On 2019/08/15 18:51:29, Joan Touzet <woh...@apache.org> wrote:
> Hi Ilya,
>
>
> On 2019-08-15 10:09, Ilya Khlopotov wrote:
> > Hello,
> >
> > There is an https://github.com/apache/couchdb/issues/1428 issue about
> > migrating to rebar3 or mix. I did an experiment to switch from rebar into
> > mix and wanted to share the results. The code for experiment is here
> > https://github.com/apache/couchdb/compare/master...cloudant:switch-to-mix.
> > Overall I am happy with the experiment.
> >
> > # Requirements
> >
> > 1. Should be able to compile dependencies which use either rebar or rebar3
> > 2. Should be able to lock dependencies to specific versions using
> > references to commits
> > 3. Should be able to call eunit test suite
> > 4. Should be able to compile NIFs
> > 5. Should be easy to install plugins
> > 6. Should support dialyzer
> > 7. Should support release managers
> > 8. Support for raw dependencies
> > 9. Produce source tarball suitable for offline compilation (ASF requirement)
> >
> > # How mix supports the requirements
> >
> > 1. Mix is able to figure out which build tool to use. There is also a
> > possibility to specify build tool individually for a given dependency.
> > `manager: mix | rebar | rebar3 | make`. There is also support for calling
> > arbitrary shell command configured via `compile: <command>`. The manual
> > step is required to install the needed tools `mix local.rebar --force`.
> > However there are means to automate that step (we can use Makefile or
> > define a `bootstrap` mix alias )
> > 2. Dependencies are locked via `mix.lock` file.
>
> I don't have any strong opinions one way or the other, but I did see
> Garren asking about 'hex' instead of / in conjunction with 'mix' for
> dependency management. Is this effort combined with his or independent?
> I don't know the tools well enough.
mix works very well with hex
>
> > 3. Eunit is supported via one of the mix plugins. Both plugins are very old
> > but it should be easy to write our own (32 lines of code).
> > - https://github.com/dantswain/mix_eunit
> > - https://github.com/talentdeficit/mixunit
> > 4. NIFs can be compiled using one of the following approaches:
> > - specify `rebar` as build manager for a dependency (easiest)
>
> Are you considering staying with rebar2 here, or moving to rebar(3)? It
> would be nice not to have to keep supporting (and shipping!) our fork of
> rebar2.
I considered to stay with rebar2 for first iteration. Could you remind me why
we are forking rebar?
I did check that our version has no differences from upstream.
git remote -v
origin https://github.com/apache/couchdb-rebar (fetch)
origin https://github.com/apache/couchdb-rebar (push)
upstream https://github.com/rebar/rebar (fetch)
upstream https://github.com/rebar/rebar (push)
git log --oneline origin/master | head
0c435f3 Update master with mainline rebar (v2)
b6d3094 Merge pull request #620 from tuncer/travis-dialyze
8a2aca0 travis-ci: allow Dialyzer job to fail
16d5dfe travis-ci: update otp versions
4e372a3 travis-ci: enable 20.1
73feb5f rebar_xref: ignore opaque type match Dialyzer warning
b05882a rebar_cover: ignore opaque type match Dialyzer warning
903c89d rebar_utils: fix Dialyzer warning
b36e72b Run Dialyzer on Travis-CI
4bd43fe Merge pull request #645 from Juliusan/log_fix
git log --oneline upstream/master | head
b6d3094 Merge pull request #620 from tuncer/travis-dialyze
8a2aca0 travis-ci: allow Dialyzer job to fail
16d5dfe travis-ci: update otp versions
4e372a3 travis-ci: enable 20.1
73feb5f rebar_xref: ignore opaque type match Dialyzer warning
b05882a rebar_cover: ignore opaque type match Dialyzer warning
903c89d rebar_utils: fix Dialyzer warning
b36e72b Run Dialyzer on Travis-CI
4bd43fe Merge pull request #645 from Juliusan/log_fix
2276f85 Log out success message with newlines
git diff origin/master upstream/master
>
> > - call a Makefile from mix.exs. See examples:
> > - https://github.com/SweetIQ/expostal/blob/master/mix.exs#L9
> > - https://github.com/asaaki/discount.ex/blob/master/mix.exs#L6
> > - use https://github.com/davisp/erlang-native-compiler
> > - use https://github.com/blt/port_compiler
>
> Paul left a comment about this on the roadmap issue just today. Can you
> respond to his comment (it was related tot dependencies-of-dependencies,
> i.e. jiffy)
>
> He also raised the concern that it may force us to use only the very
> latest Erlang versions - can you comment on this? Does the current
> hex/mix situation require a minimum version? What version is that?
I did use erlang 20.3.8.14 and elixir 1.6.6. However 1.6.6. is supported on
erlang 19 which is AFAIK a minimal erlang version CouchDB supports.
> > 5. Installing plugins for mix is similar to rebar3. It is sufficient to
> > just specify list of plugins in the deps in mix.exs
> > 6. There are multiple dyalizer plugins to choose from
> > - https://github.com/jeremyjh/dialyxir
> > - https://github.com/gabrielgatu/mix-dialyzer
>
> I know this is a requirement for Cloudant, but today isn't for base
> CouchDB. I would prefer not to introduce this requirement at the same
> time as migrating build tool.
The reason I mentioned it here is because we have 'make dialyze' command.
> > 7. Multiple release managers are supported via mix plugins
> > - distillery is supported via
> > https://hexdocs.pm/distillery/Mix.Releases.Plugin.html
> > - relx and reltools can be used via https://github.com/bitwalker/exrm
> > (deprecated in favor of distillery)
> > - we can also roll our own mix alias using
> > https://github.com/okeuday/reltool_util
> > - latest elixir versions have `release` command out of the box
> > 8. Raw dependencies are supported via `compile: false` option
> > 9. I think it is doable. In the worst case scenario we could just tar the
> > content of `src` directory after calling `mix deps.get`
> >
> > # Warts of mix in the context of CouchDB project
> >
> > - CouchDB doesn't use standard elixir directories layout which caused
> > include_lib directives to not work. To solve this issue absolute path to
> > `<root>/src` need to be added into include path
> > - for dependencies managed by mix it is done via `erlc_options` parameter
> > - Unfortunately when we call out to rebar there is no way to modify
> > arguments. The problem was solved via setting `ERL_COMPILER_OPTIONS`
> > environment variable. Which is not very elegant solution.
>
> This may also cause problems on Windows, where we have very specific
> requirements for compiler options so as to ensure the correct libraries
> are linked for couchjs and so on.
>
> Have you tried compiling on that platform? ;) (Please don't make me do
> all the work here, again...)
>
> > - We need to create empty `rebar.config` file for erlang applications which
> > do not have `rebar.config` (all applications in couchdb repo). This problem
> > can be fixed by adding mix.exs to dependencies we have control over.
> > - The mix is slow when we have to call out to rebar. This problem can be
> > fixed by adding mix.exs to dependencies we have control over.
>
> How much slower? Can you post a comparison of `time make couch` between
> master and your branch?
master:
> time bin/rebar get-deps update-deps
real 0m37.514s
user 0m1.960s
sys 0m3.300s
> time make couch
real 0m49.677s
user 0m46.910s
sys 0m22.210s
mix:
> time mix do deps.get
real 0m24.405s
user 0m1.590s
sys 0m1.270s
> time mix deps.compile
real 3m47.968s
user 0m0.860s
sys 0m0.640s
> > # Roadmap
> >
> > The full conversion would take some time and if we decide to go with mix it
> > makes sense to do it incrementally. Possible roadmap could be:
> > - initial PR
> > - offload dependency fetching to mix.exs and generate rebar.config so
> > release machinery still works
> > - Update Makefile targets to call mix
> > - Choose or implement eunit plugin for mix
> > - Support running tests individually (replacement for `make eunit apps=
> > tests= suites=`)
> > - Replace rebar plugins we use with alternative solutions
> > - Figure out how to use dialyzer
> > - Figure out how to use mix for preparing release
>
> Anything that lands on master must support release preparation. If your
> incremental approach breaks `make dist`, you'll need to stage all of
> your incremental changes on a feature branch and fix `make dist` before
> merging to master. master needs to be releasable at all times.
+1000
> > Best regards,
> > ILYA (aka iilyak)
>
> -Joan "release engineer hat is now off" Touzet
>
>
