Jenkins driven Windows CI is in the cards. We now have Azure credits we can put towards this, or perhaps one of the workers IBM is donating can run this.
It'll have to be a "snowflake" machine build, but I believe it can happen. But you should be able to run a Windows VM using VirtualBox? And we can get you a Windows license if that's an issue. It's kinda late to be solving it in CI... -Joan On 2019-08-20 8:56, Ilya Khlopotov wrote: > I am afraid that until we have a Jenkins driven Windows CI we wouldn't be > able to modernize any of the build or release plumbing. > > Best regards, > iilyak > > On 2019/08/20 12:28:12, Garren Smith <gar...@apache.org> wrote: >> I'm a big +1 on this. Initially, I know we would have some teething issues >> getting this working. But I don't think we can stay on rebar2 forever >> either. Adding in erlang dependancies is becoming increasingly more >> difficult as they expect rebar3. >> >> Cheers >> Garren >> >> On Fri, Aug 16, 2019 at 2:02 PM Ilya Khlopotov <iil...@apache.org> wrote: >> >>> >>> >>> On 2019/08/15 18:51:29, Joan Touzet <woh...@apache.org> wrote: >>>> Hi Ilya, >>>> >>>> >>>> On 2019-08-15 10:09, Ilya Khlopotov wrote: >>>>> Hello, >>>>> >>>>> There is an https://github.com/apache/couchdb/issues/1428 issue about >>> migrating to rebar3 or mix. I did an experiment to switch from rebar into >>> mix and wanted to share the results. The code for experiment is here >>> https://github.com/apache/couchdb/compare/master...cloudant:switch-to-mix. >>> Overall I am happy with the experiment. >>>>> >>>>> # Requirements >>>>> >>>>> 1. Should be able to compile dependencies which use either rebar or >>> rebar3 >>>>> 2. Should be able to lock dependencies to specific versions using >>> references to commits >>>>> 3. Should be able to call eunit test suite >>>>> 4. Should be able to compile NIFs >>>>> 5. Should be easy to install plugins >>>>> 6. Should support dialyzer >>>>> 7. Should support release managers >>>>> 8. Support for raw dependencies >>>>> 9. Produce source tarball suitable for offline compilation (ASF >>> requirement) >>>>> >>>>> # How mix supports the requirements >>>>> >>>>> 1. Mix is able to figure out which build tool to use. There is also a >>> possibility to specify build tool individually for a given dependency. >>>>> `manager: mix | rebar | rebar3 | make`. There is also support for >>> calling arbitrary shell command configured via `compile: <command>`. The >>> manual step is required to install the needed tools `mix local.rebar >>> --force`. >>>>> However there are means to automate that step (we can use Makefile >>> or define a `bootstrap` mix alias ) >>>>> 2. Dependencies are locked via `mix.lock` file. >>>> >>>> I don't have any strong opinions one way or the other, but I did see >>>> Garren asking about 'hex' instead of / in conjunction with 'mix' for >>>> dependency management. Is this effort combined with his or independent? >>>> I don't know the tools well enough. >>> >>> mix works very well with hex >>> >>>> >>>>> 3. Eunit is supported via one of the mix plugins. Both plugins are >>> very old but it should be easy to write our own (32 lines of code). >>>>> - https://github.com/dantswain/mix_eunit >>>>> - https://github.com/talentdeficit/mixunit >>>>> 4. NIFs can be compiled using one of the following approaches: >>>>> - specify `rebar` as build manager for a dependency (easiest) >>>> >>>> Are you considering staying with rebar2 here, or moving to rebar(3)? It >>>> would be nice not to have to keep supporting (and shipping!) our fork of >>>> rebar2. >>> I considered to stay with rebar2 for first iteration. Could you remind me >>> why we are forking rebar? >>> I did check that our version has no differences from upstream. >>> git remote -v >>> origin https://github.com/apache/couchdb-rebar (fetch) >>> origin https://github.com/apache/couchdb-rebar (push) >>> upstream https://github.com/rebar/rebar (fetch) >>> upstream https://github.com/rebar/rebar (push) >>> >>> git log --oneline origin/master | head >>> 0c435f3 Update master with mainline rebar (v2) >>> b6d3094 Merge pull request #620 from tuncer/travis-dialyze >>> 8a2aca0 travis-ci: allow Dialyzer job to fail >>> 16d5dfe travis-ci: update otp versions >>> 4e372a3 travis-ci: enable 20.1 >>> 73feb5f rebar_xref: ignore opaque type match Dialyzer warning >>> b05882a rebar_cover: ignore opaque type match Dialyzer warning >>> 903c89d rebar_utils: fix Dialyzer warning >>> b36e72b Run Dialyzer on Travis-CI >>> 4bd43fe Merge pull request #645 from Juliusan/log_fix >>> >>> git log --oneline upstream/master | head >>> b6d3094 Merge pull request #620 from tuncer/travis-dialyze >>> 8a2aca0 travis-ci: allow Dialyzer job to fail >>> 16d5dfe travis-ci: update otp versions >>> 4e372a3 travis-ci: enable 20.1 >>> 73feb5f rebar_xref: ignore opaque type match Dialyzer warning >>> b05882a rebar_cover: ignore opaque type match Dialyzer warning >>> 903c89d rebar_utils: fix Dialyzer warning >>> b36e72b Run Dialyzer on Travis-CI >>> 4bd43fe Merge pull request #645 from Juliusan/log_fix >>> 2276f85 Log out success message with newlines >>> >>> git diff origin/master upstream/master >>> >>>> >>>>> - call a Makefile from mix.exs. See examples: >>>>> - https://github.com/SweetIQ/expostal/blob/master/mix.exs#L9 >>>>> - https://github.com/asaaki/discount.ex/blob/master/mix.exs#L6 >>>>> - use https://github.com/davisp/erlang-native-compiler >>>>> - use https://github.com/blt/port_compiler >>>> >>>> Paul left a comment about this on the roadmap issue just today. Can you >>>> respond to his comment (it was related tot dependencies-of-dependencies, >>>> i.e. jiffy) >>>> >>>> He also raised the concern that it may force us to use only the very >>>> latest Erlang versions - can you comment on this? Does the current >>>> hex/mix situation require a minimum version? What version is that? >>> >>> I did use erlang 20.3.8.14 and elixir 1.6.6. However 1.6.6. is supported >>> on erlang 19 which is AFAIK a minimal erlang version CouchDB supports. >>> >>>>> 5. Installing plugins for mix is similar to rebar3. It is sufficient >>> to just specify list of plugins in the deps in mix.exs >>>>> 6. There are multiple dyalizer plugins to choose from >>>>> - https://github.com/jeremyjh/dialyxir >>>>> - https://github.com/gabrielgatu/mix-dialyzer >>>> >>>> I know this is a requirement for Cloudant, but today isn't for base >>>> CouchDB. I would prefer not to introduce this requirement at the same >>>> time as migrating build tool. >>> >>> The reason I mentioned it here is because we have 'make dialyze' command. >>> >>>>> 7. Multiple release managers are supported via mix plugins >>>>> - distillery is supported via >>> https://hexdocs.pm/distillery/Mix.Releases.Plugin.html >>>>> - relx and reltools can be used via >>> https://github.com/bitwalker/exrm (deprecated in favor of distillery) >>>>> - we can also roll our own mix alias using >>> https://github.com/okeuday/reltool_util >>>>> - latest elixir versions have `release` command out of the box >>>>> 8. Raw dependencies are supported via `compile: false` option >>>>> 9. I think it is doable. In the worst case scenario we could just tar >>> the content of `src` directory after calling `mix deps.get` >>>>> >>>>> # Warts of mix in the context of CouchDB project >>>>> >>>>> - CouchDB doesn't use standard elixir directories layout which caused >>> include_lib directives to not work. To solve this issue absolute path to >>> `<root>/src` need to be added into include path >>>>> - for dependencies managed by mix it is done via `erlc_options` >>> parameter >>>>> - Unfortunately when we call out to rebar there is no way to modify >>> arguments. The problem was solved via setting `ERL_COMPILER_OPTIONS` >>> environment variable. Which is not very elegant solution. >>>> >>>> This may also cause problems on Windows, where we have very specific >>>> requirements for compiler options so as to ensure the correct libraries >>>> are linked for couchjs and so on. >>>> >>>> Have you tried compiling on that platform? ;) (Please don't make me do >>>> all the work here, again...) >>>> >>>>> - We need to create empty `rebar.config` file for erlang applications >>> which do not have `rebar.config` (all applications in couchdb repo). This >>> problem can be fixed by adding mix.exs to dependencies we have control over. >>>>> - The mix is slow when we have to call out to rebar. This problem can >>> be fixed by adding mix.exs to dependencies we have control over. >>>> >>>> How much slower? Can you post a comparison of `time make couch` between >>>> master and your branch? >>> >>> master: >>> > time bin/rebar get-deps update-deps >>> real 0m37.514s >>> user 0m1.960s >>> sys 0m3.300s >>> > time make couch >>> real 0m49.677s >>> user 0m46.910s >>> sys 0m22.210s >>> >>> mix: >>> > time mix do deps.get >>> real 0m24.405s >>> user 0m1.590s >>> sys 0m1.270s >>> > time mix deps.compile >>> real 3m47.968s >>> user 0m0.860s >>> sys 0m0.640s >>> >>>>> # Roadmap >>>>> >>>>> The full conversion would take some time and if we decide to go with >>> mix it makes sense to do it incrementally. Possible roadmap could be: >>>>> - initial PR >>>>> - offload dependency fetching to mix.exs and generate rebar.config >>> so release machinery still works >>>>> - Update Makefile targets to call mix >>>>> - Choose or implement eunit plugin for mix >>>>> - Support running tests individually (replacement for `make eunit >>> apps= tests= suites=`) >>>>> - Replace rebar plugins we use with alternative solutions >>>>> - Figure out how to use dialyzer >>>>> - Figure out how to use mix for preparing release >>>> >>>> Anything that lands on master must support release preparation. If your >>>> incremental approach breaks `make dist`, you'll need to stage all of >>>> your incremental changes on a feature branch and fix `make dist` before >>>> merging to master. master needs to be releasable at all times. >>> +1000 >>> >>>>> Best regards, >>>>> ILYA (aka iilyak) >>>> >>>> -Joan "release engineer hat is now off" Touzet >>>> >>>> >>> >>
