The point is exactly that we cannot assume that the underlying coverage tool is using .coverdata or that it would accept it as input. They need to provide their own APIs that handle those. The only callback we have today for external coverage tool is handling out to them the whole coverage process, it is not meal peace based on .coverdata.
On Thu, Nov 12, 2020 at 12:41 PM Ilya Borovitinov <icehaun...@gmail.com> wrote: > Hello again, thanks for your time and prompt reply! > > I am not sure, I might not have been clear enough. For example, given an > umbrella project, I can run `mix test --export-coverage` from the project > root. Even if `:covertool` is specified as a tool in `test_coverage:`, the > .coverdata files are exported correctly alongside XML within each folder. > > The problem is: when I run `mix test.coverage`, it takes all .coverdata > files and compiles them into an HTML report > <https://github.com/elixir-lang/elixir/blob/master/lib/mix/lib/mix/tasks/test.coverage.ex#L179> > without any consideration for the specified coverage tool. Yes, this issue > could be solved with a custom command within the package to operate upon > coverdata files, but I still believe that this behavior is > counterintuitive. Maybe another option could be specified explicitly for > report generation based on multiple coverdata files? > > One more example (although not my use case) could be the `--partitions` > flag. It implicitly generates coverdata files, which are then expected to > be compiled via `mix test.coverage`. So if a person has the tool specified > in `test_coverage`, then running `mix test` will produce different reports > from `mix test --partitions` and `mix test.coverage` because one will use > the coverage tool and the other will not. > > Thank you again for your attention! > On Thursday, 12 November 2020 at 11:39:02 UTC+3 José Valim wrote: > >> Hi Ilya, thanks for the report! >> >> mix test.coverage is dependent on how the default coverage tool exports >> data. We cannot aggregate coverage reports from random tools because we >> don't know how these tools export their results. Therefore, it is up to >> these tools to do two changes: >> >> 1. Respect the :export option in their coverage tool >> 2. Provide their own mix task that aggregate the results >> >> So please reach out to them and consider sending pull requests so they >> make this feature available, >> >> Thank you! >> >> On Wed, Nov 11, 2020 at 10:59 PM Ilya Borovitinov <iceha...@gmail.com> >> wrote: >> >>> Hello! >>> >>> Today I encountered a problem when trying to set up testing + coverage >>> for the new Elixir project with Elixir 1.11. >>> >>> We are using Gitlab as our source control and CI, and Gitlab can display >>> covered lines if you were to provide coverage in the Cobertura XML format. >>> That's all good, and Elixir/Erlang ecosystem even has `covertool` >>> <https://github.com/covertool/covertool> package for this purpose. >>> >>> I have an umbrella project with 3 apps currently, and so I read in the >>> release notes that I could use `mix test` with `--export-coverage` flag to >>> collect coverage from each app and then compile them with `mix >>> test.coverage`. That's awesome. >>> >>> Here's the problem though: `mix test` calls upon the specified tool to >>> process coverage data, but `mix test.coverage` just generates HTML, so we >>> cannot use any tools to properly compile and process coverage data across >>> all apps (or partitions). Basically, I feel that there shouldn't be such a >>> discrepancy in usage between these two commands. >>> >>> With great admiration for the language, >>> Ilya Borovitinov >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "elixir-lang-core" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to elixir-lang-co...@googlegroups.com. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/elixir-lang-core/848b4145-ef5e-48fa-863f-d0fb65f2a8cfn%40googlegroups.com >>> <https://groups.google.com/d/msgid/elixir-lang-core/848b4145-ef5e-48fa-863f-d0fb65f2a8cfn%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> >> -- > You received this message because you are subscribed to the Google Groups > "elixir-lang-core" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to elixir-lang-core+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/elixir-lang-core/9ceb51e7-82cf-4edb-8315-cd05df5a1bbfn%40googlegroups.com > <https://groups.google.com/d/msgid/elixir-lang-core/9ceb51e7-82cf-4edb-8315-cd05df5a1bbfn%40googlegroups.com?utm_medium=email&utm_source=footer> > . > -- You received this message because you are subscribed to the Google Groups "elixir-lang-core" group. To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-core+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/CAGnRm4K%2BtAs84r3W7dPuBK_rznLDR7BRL73Dn-%2B4w8H4y4dJeQ%40mail.gmail.com.
