Sergey,
We are working on a full automated build pipeline that will test against a wide
range of configurations. Ideally, the pipeline will work as follows:
1. PR is submitted -> the PR is built, unit tested, and run through static
analysis. Smoke tests are run on a matrix of configurations
2. The PR is reviewed by two or more committers
3. When a PR commit receives two or more LGTMs and no -1s, it is enqueued
to be merged to the target branch
4. When it’s PR’s turn to be merged, it merged against the target release
branch and rebuilt (unit tests + static analysis), packaged, and tested (smoke
test + a set of component tests) on a matrix of configurations
5. If the PR needs to be forward merged, it is enqueued for the next
branch and Step 4 is re-executed
In this system, only one PR would be actively merged at a time. Steps 4 and 5
are repeated until the PR is merged to master. This process not only ensures
that the PR is adequately reviewed and tested, and it also detects introduced
by merges of adjacent PRs. blueorganatan is first step in the process which
provides a way for indicate that a PR is ready for the next step.
We are working towards such a system, but, obviously, there are a lot of pieces
we need to refine/develop and integrate. I am working up a release management
roadmap to capture these ideas in more detail which I hope publish in the near
future,
Thanks,
-John
>
[email protected]
www.shapeblue.com
53 Chandos Place, Covent Garden, London VA WC2N 4HSUK
@shapeblue
On Sep 12, 2016, at 3:29 PM, Sergey Levitskiy <[email protected]>
wrote:
>
> @swill @jburwell From my, non-committer, point of view introducing fully
> automated test system integrated with PR submission and leveraging complete
> list of supported hypervisors is the key to stability. I really like
> blueorangutain idea and if this can be brought back and also cover full set
> of integration tests along with packaging that would be awesome. When we
> develop PR unfortunately we can only test it only with a certain set of
> hypervisors and network configs so although giving test LGTM it doesn’t cover
> all possible integration points. I believe most other test results people
> execute manually and post on PRs are the same.
>
>
> On 9/12/16, 11:38 AM, "jburwell" <[email protected]> wrote:
> Even
> Github user jburwell commented on the issue:
>
> https://github.com/apache/cloudstack/pull/1658
>
> @swill where hardware are being varies by PR. In some cases, we have
> people running them in their labs and reporting results. Other cases, it's
> blueorangatan going through ShapeBlue Jenkins + trillian. But yes, PRs are
> getting tested on hardware.
>
> (Sadly, we are still recovering from a dead storage array, so
> blueorangatan has been out of commission for a little bit),
>
>
> ---
> If your project is set up for it, you can reply to this email and have your
> reply appear on GitHub as well. If your project does not have this feature
> enabled and wishes so, or if the feature is enabled but not working, please
> contact infrastructure at [email protected] or file a JIRA ticket
> with INFRA.
> ---
>
>
>
