Hi Joan, hi everybody,
thank you for this small tale! There are multiple points I'd like to
add some opinions from my side, but first and foremost thank you for
the effort of getting tests and packing automated again! It's not an
easy task and requires a good amount of discipline, and more
important, help from all the sides.
But let's get back to Docker. At my company we're using docker
extensively, and the "tale of two dockers" sort of became a standard
for us: It merely translates into having a Docker image which contains
all the build-tools necessary to build a project and run the tests and
the "runtime image", a minimal image containing only the artefacts
from the compilation step and the necessary libraries/runtimes needed
for running the project. Not only this helps in keeping runtime images
lean, but also the Dockerfiles itself are more comprehensible. Note
that using this approach Docker Hub was never an option for us since
it pushes you into the "do-everything-in-Dockerfile" way of doing.
Now for me it feels a little as if couchdb could go into the same
direction. While couchdb-ci has already all the necessary build and
test steps, those are sort of duplicated in couchdb-docker, only that
in the latter it's a little harder to create all sorts of permutations
of os/erlang version. For me, ideally, couchdb-ci would be able to
create the end-user image just as another form of packaging. This
would also mean that couchdb-docker could disappear at some point.
With the build infrastructure already contained in couchdb-ci, the
Dockerfiles representing the actual "runtime images" will become much
simpler and may find their place in couchb-ci as well.
Hope I could add some valuable information, thanks for reading and
thanks again Joan for the great work around the build and testing
automation!
On Fri, Apr 28, 2017 at 6:52 PM, Joan Touzet wrote:
> Hey everyone,
>
> I had a brief private exchange with Garren about Docker, and figured it
> would be better if everyone could benefit from the email chain. So here
> it is!
>
> The couchdb-docker repository has a self-contained, ready-to-run CouchDB
> configuration. The intent for this is to be used in development
> scenarios, for instance when the Fauxton team is improving Fauxton and
> needs a backing CouchDB against which to test everything. The details
> are here:
>
> https://github.com/apache/couchdb-docker
>
> Currently we use Docker a little differently for the Jenkins CI runs.
> There is some useful overlap between the two approaches.
>
> Current our CI setup is split across Travis and Jenkins. We use Travis
> for (slightly faster) smoke tests across 4 varieties of Erlang, and one
> variety of Linux (Ubuntu). The builds run "make check". Each of these
> builds takes about 15 minutes, and they tend to run in parallel.
>
> Jenkins, on the other hand, tests within custom-built Docker images
> across 2 varieties of Erlang (right now the OS default, and 18.3) and
> across multiple OS variants: Debian x1, Ubuntu x3, CentOS x2) and will
> soon include macOS and FreeBSD builds. Builds on Jenkins are more
> involved - we run 'make dist' to create a tarball, then run 'make check'
> from that tarball. The intent is to simulate the full release process.
> Builds take about 20 minutes as a result.
>
> The Docker images are provisioned using Ansible to keep setup steps
> consistent across different OSes. The details are here:
>
> https://github.com/apache/couchdb-ci
>
> Images are pushed up to the couchdbdev/ namespace at Docker Cloud by
> hand, since they don't change very often. I considered having these
> under the apache/ Docker namespace but since they are more
> dev-consumable rather than end-user consumable it didn't make much
> sense. Garren, klaemo and I all have access to the couchdbdev/
> organization.
>
> Also, I've added the packaging tools to the images for Debian & Ubuntu,
> with CentOS builds not far behind. Most developers won't need these,
> either, but we definitely need them if we intend to build packages for
> each check-in -- which is a direction I'd like to go.
>
> Note that the Docker remote build stuff is transitioning to a paid
> model, see https://docs.docker.com/docker-cloud/builds/automated-build/
> and specifically the text "This hosted build service is free while it is
> in Beta." (Docker Hub is being replaced by Docker Cloud.) We need to be
> careful about leveraging a service that is about to disappear; we may
> have to start building those Docker images and publishing them from
> Apache Jenkins in the same way that I intend to start publishing daily
> packages as well.
>
> I'm very concerned about pushing forward with building anything
> automatically (Docker images, snap packages, Debian/Ubuntu/CentOS
> packages, etc.) until the test suite is fixed. I've been begging for
> help on the mailing lists to no avail. I don't feel right with
> suggesting anyone run anything "nightly" or "direct from master" if any
> tests fail.
