Re: [Pulp-dev] Announcing the plugin-template change to use containers for CI

[Mike DePaulo]

On Wed, Sep 4, 2019 at 11:14 AM Dennis Kliban  wrote:

> On Wed, Sep 4, 2019 at 11:11 AM Mike DePaulo 
> wrote:
>
>> *Why?*
>> pulp_rpm, and likely other plugins in the future, have C dependencies.
>> They are often difficult to satisfy on Ubuntu's LTS releases that Travis
>> provides. Since we are developing containers and a Kubernetes operator
>> anyway, we decided to migrate plugin-template's Travis CI scripts to using
>> them. These containers are currently based on Fedora 30, but will likely
>> later be based on CentOS 7.7 or 8.0, which contain python 3.6.
>> This is task #5004 .
>>
>> *Overview:*
>> 1. During install.sh, the container image named like "pulp_file" is built
>> according to pulpcore/containers/
>> 2. During install.sh, k3s, the lightweight kubernetes implementation, is
>> installed & configured on the Travis Ubuntu VM, which already has docker.
>> 3. During install.sh, pulp-operator is deployed (up.sh), which in turn
>> deploys the postgres database, redis, and the pulp_* image you built. It
>> spins the single image up 4 separate containers (services/processes):
>> pulp-api, pulp-content, pulp-resource-manager, and pulp-worker .
>> 4. During script.sh, we install some commonly needed testing tools into
>> the pulp-api container by prefixing the install command with $CMD_PREFIX .
>> So long as this container never gets restarted, they will remain installed
>> in it. These tools are not available to the other containers.
>> 5. We use $CMD_PREFIX to run the unit tests
>> 6. From the Travis Ubuntu VM, we run pulp-smash functional tests, build
>> docs, or use bindings, etc.
>> 7. Note that many other pieces of the puzzle were modified, including
>> pulp-smash, to make this integration all work.
>>
>> *Changes you may need to make:*
>> (in addition to the usual plugin-template commands):
>> 1. Ensure that in template_config.yml, "plugin_name" is set to a value
>> like "pulp_file" rather than "file"
>> 2. If you write out ~/.netrc , the pulp application password has been
>> changed from "admin" to "password" (which other pulp test / development
>> code uses.)
>>
>
> You only need to make this change if you are continuing to use
> ansible-pulp to deploy pulp on Travis. Otherwise, the containers built
> using the latest Travis config will have the right password.
>

Actually, I encountered this during pulp_file's docs tests, which did not
use ansible-pulp. Its docs/_scripts/
 shell scripts
(which get embedded in the docs) are tested via httpie (`http`), and
therefore used the .netrc.
I suppose you could use ansible-pulp in your manually-written
.travis/{pre,post}*.sh scripts to install pulp on the Travis VM (which is
the container host), but I do not recommend it. The templated .travis/*.sh
scripts would still always deploy it in containers also. And I set things
like PYTHONPATH so you don't need to call ansible-pulp for that specific
reason.

>
>
>> 3. git rm .travis/{playbook,postgres,mariadb}.yml
>>
>
4. Append to PYTHONPATH (do not override it, your plugin's dir is in it. It
is set in .travis/script.sh).

>
>> *Planned improvements include:*
>> 1. Speeding up in the image build, largely through 3.
>> 2. Publishing images to quay.io. #5062 
>> 3. Only building your pulp_* image from scratch if you have a required PR
>> for pulpcore or pulpcore-plugin (or pulp-certguard.) Otherwise, your image
>> will be layered on top of the "pulpcore" image. #5062
>> 
>> 4. re-introducing python 3.6 testing alongside 3.7 (Fedora 30 is Python
>> 3.7 based)
>> 5. re-introducing codecov
>> 6. (possible / long-term): Improving how testing packages are installed. 
>> #5404
>> 
>>
>> *Feedback wanted on:*
>> 1. Any confusing/unclear output.
>> 2. Any reliability issues.
>>
>>
>> --
>>
>> Mike DePaulo
>>
>> He / Him / His
>>
>> Service Reliability Engineer, Pulp
>>
>> Red Hat 
>>
>> IM: mikedep333
>>
>> GPG: 51745404
>> 
>> ___
>> Pulp-dev mailing list
>> Pulp-dev@redhat.com
>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>
>

-- 

Mike DePaulo

He / Him / His

Service Reliability Engineer, Pulp

Red Hat 

IM: mikedep333

GPG: 51745404

___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Announcing the plugin-template change to use containers for CI

[Dennis Kliban]

On Wed, Sep 4, 2019 at 11:11 AM Mike DePaulo  wrote:

> *Why?*
> pulp_rpm, and likely other plugins in the future, have C dependencies.
> They are often difficult to satisfy on Ubuntu's LTS releases that Travis
> provides. Since we are developing containers and a Kubernetes operator
> anyway, we decided to migrate plugin-template's Travis CI scripts to using
> them. These containers are currently based on Fedora 30, but will likely
> later be based on CentOS 7.7 or 8.0, which contain python 3.6.
> This is task #5004 .
>
> *Overview:*
> 1. During install.sh, the container image named like "pulp_file" is built
> according to pulpcore/containers/
> 2. During install.sh, k3s, the lightweight kubernetes implementation, is
> installed & configured on the Travis Ubuntu VM, which already has docker.
> 3. During install.sh, pulp-operator is deployed (up.sh), which in turn
> deploys the postgres database, redis, and the pulp_* image you built. It
> spins the single image up 4 separate containers (services/processes):
> pulp-api, pulp-content, pulp-resource-manager, and pulp-worker .
> 4. During script.sh, we install some commonly needed testing tools into
> the pulp-api container by prefixing the install command with $CMD_PREFIX .
> So long as this container never gets restarted, they will remain installed
> in it. These tools are not available to the other containers.
> 5. We use $CMD_PREFIX to run the unit tests
> 6. From the Travis Ubuntu VM, we run pulp-smash functional tests, build
> docs, or use bindings, etc.
> 7. Note that many other pieces of the puzzle were modified, including
> pulp-smash, to make this integration all work.
>
> *Changes you may need to make:*
> (in addition to the usual plugin-template commands):
> 1. Ensure that in template_config.yml, "plugin_name" is set to a value
> like "pulp_file" rather than "file"
> 2. If you write out ~/.netrc , the pulp application password has been
> changed from "admin" to "password" (which other pulp test / development
> code uses.)
>

You only need to make this change if you are continuing to use ansible-pulp
to deploy pulp on Travis. Otherwise, the containers built using the latest
Travis config will have the right password.


> 3. git rm .travis/{playbook,postgres,mariadb}.yml
>
> *Planned improvements include:*
> 1. Speeding up in the image build, largely through 3.
> 2. Publishing images to quay.io. #5062 
> 3. Only building your pulp_* image from scratch if you have a required PR
> for pulpcore or pulpcore-plugin (or pulp-certguard.) Otherwise, your image
> will be layered on top of the "pulpcore" image. #5062
> 
> 4. re-introducing python 3.6 testing alongside 3.7 (Fedora 30 is Python
> 3.7 based)
> 5. re-introducing codecov
> 6. (possible / long-term): Improving how testing packages are installed. #5404
> 
>
> *Feedback wanted on:*
> 1. Any confusing/unclear output.
> 2. Any reliability issues.
>
>
> --
>
> Mike DePaulo
>
> He / Him / His
>
> Service Reliability Engineer, Pulp
>
> Red Hat 
>
> IM: mikedep333
>
> GPG: 51745404
> 
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

[Pulp-dev] Announcing the plugin-template change to use containers for CI

[Mike DePaulo]

*Why?*
pulp_rpm, and likely other plugins in the future, have C dependencies. They
are often difficult to satisfy on Ubuntu's LTS releases that Travis
provides. Since we are developing containers and a Kubernetes operator
anyway, we decided to migrate plugin-template's Travis CI scripts to using
them. These containers are currently based on Fedora 30, but will likely
later be based on CentOS 7.7 or 8.0, which contain python 3.6.
This is task #5004 .

*Overview:*
1. During install.sh, the container image named like "pulp_file" is built
according to pulpcore/containers/
2. During install.sh, k3s, the lightweight kubernetes implementation, is
installed & configured on the Travis Ubuntu VM, which already has docker.
3. During install.sh, pulp-operator is deployed (up.sh), which in turn
deploys the postgres database, redis, and the pulp_* image you built. It
spins the single image up 4 separate containers (services/processes):
pulp-api, pulp-content, pulp-resource-manager, and pulp-worker .
4. During script.sh, we install some commonly needed testing tools into the
pulp-api container by prefixing the install command with $CMD_PREFIX . So
long as this container never gets restarted, they will remain installed in
it. These tools are not available to the other containers.
5. We use $CMD_PREFIX to run the unit tests
6. From the Travis Ubuntu VM, we run pulp-smash functional tests, build
docs, or use bindings, etc.
7. Note that many other pieces of the puzzle were modified, including
pulp-smash, to make this integration all work.

*Changes you may need to make:*
(in addition to the usual plugin-template commands):
1. Ensure that in template_config.yml, "plugin_name" is set to a value like
"pulp_file" rather than "file"
2. If you write out ~/.netrc , the pulp application password has been
changed from "admin" to "password" (which other pulp test / development
code uses.)
3. git rm .travis/{playbook,postgres,mariadb}.yml

*Planned improvements include:*
1. Speeding up in the image build, largely through 3.
2. Publishing images to quay.io. #5062 
3. Only building your pulp_* image from scratch if you have a required PR
for pulpcore or pulpcore-plugin (or pulp-certguard.) Otherwise, your image
will be layered on top of the "pulpcore" image. #5062

4. re-introducing python 3.6 testing alongside 3.7 (Fedora 30 is Python 3.7
based)
5. re-introducing codecov
6. (possible / long-term): Improving how testing packages are installed. #5404


*Feedback wanted on:*
1. Any confusing/unclear output.
2. Any reliability issues.


-- 

Mike DePaulo

He / Him / His

Service Reliability Engineer, Pulp

Red Hat 

IM: mikedep333

GPG: 51745404

___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev