Re: [Pulp-dev] 2.17.0 Dev Freeze - Tuesday, August 7 at 16:00 UTC

[Jeremy Audet]

#3875 was discovered when running tests against nightly RPMs.
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] 2.17.0 Dev Freeze - Tuesday, August 7 at 16:00 UTC

[Jeremy Audet]

I don't think that's the case. This regression was detected by existing
automated tests that were added on September 6th, 2016. See:
https://github.com/PulpQE/pulp-smash/commit/346244cd5cc3aac4ec2551afe1fd38c996a41bf6
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] 2.17.0 Dev Freeze - Tuesday, August 7 at 16:00 UTC

[Jeremy Audet]

https://pulp.plan.io/issues/3875 is a regression and does not appear to be
included in the list of fixed bugs, above.
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

[Pulp-dev] User-facing changes to Pulp Smash

[Jeremy Audet]

Two significant user-facing changes are occurring in Pulp Smash right now.
One change has been merged, and the other is pending.

First, the definition of a "valid" Pulp Smash configuration file has
changed. This has been done so that both Pulp 2 and Pulp 3 can be fully
supported. As a result, some existing configuration files will now be
considered invalid. To fix this, simply install the latest version of Pulp
Smash and run `pulp-smash settings create`. It will create a different form
of settings file depending on which version of Pulp is under test. To see
an example of what it will create for Pulp 3, see here
.

Second, Pulp 2 tests are being moved from Pulp Smash to Pulp 2 Tests
. Scripts that automatically
install Pulp Smash and run its tests must be updated.

I'm happy to assist with this transition. A number of supporting PRs have
already been submitted, such as:

   - https://github.com/pulp/pulp/pull/3553
   - https://github.com/pulp/pulp-ci/pull/561
   - https://github.com/pulp/pulp-ci/pull/562

More will assuredly be necessary.
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] CI not working on RHEL7 (for pulp_rpm)

[Jeremy Audet]

https://github.com/pulp/pulp-ci/pull/557 was also part of this effort.
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Pagination in Pulp

[Jeremy Audet]

FYI, Pulp Smash integrates closely with Pulp's current pagination behaviour:

   - The page_handler
   

   response handler walks through pages of results and collects them into a
   single list. It's used extensively.
   - The pagination test case
   

   "assumes that Pulp returns 100 elements in each page of results. This is
   configurable, but the current default set by all known Pulp installers."

If pagination behaviour is changed, then this will break a significant
number of tests. Also, introducing breaking changes into a fundamental
aspect of Pulp's API during beta would be unfortunate.
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] 'id' versus 'pulp_id' on Content

[Jeremy Audet]

Base URLs should never change. That's an expectation that all web
application clients everywhere should be able to rely on. "Cool URIs don't
change." If anything, storing IDs is the worse practice, because that
implies that the client is going to use pre-existing knowledge to locally
build URLs, instead of asking Pulp for the URLs it needs.
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] 'id' versus 'pulp_id' on Content

[Jeremy Audet]

> I'm -1 on going the underscore idea, partly because of the aforementioned
> confusion issue, but also partly because but I've noticed that in our API,
> the "underscore" basically has a semantic meeting of "href, [which is]
> generated on the fly, not stored in the db".
>
> Specifically:
>
>- '_href'
>- '_added_href'
>- '_removed_href'
>- '_content_href'
>
> So I think if we use a prefix, we should avoid using one that already has
> a semantic meaning (I don't know whether we actually planned for that to be
> the case, but I think it's a useful pattern / distinction and I don't think
> we should mess with it).
>

Outside perspective: My experience with Python, JavaScript, Ruby, C++, and
so on has led me to believe that the leading underscore means "only touch
if you know what you're doing." However, the _href attribute is something
that I, as an end user, have to use all the time. Thus, the lesson I've
taken away from this naming convention is "pulp abuses naming conventions."
I certainly didn't think that the leading underscore means "generated on
the fly" or "some kind of href." Others might think similarly.
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Pulp api seemingly incompatible with generated bindings

[Jeremy Audet]

> +1. Exposing UUIDs is definitely preferable to using hrefs as ids.  "The
app just looks at the relative path"  -> what if pulp wants the flexibility
to change repositories end point (highly improbable but you never know).

Is it better, though? URIs were chosen specifically with immutability in
mind. "Cool URIs don't change." This is reflected in the application's
behaviour. If one changes an object's attributes (e.g. UUID), its href
doesn't change.

And in what case are hostname and port changing? If that's a common
deployment issue, I would contend that the deployment at hand is screwed up.
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] SQlite Issues w/ Pulp3. Switch back to PostgreSQL

[Jeremy Audet]

+1 to switching away from SQLite. I've often pondered a failed test and
wondered "is this a genuine bug, or a side-effect of using SQLite?" This
question is suspiciously similar to "is this a genuine bug, or a
side-effect of using Mongo?"

For more information on SQLite and concurrency, see isolation in SQLite
.
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Pulp 3 Release Process Questions

[Jeremy Audet]

Do packages produced by Travis have to go straight to PyPI? Or is it
possible to build artifacts (like wheels) and later retrieve those
artifacts?

Alternately, are reproductible builds possible for Pulp 3? That is, can you
and I both clone the same source code, check out a particular commit, and
run a one-liner to produce identical build artifacts?

I'm not advocating for any particular solution with these questions. I'm
just exploring the problem space.
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Pulp 3 Release Process Questions

[Jeremy Audet]

Using Travis jobs instead of Jenkins jobs for testing Pulp 3 releases begs
several questions:

   - Are those Travis jobs testing combinations of web servers, AMQP
   brokers, databases, etc? If not, is testing across those combinations a
   goal?
   - Let's say pulp_file is being tested. Is both it and pulpcore installed
   from source? Or is pulpcore installed from egg/wheel?
   - Who is looking at Jenkins jobs? If only QE, why have Jenkins jobs?
   (Possible answer: So that we can trigger test runs on an as-needed basis,
   without making new git commits.)
   - What does this mean for goals like multi-host testing? That is, what
   does this mean for the case where a single Pulp 3 application is deployed
   across multiple hosts? Can that be tested with Travis? Pulp Smash is
   architected with this in mind, and it's been a long standing goal. The
   biggest impediment has been the use of legacy Jenkins (1.x) and nodepool,
   and recent upgrades make this a more realistic testing target.

If these questions have already been considered and answered somewhere,
please point me to the correct resource.
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Pulp 3 release quality

[Jeremy Audet]

> I think running all of them for now is the best way. If the runtime on
Travis becomes unbearable we can look into solutions then.

Smoke tests were created at the behest of developers for precisely this
situation. This line of code:

py.test -v --color=yes --pyargs pulp_smash.tests.pulp3

...can be changed to this:

py.test -v --color=yes --pyargs $(pulp-smash smoke-tests)

Use your test runner of choice. The official unittest test runner has
better support for features like subTests.

> How about running the full test suite nightly to catch things from all
PRs that day rather than waiting until the next release?  Is that being
done?  We should want things to fail sooner rather than later if there's
regressions.

This seems sensible.
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Pulp 3 Unit Test Plan Proposal

[Jeremy Audet]

> I'm also generally -1 against trying to pick a number (100%, 80%, 60%) 
> up-front.  We should unit test what makes sense to unit test, push that 
> number as high as reasonable, and otherwise focus on pulp-smash, which I 
> think has historically been more useful.

QE is flattered by the focus on Pulp Smash. We're happy that the smoke
tests are being executed as a pull request check.

However, it's important to remember that unit tests are far faster
than integration tests, typically by several orders of magnitude. In
addition, Pulp Smash's smoke tests are intentionally limited. They're
designed to execute quickly and to detect catastrophic regressions.
They're not intended to be comprehensive. In fact, some of the
already-written test cases may be stripped of their "smoke test"
status for the sake of speed. Psychology is important here: it's bad
if a developer locally fires off smoke tests, gets bored, and opens up
a new web browser tab.

Please keep this limitation in mind when deciding on policies
regarding smoke tests.

___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Port Pulp3 to use RQ

[Jeremy Audet]

> RQ does not support revoking tasks.  If you send the worker a SIGINT, it will 
> finish the task and then stop processing new ones.  If you send the worker 
> SIGKILL, it will stop immediately, but I don't think it gracefully handles 
> this circumstance.

Nothing handles SIGKILL gracefully. Processes can't catch that signal.
`kill -9 $pid` sends SIGKILL.

If one is looking for a way to gracefully, immediately kill an RQ
worker, then SIGTERM may do the trick. Anecdotally, many processes
handle this signal in a hurried fashion. Semantically, this is
appropriate: SIGINT is the "terminal interrupt" signal (Ctrl+c sends
SIGINT), whereas SIGTERM is the "termination signal."

___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Removing a few sprint items?

[Jeremy Audet]

I should clarify tha my main concern with regards to #2921 is to have *any*
code in place that configures web serves for Pulp 3. Publicising it is a
nice thing too, but at this point in time, my target audience is developers
and QE. On a related note, references to Ansible were dropped from the Pulp
3 documentation.
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Plugin relationship to tasks

[Jeremy Audet]

I reviewed this proposal with Austin, mainly from the perspective of an end
user. In my opinion, this proposal is a good one. I find the semantics
intuitive and I think they do a good job of adhering to the Richardson
Maturity Model
. And
although I don't have a great knowledge of how plugins work, I think this
will grant more flexibility to plugins.
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Pulp3 Ansible Installer Temporarily Removal and Unification

[Jeremy Audet]

I'd like to request that we actually do something different:

   1. For now, leave the ansible-pulp3 installer alone. If someone makes a
   good pull request, we merge it. Otherwise, we leave it alone.
   2. Work on the Ansible code in pulp/devel. Make it satisfy developer,
   QE, and eventually end-user use cases.
   3. Once the Ansible code in pulp/devel is good enough, move it into the
   ansible-pulp3 repository, and publish it as a role. Do future development
   work there.

I'm confident that this will produce some good results. Using this
approach, *I've made the pulp/devel Ansible installer support RHEL 7.* I'm
sure I could also accomplish other goals like making it easy to switch
between AMQP brokers, web servers, etc. I'd love it if y'all could start
reviewing my pull requests against pulp/devel
.
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] pulp 3 PR testing with pulp-smash

[Jeremy Audet]

> If only pulpcore tests are run we could get into a situation where the PR
breaks the plugin API and we don't learn this until after the code is
merged.

This. We want to know if changes to pulpcore break plugins. Intentional
breaking changes should be unusual, and in those cases, I think it's
reasonable to ask the dev making the PR to explicitly state that they're
breaking the product.

Also: KISS. Having just one job is easy to understand and maintain. Only
make it complicated if we need to.
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Release Announcing and Versioning

[Jeremy Audet]

 > Nope, it does not.  THe tooling I'm looking at modifying to use with
pulp from my end can handle that situation, as wel as multiple commits to
an issue.

Great!
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Release Announcing and Versioning

[Jeremy Audet]

> Also it would verify that each issue has an associated commit because the
build machinery will fail if there are fewer commits.

Does this mean that a single commit can't fix two issues? If so, this seems
like a case of the tail wagging the dog.
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Github Required Checks

[Jeremy Audet]

> Regarding the plugin repos, last year we talked about plugins being
completely autonomous (aside from abiding by our Code of Conduct). Wouldn’t
setting the required checks for projects like pulp_file, pulp_python,
pulp_deb, etc violate this autonomy? In other words, shouldn’t we let
plugin teams decide their own policy and what checks to enable?

Are pulp_file, pulp_python, pulp_deb, and so on autonomous projects? The
fact that they're hosted on GitHub under the pulp organization [1]
indicates that they're under our control. Since they're under our control,
we get to set the rules. If any of these projects really are autonomous,
then somebody please kick them out of the pulp organization.

If I was writing paychecks to a team of devs, and they refused to adopt
basic QA processes for their projects, I'd happily fire the entire dev
team. I can't be the only one who's had this thought.

[1] https://github.com/pulp
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Github Required Checks

[Jeremy Audet]

> I _do_ think we need to formalize a set of rules about merging, though,
and decide how strict we want to be about it.  There are a few
possibilities:

I'm only indirectly affected by this decision, so take my opinion with a
grain of salt.

   1. I dislike option 1, because it unnecessarily ties us to a particular
   policy implementation. Yes, it's *nice* to always have green Travis
   reports. But Travis and other service providers break, and their screw-ups
   shouldn't stop us from doing productive work.
   2. I like option 2, because it lets us assert that QA process failures
   must be fixed, without tying oursevles too closely to an unreliable third
   party.
   3. I dislike option 3, because it trains devs to think that QA process
   failures is OK and normal. It's not. Technical debt that affects QA
   processes should always be paid off.

Distinguishing between policy and implementation is tricky. Ask ICANN about
that. But I still think it's a valuable goal to aim for. Here's a sample
policy statement that might apply to this team:

Every PR must pass the checks defined by the `all` make target, for all of
> the interpreters listed in `setup.py`.
>

This policy statement doesn't include implementation details such as:

   - Are these checks automatically executed or manually executed?
   - If automatically executed, which service provider is used? Travis?
   CircleCI? Or perhaps a custom Jenkins or Buildbot application that rents
   VMs from an IaaS provider?
   - Are builds performed on RHEL 7, or in Docker containers based on
   Ubuntu 16.04, or in systemd-nspawn containers based on Arch, or something
   else?

Notably, these implementation details can change, while the policy stays
the same.
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Pulp3 now defaults Qpid

[Jeremy Audet]

> Hi Jeremy, sorry for the very long delay. Yes enabling only for the dev
environments was intentional (issue 2259). I wanted to only have the
developers try it at first as the most basic smoke test during day to day
development. I haven't heard of any reports of issues from the dev
environments, so that is good. For enabling it for Pulp3 users, that is
issue 3134. The overall effort is tracked as epic
https://pulp.plan.io/issues/3291

This clarifies things for me. Thank you.
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Pulp3 now defaults Qpid

[Jeremy Audet]

This change only appears to affect the developer installs of Pulp 3, not
end-user installs of Pulp 3. Is this intentional?
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Pulp 3 installer re-work proposal

[Jeremy Audet]

Any more feedback on this? The current state of affairs is still a daily
drag on the productivity of anyone writing tests for Pulp 3 - people like
me.
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Github keywords

[Jeremy Audet]

> I think we should (if possible) change our redmine automation to:
> closes https://pulp.plan.io/issues/1234
>
> I like this because there is no ambiguity and it will prevent us from
> having to also add the href to the PR as a comment.
>

As a nice bonus, if GitHub *is* used as the issue tracker, then comments in
this form will still automatically close issues.
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Repo version implementation

[Jeremy Audet]

Regarding the second option: What happens if I (as a user) add a content
unit to a repo, later remove it, later add it again, and later remove it
again? Would this result in two "version_added" and two "version_removed"
records?
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Pulp 3 installer re-work proposal

[Jeremy Audet]

Here's what I *meant* to send.


> Jeremy,
>
> I know we discussed this idea before - thank you very much for putting
> together a write up & sending it to the list for discussion. I hope
> you get lots of feedback.
>
> Quick feedback:
> #1. A date of when you would like feedback would be helpful.
>

Sooner is better. There is no deadline for this. It's a case of "the longer
this drags on, the longer the technical debt under discussion will hurt us."

How about wrapping up this discussion by Monday, January 8th? That'll give
us a little over two weeks to discuss this.


> #2. Implementation Strategy section - it would be helpful is you named
> each option something short & sweet, and then refer to them farther
> on. In quickly reading over "this strategy" is a bit hard to keep
> track of which option you are providing thoughts on.
>

Good point. I'll revise it.


> #3. In quick scanning of this write up, I remember you anticipated
> that developer's had different needs than end users and QE. I don't
> see them listed very clearly - have you had a chance to describe your
> knowledge/assumption of what those needs are and how the proposals
> either do or don't address them? I take it that in proposing the 3rd
> option that those concerns are addressed?
>

In short, "the developer installer installs Pulp directly from source in
editable mode, along with numerous extra tools, and it enables settings
like debug mode. The end user installer installs Pulp from PyPI packages,
without extra tools, and without enabling settings like debug mode."

There are some other differences in how developers and end users use the
Pulp 3 installer(s). But I'm intent on resolving those differences by
making the installer(s) more capable. For example, developers currently
deploy Pulp 3 on Vagrant (Fedora or RHEL) hosts, whereas end users
currently deploy Pulp on arbitrary (Fedora or RHEL) hosts. My solution is
to make the dev installer more capable, so that it can install Pulp 3  on
any (Fedora or RHEL) host.


> #4 I'd end your proposal with a quick summary of your proposed solution.
>

To summarize:

   1. Continue distributing Pulp 3 with Ansible.
   2. Enable the following use case:

   git clone https://github.com/pulp/devel.git --branch 3.0-devcd devel/ansible
   ansible-galaxy dev-install.yml -i pulp-dev.example.com,
   ansible-galaxy src-install.yml -i pulp-src.example.com,
   ansible-galaxy pypi-install.yml -i pulp-pypi.example.com,

   3. Get there by evolving the developer installer.



> I'm still not quite clear what next steps are. But I don't have much
> experience in this area. It seems like you are asking for feedback in
> support of the last option, but I am not clear on what next steps or
> what proceeding with support on that proposal would look like.
>

How about this?

   1. Approve the work.
   2. Assign someone to the task. I'm game.

I'm unsure how y'all do this. There's PUPs, several issue trackers, lots of
meetings, etc. Can you help me out here?


> Thanks for sharing your thoughts here.
> Robin
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Pulp 3 installer re-work proposal

[Jeremy Audet]

Gosh darn it, I'm so used to vim that I hit "tab" and "space", which
pressed the send button.

So.

#4 I'd end your proposal with a quick summary of your proposed solution.
>

To summarize:

   1. Continue distributing Pulp 3 with Ansible.
   2. Enable the following use case:

   git clone https://github.com/pulp/devel.git --branch 3.0-devcd devel/ansible
   ansible-galaxy dev-install.yml -i pulp-dev.example.com,
   ansible-galaxy src-install.yml -i pulp-src.example.com,
   ansible-galaxy pypi-install.yml -i pulp-pypi.example.com,

   3. Get there by evolving the developer installer.
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Pulp 3 installer re-work proposal

[Jeremy Audet]

>
> Jeremy,
>
> I know we discussed this idea before - thank you very much for putting
> together a write up & sending it to the list for discussion. I hope
> you get lots of feedback.
>
> Quick feedback:
> #1. A date of when you would like feedback would be helpful.
> #2. Implementation Strategy section - it would be helpful is you named
> each option something short & sweet, and then refer to them farther
> on. In quickly reading over "this strategy" is a bit hard to keep
> track of which option you are providing thoughts on.
> #3. In quick scanning of this write up, I remember you anticipated
> that developer's had different needs than end users and QE. I don't
> see them listed very clearly - have you had a chance to describe your
> knowledge/assumption of what those needs are and how the proposals
> either do or don't address them? I take it that in proposing the 3rd
> option that those concerns are addressed?
>

In short, "the developer installer installs Pulp directly from source in
editable mode, along with numerous extra tools, and it enables settings
like debug mode. The end user installer installs Pulp from PyPI packages,
without extra tools, and without enabling settings like debug mode."

There are some other differences in how developers and end users use the
Pulp 3 installer(s). But I'm intent on resolving those differences by
making the installer(s) more capable. For example, developers currently
deploy Pulp 3 on Vagrant (Fedora or RHEL) hosts, whereas end users
currently deploy Pulp on arbitrary (Fedora or RHEL) hosts. My solution is
to make the dev installer more capable, so that it can install Pulp 3  on
any (Fedora or RHEL) host.


> #4 I'd end your proposal with a quick summary of your proposed solution.
>

To summarize:

   1. Continue distributing Pulp 3 with Ansible.
   2. Enable a



> I'm still not quite clear what next steps are. But I don't have much
> experience in this area. It seems like you are asking for feedback in
> support of the last option, but I am not clear on what next steps or
> what proceeding with support on that proposal would look like.
>



Thanks for sharing your thoughts here.
> Robin
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

[Pulp-dev] Pulp 3 installer re-work proposal

[Jeremy Audet]

All,

I've written up a proposal outlining why I think some changes should be
made to the Pulp 3 installer, what those changes should be, and how we
could make those changes happen. It's attached. I'd love to get some
feedback. Please have a read and write down your thoughts. If the feedback
is positive, we can make redmine issues and start going through the usual
planning processes.

-Jwewmy
Title: Proposal to Consolidate the Pulp 3 Installers






Proposal to Consolidate the Pulp 3 Installers


Contents

What?
Why?
Better Installer, Less Work
Improve Pulp's Public Image
Encourage Community Involvement
More Installation Strategies


How?
Technology
Workflow Example
Sample Playbooks
Sample Role
Why One Variable?
Implementation Strategy





What?
Currently, one installer is used by the Pulp developers, and another installer
is used by Pulp QE and end users. I propose that a single installer be used by
all three audiences. This document explains why I think this change should be
made, and it outlines a possible plan of action for making this change.


Why?
Why should a single installer be used? The most succinct justification is that
using a single installer improves dev/prod parity, where dev/prod parity
means "[keeping] development, staging, and production as similar as possible."
In our case, QE serves as staging, and our users serve as production.
More verbosely, a single installer should be used because doing so will likely:

Produce a better installer.
Reduce the workload on developers and quality engineers.
Improve Pulp's public image.
Encourage community involvement.
Allow for additional installation strategies.


Better Installer, Less Work
To illustrate the first two points, consider the scenario where a quality
engineer discovers an issue with the installer they're using.
The engineer is obliged to check and see if the issue is also present in the
second installer. If so, the engineer will either file one ticket that mentions
both installers, or they will file two independent tickets, depending on their
mindset. When a developer decides to work on the issues, they need to develop
two similar fixes for two similar installers, they need to test both changes,
and they need to file two pull requests. After the pull requests have been
merged, the developer needs to either close out one or two tickets.
This workflow is logically sound. So long as all parties are mindful of what
they're doing, nothing will go wrong. But humans aren't like that! Consider the
case of Pulp #3031. That ticket describes how the end user installer inserts
junk text into /etc/pulp/server.yaml. I filed this ticket on September 25th,
and I submitted a fix for the issue on the same day. Unfortunately, I forgot to
file a second ticket noting that the exact same issue is also present in the
developer installer. As of this writing on November 21st, the developer
installer is affected by this issue. The developer installer still works due to
the particular nature of the bug, but at any time, an unrelated change to the
configuration file could break the development environment.


Improve Pulp's Public Image
To illustrate the third point, consider what happened when the Pulp 3 plugin API
alpha was released. The announcement was pushed back by a day or two so that QE
could have a chance to play with the product before it went out the door. QE
discovered several issues, one of them being the fact that the end user
installer simply did not work. (The issue was fixed by pulp/devel
396c93e775542484b585b8841bc9adc042b71731.) This gives the impression that the
end user installer is a second class product. So long as we have two
non-orthogonal installers, Pulp's public image is vulnerable to incidents like
this.


Encourage Community Involvement
To illustrate the fourth point, consider some of the differences between the end
user and developer installers. Using the end user installer requires the
following:

Decide where you would like to install Pulp. You could install it on your
local system, on a beaker system, on a libvirt + kvm VM, on a libvirt +
virtualbox VM, on a Linode VM, on a DigitalOcean VM, or just about anywhere
else.

Get shell access to the system and execute the following:

setenforce 0
systemctl stop firewalld  # or open appropriate ports


From your local system, execute the following:

git clone https://github.com/pulp/devel.git --branch 3.0-dev
cd devel/ansible
ansible-galaxy install -r requirements.yml -p roles
ansible-galaxy deploy-pulp3.yml -i pulp3.example.com,



In contrast, using the developer installer requires the following:

Set up Vagrant. A development environment can't be deployed on any other type
of host. You're SOL if you'd like to use some other type of back-end.

From your local system, execute [1] the following:

git clone https://github.com/pulp/devel.git --branch 3.0-dev
cd devel
vagrant up



Those familiar with Pulp's installers might also notice that the end user
installer doesn't install any 

Re: [Pulp-dev] Proposal and feedback request: un-nest urls

[Jeremy Audet]

>
> For deletes, the db relationships are all there, so I expect deletes to
>> cascade to other objects with any url
>> structure. I believe closer to the release, we'll have to look at the
>> cascading delete relationships to see if
>> the behaviors that we have are correct.
>>
>> Overall, I'm +1 on un-nesting. I think it would result in a good user
>> experience. I know it goes against the
>> logical composition arguments, which have been well laid out. We want
>> Pulp to be really simple, and the nested
>> URL in the top of this thread is anything but simple. Consider another
>> project like Ansible Galaxy (who also
>> uses Django and DRF). Their API is very flat and as an outsider I find it
>> very approachable:
>> https://galaxy.ansible.com/api/v1/  Pulp could be that simple.
>>
>
> Clicking through the Galaxy API, there seems to be a good bit of nesting.


There is. Let me be explicit. Here's some of the available URLs:

   - /api/v1/notifications/
   - /api/v1/notifications/120165/imports/
   - /api/v1/imports/
   - /api/v1/imports/166952/notifications/

In a filter-heavy world, they could be written as:

   - /api/v1/notifications/
   - /api/v1/imports/?notification=/api/v1/notifications/120165/
   - /api/v1/imports/
   - /api/v1/notifications/?import=/api/v1/imports/166952/

Or:

   - /api/v1/notifications/
   - /api/v1/imports/?notification__id=120165
   - /api/v1/imports/
   - /api/v1/notifications/?import__id=166952

...or something else.

There's more than one way to structure a RESTful HTTP API. I've had (very
limited) positive experiences with flattended out APIs, and like them
because they leave the door open to new use cases with little change. But
nested URLs are also perfectly valid, as evidenced by literature
 from
the likes of Martin Fowler.

I guess what I'm trying to say is: I think either approach is perfectly
valid, and concerns like "what do I find easy to conceptually understand"
or "what's easy to implement under the hood" are valid criteria for making
a choice.

2c from a non-dev. :-p
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Pulp 3: using JWT to request a JWT

[Jeremy Audet]

Good points.

> Another scenario: someone tcpdumps my traffic (yes, somehow they have the
SSL cert, work with this assumption for now). They can come back 3 days
from now, browse the tcpdump output, and renew the token. That would not be
possible with a short-lived token and no renewal past expiration.

Renewal with expired tokens isn't being proposed. This is a straw man
argument.
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Pulp 3: using JWT to request a JWT

[Jeremy Audet]

>
> Jeremy, I don't think I understand your comment.
>
> You *will* have to use basic auth to refresh the token when the original
> one expires.
>

Right. I understand that. I'm not arguing that we allow the user to
generate a valid JWT token using an expired, invalid JWT token. I'm arguing
that we allow the user to generate a valid JWT token using an unexpired,
valid JWT token. This allows use cases such as a client (web browser, CLI
tool, etc) being able to re-authenticate itself without re-prompting the
user for a username and password. This is especially relevant if the token
expiration time is set to a short value, such as 15 minutes.

So there are limitations to a JWT, and for good reasons. A JWT is a weaker
> authenticator than a username+password because it expires. Because it is
> timestamped, it reduces the risk of compromising your account if someone
> sniffs the traffic.
>

If there's security concerns here, then that's important, and they should
be weighted heavily.

Note that there's an easy-to-use mechanism for invalidating a user's tokens.


> Refreshing the token with a JWT seems marginally useful to me.
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Pulp 3: using JWT to request a JWT

[Jeremy Audet]

+1. I think one should be able to get a JWT with a JWT. This user
experience:

> I can authenticate any API call with a JWT token.

...is nicer than this user experience:

> I can authenticate any API call with a JWT token. Oh, wait, exept getting
a new JWT token. I wonder why? Is there some security risk here? I wonder
if there's other API calls that also don't let me use JWT tokens? Perhaps I
should use basic auth for all authentication?
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Proposal and feedback request: un-nest urls

[Jeremy Audet]

>
> Cons:
>
>1. Lost investment in nesting.
>2. Different structure than Pulp 2.
>
> The "lost investment in nesting" con isn't a con. This is an example of
the the "sunk cost" fallacy.

I don't think it matters much if our URL structure is different than in
Pulp 2. Pulp 3 is already massively different. This is a small detail
compared to the other changes being made. I don't think anyone will start
switching to Pulp 3 and suddenly shout "oh no, my API client doesn't work
because of the lack of nested paths!" If they do, the bigger issue is that
their API client is bad.

I've written some small API-focused applications in the past, and I've
found that nested URLs were problematic because they enforced strict
sub-set relationships that often became problematic in the future. For
example, if I have this URL structure:

/api/v1/people/{person_id}/items/{item_id}

...then it's easy to get info about the items a person has. But what if I
want to find out which of my party members has concentrated sprayflesh,
because a band of cloned delusional megalomaniacal robots is about to
successfully murder a teammate, and I need to heal them *now*? Do I need to
make four API calls?

GET /api/v1/people/1/items?name=Concentrated%20Sprayflesh
GET /api/v1/people/2/items?name=Concentrated%20Sprayflesh
GET /api/v1/people/3/items?name=Concentrated%20Sprayflesh
GET /api/v1/people/4/items?name=Concentrated%20Sprayflesh

Worse, let's say I'm a developer debugging an issue, and I want to find
every concentrated sprayflesh in the known world. Do I need to make several
thousand API calls? Both of these use cases are more elegantly solved by
allowing users to work directly with items, and by providing good filtering
support. This is based on my experience with some small old apps though,
and none of them went into production. Take it as you will.
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Proposal to replace pulp 2.15's nightly jobs

[Jeremy Audet]

> I'm assuming that with the Pulp 2 - Master UI tab, that there will be
lots of job here.

I think the "Pulp 2 - master" UI tab will have four jobs, with names like:

   - fedora-24-pulp-2-master
   - fedora-25-pulp-2-master
   - fedora-26-pulp-2-master
   - rhel-7-pulp-2-master

Patrick?
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Request to explicitly stop 2.14 at 2.14.3

[Jeremy Audet]

Do you think it will be possible to push an emergency 2.14.4 build out the
door if necessary? Or an emergency 2.13.z build? I love the idea of
throwing away old processes that are weighing us down. But there are
business needs to consider.
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Proposal to replace pulp 2.15's nightly jobs

[Jeremy Audet]

>
> Hosting packages in just one place is simpler than hosting packages in
>> multiple places. There's
>> less room for error when the simpler thing is done.
>>
>
> It shouldn't be too hard to set up.
>

Fair enough. I also think that hosting packages on one location helps to
prevent end-user confusion. But we can host packages wherever is
appropriate, and I don't have a terribly strong opinion here.


> I would probably want to keep it in a 'nightly' or 'master'
> folder instead of a versioned folder, to help aign the intent of
> explicitly distinguishing this
> workflow from others.  Thoughts?


Yes, please. If there's a directory called "2.15," then I think that
there's a 2.15 release. If there's a directory called "nightly" or
"master," then I think that there are nightly builds, or builds from master.

To nitpick: I like the idea "master" a little bit more. What if we improved
our development and build processes so that there were two builds in a day?
"master" reflects the idea that the builds come from the master branch,
whereas "nightly" reflects the idea that there's one build per day.
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Proposal to replace pulp 2.15's nightly jobs

[Jeremy Audet]

On Thu, Nov 9, 2017 at 4:58 PM, Patrick Creech  wrote:

> As part of the ongoing release engineer changes, I would like to propose
> utilizing the pulp-
> packaging* jobs in jenkins for nightlies going forward and turning off the
> build-automation jobs
> starting monday.
>

We have any releases in progress. The next beta build will be 2.14.3 on Nov
14th. This seems like a great time to switch to something new.


> I would also like to propose switching from having the next .y version in
> the name for master to
> just utilizing 'master' (or 2-master, to account for pulp2 vs pulp3).
> This will allow us to have an
> always up to date 'pulp master' tab that has accurate reporting of the
> latest code status instead of
> having to depend on someone setting up a new tab accurately at .y time.
>

Names are important. UI is important. Let's please have a "master" tab
instead of a misleading "2.15" tab.


> Something of note, the location for pulp-packaging's nightly rpms are at
> http://koji.katello.org/rel
> eases/yum/pulp-nightly/pulp/ (bits are tested there).  Would this be ok
> as the current new place for
> nightly rpms, or would we want to sync them over to repos.fedorapeople.org
> ?
>

Hosting packages in just one place is simpler than hosting packages in
multiple places. There's less room for error when the simpler thing is done.


> I'm also open to suggestions for more reflective job names than
> 'pulp-packaging*'
>
> You're friendly neighborhood Release Engineer
> Patrick
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Webserver owning the entire url namespace?

[Jeremy Audet]

>
> All of that said, there's no reason why a user couldn't use a web server
>> like httpd to run all three WSGI apps in the same process, multiplied
>> across its normal pool of processes. We should make the apps available as
>> separate WSGI apps, and users can deploy them in whatever combinations meet
>> their needs.
>>
>
> As mentioned above, Pulp should use configuration settings to disable and
> enable the REST API and the individual content APIs. Separate WSGI
> applications makes the deployment process more complicated.
>

...

Don't we also want to support having a single WSGI process serve both
> content and the REST API? There are practical reasons why users may want to
> deploy this way too. For example having a smaller memory footprint by
> having fewer process groups, or maybe they just want a simpler deployment.
> If we ship one WSGI application with Pulp then we allow for both deployment
> models (together or separate REST API and content serving). Users who want
> to use separated WSGI processes should configure the webserver to
> instantiate the one WSGI application Pulp would ship twice, and route
> content urls one WSGIProcessGroup and the REST API calls to another
> WSGIProcessGroup. We could document that in a deployments page which would
> be cool.
>
> So for ^ reasons I think having Pulp ship one WSGI application that could
> handle all Pulp urls would allow for the most deployment models.
>

Can someone write up an nginx configuration to show what these different
deployment options would look like in practice? I ask for this because some
of the assertions made in this thread don't match my practical experience.
For example, if I understand correctly, one proposed deployment model is
for WSGI applications to run directly in a web server's memory space. (That
is, one proposed deployment model is for WSGI applications to run as part
of a web server's process.) If my understanding is correct, then I find
this very strange. Doing so introduces unnecessarily tight coupling between
different services, making it impossible to do simple things like restart
one WSGI service while leaving others up. Furthermore, in my experience,
it's easy to run multiple WSGI applications behind a web server.

Here's a concrete example. Let's say I have two web applications that share
a single SSL certificate, and that are available at different paths. Here's
a snippet of an appropriate nginx configuration:

server {
listen 80;
server_name app.example.com;
return 301 https://$host$request_uri;
}
server {
listen 443 ssl;
server_name  app.example.com;
ssl_certificate  ssl/app.example.com.chained.crt;
ssl_certificate_key  ssl/app.example.com.key;
ssl_protocols TLSv1 TLSv1.1 TLSv1.2;  # SSLv3 is insecure
location /user-1/ {
proxy_set_header  Host   localhost;
proxy_set_header  X-Forwarded-For$proxy_add_x_forwarded_for;
proxy_set_header  X-Forwarded-Proto  $scheme;
proxy_set_header  X-Real-IP  $remote_addr;
proxy_pass http://127.0.0.1:8384/;
}
location /user-2/ {
proxy_set_header  Host   localhost;
proxy_set_header  X-Forwarded-For$proxy_add_x_forwarded_for;
proxy_set_header  X-Forwarded-Proto  $scheme;
proxy_set_header  X-Real-IP  $remote_addr;
proxy_pass http://127.0.0.1:8385/;
}
}

One especially nice thing to notice about this configuration is that the
proxied-to applications are completely opaque. They could be WSGI
applications (e.g. Gunicorn ), or they could be Go
applications, or they could be Java applications, or so on. It's a nice
decoupling of concerns.
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Webserver owning the entire url namespace?

[Jeremy Audet]

+1 to requiring that Pulp be able to operate underneath some path.
Well-behaved web application should support this. It's a common
expectation, and we should live up to common expectations.

For a real world example of how to make an application behave well
underneath a base path, see Syncthing's Reverse Proxy Setup
 documentation.
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Twitter Announcements for Release

[Jeremy Audet]

>
> @bizhang, that sounds great! Thank you. Speaking of alpha pushes to PyPI,
>> does it make any sense to also email pulp-dev when its pushed also? If not
>> that's ok too.
>>
>
> I don't use twitter, so emails are useful.
>
> If PyPI supports Atom or RSS, that'd also work. But I'm not aware of them
> supporting that.


What if Twitter supports Atom or RSS? 

We can do just Twitter. I don't think there's a huge impact either way. I'm
just pointing out the value of announcing information through a
standardized, non-proprietary mechanism, especially of the "pull" variety.

I'm going to drink some black coffee while reminiscing about the bad old
days of Newgrounds and LMR-based ATA hard drives. Mumble grumble grumble.
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Twitter Announcements for Release

[Jeremy Audet]

> @bizhang, that sounds great! Thank you. Speaking of alpha pushes to PyPI,
does it make any sense to also email pulp-dev when its pushed also? If not
that's ok too.

I don't use twitter, so emails are useful.

If PyPI supports Atom or RSS, that'd also work. But I'm not aware of them
supporting that.
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] [pulp 3] proposed change to publishing REST api

[Jeremy Audet]

> 2) Have the user find the publication via query that sorts on time and
filters only for a specific publisher. This could be fragile because with a
multi-user system and no hard references between publications and tasks,
answering the question "which is the publication for me" is hard because
another user could have submitted a publish too. While not totally perfect,
this could work.

A "no hard references" design is used in part of Pulp 2, for tracking tasks
that result from a scheduled pulp-consumer task. The design is extremely
problematic. It prevents QE from testing scheduled pulp-consumer tasks.
Thankfully, pulp-consumer is going away, so it's not a huuuge problem with
Pulp 2.

If Pulp QE is to test a feature in Pulp 3, please avoid a "no hard
references" design for that feature.
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] [pulp 3] proposed change to publishing REST api

[Jeremy Audet]

>
> As an example, HAL is one common way to do this:
>
> http://stateless.co/hal_specification.html


HAL looks neat. It's a reasonably well defined standard for defining HTTP
responses, in much the same vein as JSON schema.

If it's adopted, the one thing I'd suggest is to avoid the use of CURIEs.
It's true that there's an (expired) W3C proposal for them. However, using
CURIEs would add a non-trivial amount of complexity into both server and
client applications. Additionally, I don't see any libraries supporting
CURIEs, either in the Python standard library or on PyPI. The only benefit
in our case seems to be a slight reduction in response sizes.
(Content-encoding can do that better, anyway.)

Importantly, HAL and its brethren don't solve the problems discussed in
this thread. It's merely a standard for presenting responses. It doesn't
help y'all define what requests are available to clients.
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Pulp Packaging Redesign

[Jeremy Audet]

> I wrote a story for us to create a packaging guide here:
https://pulp.plan.io/issues/3092

Cool!
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] [pulp 3] proposed change to publishing REST api

[Jeremy Audet]

>
> Normally in REST when a user creates a resource via POST to a collection
>> endpoint, they are expected to provide a representation of the new
>> resource, even if it is only partial.
>>
>
>>
> No, not true.
>

Sorry, Michael. What you said *is* true. I'm just aiming to add some
additional depth to that statement, and my contrariness isn't warranted.

If one wishes to perform a partial update of an existing resource, then
> it's appropriate to make an HTTP POST request whose body is a partial
> representation. But there's more than one way to use POST. If one wishes to
> create a new resource without specifying the path to that resource, then
> it's appropriate to make an HTTP POST request, whose body is completely
> opaque.
>
> HTTP POSTs are an escape hatch. You can do all sorts of crazy things with
> them, and nobody can cache the request or response.
>
> Slight tangent, but have I pointed out that one can create a resource with
> an HTTP PUT? It's entirely correct to do so. The only interesting
> restrictions are that:
>
> * The body of that request must contain a complete representation of the
> resource being created.
> * The full path to the resource being created must be specified.
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] [pulp 3] proposed change to publishing REST api

[Jeremy Audet]

> Once the task has completed, the user should be free to forget the task
ID and be able to use natural keys to find and inspect resources that got
created/updated.

I sure hope we won't be asking the user to build URLs themselves. That
promotes a much tighter coupling between clients and API versions, making
client-side code more complex, and making server-side upgrades and changes
harder. Making a discoverable API is hard, but there are pay-offs.
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] [pulp 3] proposed change to publishing REST api

[Jeremy Audet]

> 'example' is an identifier for the publisher type

So you might have two "rpm" publishers, for example? OK.

> Normally in REST when a user creates a resource via POST to a collection
endpoint, they are expected to provide a representation of the new
resource, even if it is only partial.

No, not true.

If one wishes to perform a partial update of an existing resource, then
it's appropriate to make an HTTP POST request whose body is a partial
representation. But there's more than one way to use POST. If one wishes to
create a new resource without specifying the path to that resource, then
it's appropriate to make an HTTP POST request, whose body is completely
opaque.

HTTP POSTs are an escape hatch. You can do all sorts of crazy things with
them, and nobody can cache the request or response.

Slight tangent, but have I pointed out that one can create a resource with
an HTTP PUT? It's entirely correct to do so. The only interesting
restrictions are that:

* The body of that request must contain a complete representation of the
resource being created.
* The full path to the resource being created must be specified.
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Pulp Packaging Redesign

[Jeremy Audet]

> The goal here is to move release engineering content out of the project
repos and into something purpose built for release engineering, decoupling
the rpm generation process from the python codebase while also helping aid
in the consumption of pulp bits for various downstream projects.

What's this? Loose coupling? Single responsibility principle? Say no more!
I'm on board.

More seriously, though, I would love to see the level of complexity
surrounding Pulp's release engineering reduced. If I was to try packaging
up Pulp for distribution on some other Linux distribution as in independent
third party, I'd quickly feel overwhelmed by how the various bits and
pieces are spread across all of the pulp and pulp_* repos,
pulp/pulp_packaging, pulp/packaging (and all of its branches!), etc. And I
say this as someone who's been around the product full-time for a couple
years now. This proposal seems sane, it targets those concerns, and it
comes from someone who's highly qualified. If QE needs to do anything, say
so.
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] [pulp 3] proposed change to publishing REST api

[Jeremy Audet]

> http://localhost:8000/api/v3/repositories/foo/publishers/
example/bar/publications/


AFAIK, the form of this URL is:
scheme://netloc/api/v3/repositories/{repository_id}/publishers/example/{publisher_id}/publications/.
Is "example" supposed to be in this URL? I don't see what purpose it
serves.
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] REST API Milestone/alpha?

[Jeremy Audet]

I don't see any API endpoints for getting JWT tokens. How will
authorization be handled?
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Publishing an Alpha

[Jeremy Audet]

> Installing direct from git would be great, but don't we break that by
having our python packages live not at the root of the repo? Or has pip
gotten smarter about that?  If it hasn't, requiring someone to clone a
local copy of the repo and then pip install from there wouldn't be too bad,
since it's a plugin API alpha intended for developers.

You can install subdirectories straight from git.

git+
https://github.com/pulp/pulp.git@3.0-dev#egg=pulpcore=platform
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] pulp3: Publishing Proposal

[Jeremy Audet]

I don't have a deep knowledge of Pulp's architecture, so my opinion
shouldn't be taken too seriously. That said, this proposed architecture
makes sense to me. If I was creating a web application from scratch, I'd
pursue an architecture in this vein. No symlink farm on disk. No
/var/lib/pulp/published/* directory hierarchy that mirrors the available
URLs. Heavier use of the database to track content.

Beyond the changes to logic, I suspect that a change like this would
improve performance. I don't dare run Pulp on a spinning rust hard drive
because of the high I/O demands. (It gets worse in my case where I run 6+
Pulp instances!)
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Terms: unassociate vs. disassociate

[Jeremy Audet]

> As an end-user I agree with the add/remove lexicon being more clear to
users, if not more technically accurate.

Same. Either phrasing gets the message across, but IMO, "add content to a
repository" is more unambiguous.
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] python namespace proposal

[Jeremy Audet]

This all seems reasonably sane. +0.
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] PyPI names for Pulp3

[Jeremy Audet]

> 2) pulp3
> - is ambiguous regardless of if the 3 is for Pulp3 or Python3.

Yes. Also, using "pulp3" makes it easy for a naming conflict to occur with
the existing PuLP  package. What happens
when their project also gets to version 3? Or when their project gets to
version 4, and downstream packagers want to have a separate package for
versions 4 and 3 of PuLP? One could say that this won't be an issue on
account of the differing capitalization of the projects' names. But I don't
think that capitalization is a good differentiator, as demonstrated by the
NetworkManager project.

And less importantly, using "pulp3" will force us to switch to "pulp4,"
"pulp5," etc as new major versions of Pulp come out.
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] PyPI names for Pulp3

[Jeremy Audet]

Beware that using a name of "pulp3" can cause packaging issues. Consider
this scenario:

   1. The (non-Red Hat) pulp application jumps from version 3 to 4.
   2. Downstream packagers upgrade their (non-Red Hat) "pulp" package from
   version 3 to 4.
   3. Downstream packagers create a "pulp3" package that provides version 3
   of the (non-Red Hat) pulp application.

This scenario I just outlined isn't weird or unusual. For example, on my
system, I can install the following packages:

   - python (Python 3.6.0)
   - python35 (Python 3.5.3)
   - python34 (Python 3.4.6)
   - python33 (Python 3.3.6)
   - python32 (Python 3.2.6)
   - python30 (Python 3.0.1)
   - python2 (Python 2.7.13)

How weird would it be if someone started distributing ann unrelated
"python4" package?

By distributing the (Red Hat) pulp application under the name pulp3, we
make life harder for downstream package maintainers.
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] 2.12.2 potential blockers

[Jeremy Audet]

I've walked through the remaining test failures for the 2.12 nightlies, and
I don't see any remaining issues in Pulp itself. More precisely, there are
three test failures produced by the automated test suite. Two will be fixed
by https://github.com/PulpQE/pulp-smash/pull/577, and the third will be
fixed by a future PR.
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] RFC process

[Jeremy Audet]

> Because i cannot guarantee that i will not miss somehow the email or
etherpad. I actually often find myself trying to dig through dozens of
mails to find the right one. Same with the etherpads :)

I have trouble finding etherpads even for regular meetings.
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] release process updates

[Jeremy Audet]

Thank you, Sean! (And I like the change in wording.)
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Pulp multinode testing

[Jeremy Audet]

>
> My question is, what is the environment we would like to test Pulp on a
> multi machine environment? I remember someone mentioning about having 3
> machines:
>
> 1. Pulp server
> 2. Pulp workers
> 3. Pulp Smash
>
> Do you think is that ok? Or should another architecture be considered?
>

That seems fine to me. We could even run Pulp Smash on the Pulp server or
on a Pulp worker, so long as Python 3.4+ is available.

* Update Pulp Smash to manage workers services on a different machine than
> the Pulp server if it is configured to do so.
>

I think this is a good way to move forward. I'm confident that Pulp Smash
can be updated so that it can handle multi-node testing in a graceful way.
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Merging forward commits

[Jeremy Audet]

As someone who has little stake in the matter but lots of opinions:
cherry-picking seems way easier. The merge-forward procedure makes my brain
melt.
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] release process updates

[Jeremy Audet]

> - Get bonus points for retroactively setting the "Verified" flag on
existing
>  issues that had status "VERIFIED" before moving to "CLOSED"​

Bonus points *galore* for this.
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] release process updates

[Jeremy Audet]

David, does my response answer your question?

Ina, same to you: does my response answer your question? Let me recap a few
points:

   - Pulp releases currently go out the door with un-verified (but fixed)
   blocker issues. None of these proposals change that policy.
   - If a developer wants a release to be held up until an issue is
   verified, they currently must email pulp-qe-list and contact (via IRC? or
   email?) the release engineer. Furthermore, finding out which issue(s) are
   holding up a release requires reading an email thread and/or pinging the
   release engineer, then looking at each affected issue. That's doable, but
   it would be easier if the information was tracked in redmine: holding up a
   release until a issue is verified would require nothing more than changing
   a field on an issue in redmine, and finding out what's holding up a release
   requires a redmine query.

Also, I appreciate the intent behind requiring that refactors or blockers
be verified before a release can go out, but let's bear in mind the
consequences of such a decision. Consider the current 2.12 release: QE is
preoccupied with fixing automated tests that broke as a result of the
repository layout changing, so that we can provide assurance that no
regressions occurred. That alone may push back the release by a day or two.
Requiring verification for all refactors/blockers/etc regardless of their
importance will almost certainly make such delays commonplace. It will also
stress QE's ability to be flexible and occasionally take on other projects.
(For example: I recently took a course on Ansible through RHU. I definitely
spent less time verifying issues while doing that.)

-Jeremy
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] release process updates

[Jeremy Audet]

> I’m not sure I completely understand why we should not be having QE
verify Refactors. What if we’re worried about introducing regressions with
a Refactor?

Consider the ways in which a refactor might unintentionally change Pulp's
behaviour. A refactor might introduce a race condition, or accidentally
shadow a variable in a different scope, or introduce a typo that causes
tracebacks inside a try/except block, or something else. How do you test
that?

Refactors are an internal change that should have no visible effect, and I
think the best way to test them is to just run the test suite we already
have. It seems more productive for QE to write tests for existing and new
features and bugs than to guess the ways in which a refactor might
unintentionally change Pulp.

> How can we state that the blocker was fixed

The same way we state that any other bug has been fixed: by marking it as
MODIFIED. Fixing a bug is different from having QE verify that it's fixed.
The addition of a "verified" field in redmine would let QE give an explicit
stamp of approval.

> and in addition what's then the point then to make the issue as blocker
and fix it in tight deadlines.

Blocker bugs determine whether or not a release can move forward or not.
It's purely a developer tool. To be clear, QE currently ignores the
'blocker' field. The status quo is to verify all bugs on a best effort
basis. Releases currently go out the door with un-verified blocker bugs.
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] RFC: Use Napoleon when Writing Docs

[Jeremy Audet]

I've really got to butt in here and say that y'all may have a
mis-understanding of what Sphinx lets you do. I'm seeing code examples like
this:

:param repo_name: The name of the repo
:type  repo_name: basestring

And criticisms like this:

> I, personally, am not a big fan of the ":param foo:" syntax.  To me, it
puts way to much information
> spread across multiple lines, increasing the length/scrolling/brainpower
needed to parse such an
> item.  I have to read multiple lines just to figure out what type  a
variable has.

But Sphinx also lets you document parameters like this:

:param basestring repo_name: The name of the repo

So - definitely consider switching to Napoleon or whatever you like. That's
cool. But please, fully understand what Sphinx offers.

-Jeremy
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev