[Pulp-list] Pulp 2.10 Release Candidate 1 Now Available

[Sean Myers]

Pulp 2.10 Release Candidate 1 is now available, and can be downloaded from
the 2.10 beta repositories:

https://repos.fedorapeople.org/repos/pulp/pulp/beta/2.10/

No blocking issues have been discovered since the release of Beta 3, so that 
release has
been promoted to Release Candidate 1. RC 1 should become Generally Available in 
one week
(9/13), but may be pushed back if:

- new blocker issues are discovered
- upgrade testing takes longer than expected
- upgrade testing fails

New features for Pulp 2.10 include the new rsync distributor, new 
high-availability
features for resource managers, as well as some improvements to RPM publishing 
and
package validation. Several bugs were also fixed; all the details can be found 
later
in this announcement.

More information about these new features can be found in the release notes:
http://docs.pulpproject.org/en/2.10/testing/user-guide/release-notes/2.10.x.html
http://docs.pulpproject.org/en/2.10/testing/plugins/pulp_rpm/user-guide/release-notes/2.10.x.html
http://docs.pulpproject.org/en/2.10/testing/plugins/pulp_docker/user-guide/release-notes/2.1.x.html

Try out the new features, and let us know how they work for you!


Upgrading
=

The 2.10 beta repositories are included in the pulp repo files:
https://repos.fedorapeople.org/repos/pulp/pulp/fedora-pulp.repo for fedora 23 & 
24
https://repos.fedorapeople.org/repos/pulp/pulp/rhel-pulp.repo for RHEL 6 & 7

After enabling the pulp-2.10-beta repository, you'll want to follow the standard
upgrade path, with migrations, to get to 2.10:

> $ sudo systemctl stop httpd pulp_workers pulp_resource_manager pulp_celerybeat
> $ sudo yum upgrade
> $ sudo -u apache pulp-manage-db
> $ sudo systemctl start httpd pulp_workers pulp_resource_manager 
> pulp_celerybeat


New Features


Here are the specific stories done for 2.10:

  Docker Support
1887As a user, I can use the rsync predistributor with docker web 
distributor
1291As a user, I can sync feeds that require username/password 
authentication
  Pulp
1990Rsync distributor
1963As a user, I can use rsync distributor to publish iso 
repositories
1759As a user, I can use rsync distributor to publish RPM 
repositores
898 As a user I can run multiple pulp_resource_managers 
concurrently with all of them actively participating
  RPM Support
1991As a user, uploaded units which don't pass the signature check 
are not imported
1982As a user, I can force a full sync
1878Support for choosing the checksum type in updateinfo
1156As a user, I can have an "signature" attribute stored for RPMs, 
SRPMs, and DRPMs

View this list in redmine:
http://bit.ly/2axDGKA


Issues Addressed


Here are the bug fixes specific to 2.10:

  Pulp
2082Cannot add importer to the repository
1948Upgrading RPMs on EL6 sometimes fails during pre script
2187rsync as predistributor logic is wrong
2202predistributor logic in platform does not account for 
force_full and updated distributor configs
2030importer config is not validated during the update
2039Not full importer config is validated in the update call
2118Reduce runtime of file path migration
1606API call to install consumers raises 500 if body contains wrong 
data type
2106Pin to flake8 2.6.2
  RPM Support
2188Make GPG signature checking is called "filtering"
2042updateinfo.xml is missing checksums
2199The RPM rsync distributor breaks when SELinux is enabled
2115The 'skip_fast_forward' in RPM rsync config should be renamed 
to 'force_full'

All bug fixes from Pulp 2.9.2 and earlier are included in Pulp 2.10.

View this list in redmine:
http://bit.ly/2awB1h4


Fedora Support
==

Fedora 22 has reached its end-of-life. As a result Pulp 2.10 is being built for 
(and tested on)
Fedoras 23 and 24.




signature.asc
Description: OpenPGP digital signature
___
Pulp-list mailing list
Pulp-list@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-list

Re: [Pulp-list] puppet_forge on Pulp 2.9.2 400 BAD REQUEST

[Russell E. Glaue]

I need some help.

I spoke too quickly... I tried with another module, and am getting the same 
"400 BAD REQUEST" error.
I still get success with the last puppet module.

But I have some bread crumbs bow.

Here is what I am seeing in Pulp.

With the (previous) successful puppet module update, I see the following in the 
Pulp-Apache access log:
-
10.11.12.13 - - [06/Sep/2016:14:10:59 -0500] "GET 
/v3/releases?module=org-centos_server_base_yum_repos HTTP/1.1" 200 3915 "-" 
"PMT/1.1.1 (v3; Net::HTTP) Puppet/3.8.7 Ruby/1.8.7-p374 (x86_64-linux)"
10.11.12.13 - - [06/Sep/2016:14:10:59 -0500] "GET 
/v3/releases?module=org-centos_server_base_yum_repos HTTP/1.1" 200 3915 "-" 
"PMT/1.1.1 (v3; Net::HTTP) Puppet/3.8.7 Ruby/1.8.7-p374 (x86_64-linux)"
-

With the new puppet module I am attempting to update, I see the following in 
the Pulp-Apache access log:
-
10.11.12.13 - - [06/Sep/2016:14:11:19 -0500] "GET 
/v3/releases?module=org-server_user_mgmt HTTP/1.1" 200 6470 "-" "PMT/1.1.1 (v3; 
Net::HTTP) Puppet/3.8.7 Ruby/1.8.7-p374 (x86_64-linux)"
10.11.12.13 - - [06/Sep/2016:14:11:19 -0500] "GET 
/v3/releases?limit=20&module=org%252Fserver_user_mgmt&offset=20 HTTP/1.1" 400 
23 "-" "PMT/1.1.1 (v3; Net::HTTP) Puppet/3.8.7 Ruby/1.8.7-p374 (x86_64-linux)"
-

I don't know what is different between the two puppet modules that would cause 
puppet update to behave differently.
BUT, notice the URL in the "400" request, it has the URL encoded string "%252F"
The error message puppet module update is giving me has the same URL, except 
for the encoded string of "%2F"

%2F = /
%24 = %

The URL that puppet tells me failed is getting doubly-encoded.
The successful module does not use a '/', but a '-', so that is why it works 
without error.

I have combed through my puppet configuration looking for non-ascii characters 
(grep -P "[\x80-\xFF]") but all are ASCII.

1. Why is puppet using '-' for one module, and '/' for another?
2. When using '/', why is the URL doubly-encoded when the request is made to 
Pulp? I think I remember reading that Pulp encodes the URLs it receives... is 
this true, and be why I am having this issue?

The "name" Pulp has for both is the same: "org/module_name" <- uses a slash in 
the name for both Pulp hosted puppet modules.

Any leads to point me towards an answer?

-RG




- Original Message -
From: "Russell E. Glaue" 
To: "pulp-list" 
Sent: Thursday, September 1, 2016 2:14:14 PM
Subject: Re: [Pulp-list] puppet_forge on Pulp 2.9.2 400 BAD REQUEST

I have resolved my own issue.

I appears that the problem is with puppet, and not with pulp.

In my last upgrade, when puppet was also upgraded, my /etc/puppet/puppet.conf 
file had some configuration reverted to the old environment settings.
regarding the [production] and [development] sections, the one that causes the 
following warning message:
-
Warning: Sections other than main, master, agent, user are deprecated in 
puppet.conf. Please use the directory environments feature to specify 
environments. (See 
http://docs.puppetlabs.com/puppet/latest/reference/environments.html)
   (at /usr/lib/ruby/site_ruby/1.8/puppet/settings/config_file.rb:82:in 
`unique_sections_in')
-

I went back and fixed the configuration to support the newer expected 
configuration for environments, setting environmentpath and basemodulepath 
variables in [main], and now I am no longer getting 400 BAD REQUEST. The 
modules install successfully.

This is really odd though. It makes me think their is some underlying 
compatibility issue when either handling this case specifically, or when 
handling some kind of regression in the request.

-RG


- Original Message -
From: "Russell E. Glaue" 
To: "pulp-list" 
Sent: Friday, August 26, 2016 5:45:04 PM
Subject: [Pulp-list] puppet_forge on Pulp 2.9.2 400 BAD REQUEST

I upgraded pulp from 2.7 to 2.8 to 2.9.2, and now am getting an error when 
performing a "puppet module upgrade" on a puppet server that obtains the puppet 
module from a pulp_puppet repository.

I am using the new syntax, as documented in the source code 
(/usr/lib/python2.7/site-packages/pulp_puppet/forge/views/releases.py)
-
def get(self, request, resource_type=None, resource=None):
"""
Credentials here are not actually used for authorization, but instead
are used to identify:

consumer ID in the username field
repository ID in the password field

This is to work around the fact that the "puppet module install"
command has hard-coded absolute paths, so we cannot put consumer or
repository IDs in the URL's path.
"""
-


Previously with Pulp 2.7 puppet_forge, to upgrade a puppet module I would do 
this successfully.
-
puppet module upgrade org-server_user_mgmt --environment production 
--module_repository 
http://pulp.example.com/pulp_puppet/forge/repository/puppet-org-modules 
--verbose
-
However, in Pulp 2.9.2, I get a "401 UNAUTHORIZED"


So, hence I use the new syntax, as follows, 

Re: [Pulp-list] Easy REST WebUI for Pulp

[Russell E. Glaue]

After reviewing django-admin, I read this: "The [django-]admin’s recommended 
use is limited to an organization’s internal management tool. It’s not intended 
for building your entire front end around."

They do not recommend using django-admin for a primary access user interface - 
only to provide admin-level task access.
So if Pulp does use this, it will still need a general-access WebUI.

-RG


- Original Message -
From: "Russell E. Glaue" 
To: "Michael Hrivnak" 
Cc: "pulp-list" 
Sent: Tuesday, August 23, 2016 4:04:08 PM
Subject: Re: [Pulp-list] Easy REST WebUI for Pulp

The django-admin requires a Python coded application to interact directly with 
the relational database under the hood.
If the pulp community is going this direction, for sure, then why build the 
boat twice by trying ng-admin? Perhaps I just wait for 3.0?

On the other side, ng-admin would work with the existing Pulp REST API. So, 
technically, the community need only build one internal query engine which is 
the REST API. All tools, including the ng-admin WebUI would be built off the 
RESTful interface.

Which would you think will require heavy-hitting DevOps contributors? A Python 
app working under the hood against a RDBM, or a Javascript app on top of the 
hood against a RESTful interface?

Other than perhaps performance, would the Python django-admin provide something 
a RESTful WebUI interface doesn't?
Has any direction been given about what functionality the django-admin app 
would drive? Admin-only, or both Admin/User functions? Or perhaps it would 
provide additional functionality the RESTful interface would not?

-RG


- Original Message -
From: "Michael Hrivnak" 
To: "Kodiak Firesmith" 
Cc: "Russell E. Glaue" , "pulp-list" 
Sent: Tuesday, August 23, 2016 2:09:58 PM
Subject: Re: [Pulp-list] Easy REST WebUI for Pulp


This is wonderful to see! 


One additional thought as we move toward Pulp 3.0 using a relational DB with 
django: We should be able to allow the use of django-admin: 


https://docs.djangoproject.com/en/1.8/ref/contrib/admin/ 



This is a web app that allows basic CRUD operations based on inspection of the 
data model. It is not intended to take the place of a proper user interface, 
but for admin use cases like being able to graphically browse task status 
reports, or create/manage users, it can be very useful. Much about the forms 
and widgets can be customized to make it an intuitive experience. This will be 
another option to explore, to see how useful it can be for basic web UI 
interactions. But of course we have to get to Pulp 3.0 first, which is going to 
be a little while. :) In the mean time I look forward to seeing where things go 
with ng-admin! 


Michael 


On Tue, Aug 23, 2016 at 2:57 PM, Kodiak Firesmith < kfiresm...@gmail.com > 
wrote: 






Awesome to see this might have legs. I'd be interested to see if this can work 
with our current auth setup which uses mod_auth_gssapi (against Active 
Directory) for users/admins, and /root/.pulp/admin.conf for login-less 
programmatic access to Pulp's API (that file is used by pulp-admin to subvert 
the /api/v2/login(?) path and/or use of a certificate. That login url is of 
course owned by mod_auth_gssapi currently, but maybe for something like this I 
could add a 2nd login path that doesn't use mod_auth_gssapi. 

So think about funky edge cases like this when you design the auth scheme 
please :) 

I'm just an end user but I'll gladly buy all the devs of this webapp a beer in 
appreciation. 

- Kodiak 





On Tue, Aug 23, 2016 at 2:45 PM, Russell E. Glaue < rgl...@cait.org > wrote: 


I appreciate all the feedback from the community. It is good to see people want 
something like this, and would support it. 
If there is interest in contributing, we can setup a git repo to facilitate 
contribution. 


I figured out how to get web based authentication from ng-admin to Pulp REST 
API working using Basic auth. 
With this change, you can remove the Apache Reverse Proxy authentication. 

Now, just deploy the ng-admin app, the index.html file I provided earlier, and 
then add this in the index.html file, under the line "var myApp = 
angular.module('myApp', ['ng-admin']);" 

- 
myApp.run(function($http) { 
$http.defaults.headers.common.Authorization = 'Basic YWRtaW46YWRtaW5='; 
}); 
- 
This assumes "admin:admin" as the username and password. 
You can change the (YWR..5=) string with a Base64 encryption of your string in 
the format of "YourUserName:YourPassword" (without quotes) 
Source: https://docs.angularjs.org/api/ng/service/$http#setting-http-headers 


Related - there is an example Angular JS app for doing HTTP basic auth login 
with AngularJS. Should be possible to add the login page as a view inside the 
ng-admin WebUI. 
http://jasonwatmore.com/post/2014/05/26/AngularJS-Basic-HTTP-Authentication-Example.aspx
 
https://github.com/cornflourblue/angular-authentication-example 

-RG 


- Original Message - 
From: "Russell E. Glaue" 

Re: [Pulp-list] repository environments and promoting

[Ina Panova]

There is one optional config parameter for importer 'retain_old_count'. It is
responsible for of how many old rpm versions to retain, unfortunately that
option works just with sync operation. For the same reason 'remove-missing' is
not applicable on copy operation and that's why RPMs that don't exist on source
won't be deleted from dest.

Maybe it would be a good candidate for filling a RFE.

As a workaround you could provide in the destination repository feed of source
repo and retain_old_count option will be applied during sync.
Or simply provide in the feed of destination repo local path --feed file:// and
sync the content into the repo.


Regards,

Ina Panova
Software Engineer| Pulp| Red Hat Inc.

"Do not go where the path may lead,
 go instead where there is no path and leave a trail."


- Original Message -
From: "Brian Bouterse" 
To: pulp-list@redhat.com
Sent: Monday, September 5, 2016 3:40:16 PM
Subject: Re: [Pulp-list] repository environments and promoting

FYI, you can copy a specific RPM if you apply a filter. The filtering is 
2.y is cumbersome so I don't have a ready example, but it can be done. 
You can do it with additional arguments to `pulp-admin rpm repo copy 
rpm`. Those filters correspond with this API endpoint [0]. There are 
also a lot of copy examples for rpm plugin types [1].

I also believe that copying only the latest is an option, but I couldn't 
find that in the docs. Maybe another user or dev can point us to them.

Copy is additive so I expect that RPMs that don't exist on the source 
won't be deleted from dest. A feature Pulp doesn't support is the 
removal of content from the destination with a copy operation. For the 
use case where you want one repo to become exactly like the other you 
could remove all units from the destination and then copy all.

[0]: 
http://docs.pulpproject.org/dev-guide/integration/rest-api/content/associate.html
[1]: http://docs.pulpproject.org/plugins/pulp_rpm/user-guide/recipes.html

-Brian

On 09/05/2016 05:11 AM, Vladimir Vasilev wrote:
> Hi,
>
> I was thinking and testing the same approach, with "copy rpm".
> A few disadvantages:
> you cannot copy specific RPM
> you cannot copy only latest versions.
> RPMs that don't exist on source won't be deleted from dest
>
> Or maybe there's some filters which can do that above.
>
> What I like more is having the prod repositories configured to sync from
> stage with:
> --feed=XXX |--remove-missing=true --download-policy=on_demand
> ||--relative-url=XXX
>
> Then you upload/publish to stage and later just run sync for prod.
> No one must push directly to prod. I just wonder how can I enforce this
> on the server side.
> |
> On 09/05/16 08:28, Vaclav Adamec wrote:
>> Hi,
>>  I'm using Ansible playbooks for deployment and Puppet+Hiera to setup
>> repozitories on servers (right now about 30 repozitories at all, about
>> 5 per server). All servers have disabled live repozitories (for
>> security patches) and enabled assigned stage (dev has live,
>> integration unstable, production stable). Than it's just a pipeline of
>> commands on Ansible playbook like this:
>>
>> # Live repo runs two times per day
>> rpm repo sync run --repo-id=centos_live
>> rpm repo publish run --repo-id=centos6_live
>>
>> #Every week
>> rpm repo copy rpm --from-repo-id=centos6_live centos6_unstable
>> rpm repo publish run ...
>>
>> #Every month
>> rpm repo copy rpm --from-repo-id= centos6_unstable centos6_stable
>> rpm repo publish run ...
>>
>> After publishing Ansible playbook will run update on all servers in
>> given stage.
>>
>> Is that something what do you want to achieve ? As a simple GUI I'm
>> using Jenkins (as a smarter crontab) and ocsreports to get back
>> installed packages and system overview. Pulp server is behind caching
>> Nginx proxies (just RPMs, not metadata). I don't using any kind of
>> registration to Pulp as for dynamic/cloud environment it's more or
>> less stupid idea.
>>
>> Vasek
>>
>>
>> On Sat, Sep 3, 2016 at 12:56 PM, Vladimir Vasilev > > wrote:
>>
>> Hi,
>>
>> I checked the latest pulp docs and can't find this..
>> Is there a way to have environments (dev->stage->prod or any) and kind
>> of promote RPMs to the upper?
>> I see some "content environment" in [1] but the idea is different.
>> There's copy from one repo to another and again to method to copy
>> specific RPMs or latest versions.
>>
>> Looks like juicer [2] is trying to solve this. We use it for one
>> client
>> and it works. Downside is that I'm stuck with 3rd party tool.
>>
>> [1]
>> https://docs.pulpproject.org/plugins/pulp_rpm/user-guide/recipes.html
>> 
>> [2] https://github.com/juicer/juicer
>> 
>>
>>
>> ___
>> Pulp-list mailing list
>> Pulp-list@redhat.com 

Re: [Pulp-list] repository environments and promoting

[Ina Panova]

Hi,

this is the example how to copy one specific RPM :

$ pulp-admin rpm repo copy rpm --from-repo-id zoo --to-repo-id zoo2 
--str-eq='filename=walrus-5.21-1.noarch.rpm'

This command may be exited via ctrl+c without affecting the request.


[\]
Running...

Copied:
  walrus-5.21-1-noarch




Regards,

Ina Panova
Software Engineer| Pulp| Red Hat Inc.

"Do not go where the path may lead,
 go instead where there is no path and leave a trail."


- Original Message -
From: "Vladimir Vasilev" 
To: "Vaclav Adamec" 
Cc: "pulp-list" 
Sent: Monday, September 5, 2016 11:11:16 AM
Subject: Re: [Pulp-list] repository environments and promoting



Hi, 
I was thinking and testing the same approach, with "copy rpm". 
A few disadvantages: 
you cannot copy specific RPM 
you cannot copy only latest versions. 
RPMs that don't exist on source won't be deleted from dest 

Or maybe there's some filters which can do that above. 

What I like more is having the prod repositories configured to sync from stage 
with: 
--feed=XXX --remove-missing=true --download-policy=on_demand --relative-url=XXX 

Then you upload/publish to stage and later just run sync for prod. 
No one must push directly to prod. I just wonder how can I enforce this on the 
server side. 

On 09/05/16 08:28, Vaclav Adamec wrote: 



Hi, 
I'm using Ansible playbooks for deployment and Puppet+Hiera to setup 
repozitories on servers (right now about 30 repozitories at all, about 5 per 
server). All servers have disabled live repozitories (for security patches) and 
enabled assigned stage (dev has live, integration unstable, production stable). 
Than it's just a pipeline of commands on Ansible playbook like this: 

# Live repo runs two times per day 
rpm repo sync run --repo-id=centos_live 
rpm repo publish run --repo-id=centos6_live 

#Every week 
rpm repo copy rpm --from-repo-id=centos6_live centos6_unstable 
rpm repo publish run ... 

#Every month 
rpm repo copy rpm --from-repo-id= centos6_unstable centos6_stable 
rpm repo publish run ... 

After publishing Ansible playbook will run update on all servers in given 
stage. 

Is that something what do you want to achieve ? As a simple GUI I'm using 
Jenkins (as a smarter crontab) and ocsreports to get back installed packages 
and system overview. Pulp server is behind caching Nginx proxies (just RPMs, 
not metadata). I don't using any kind of registration to Pulp as for 
dynamic/cloud environment it's more or less stupid idea. 

Vasek 


On Sat, Sep 3, 2016 at 12:56 PM, Vladimir Vasilev < vvasi...@redhat.com > 
wrote: 


Hi, 

I checked the latest pulp docs and can't find this.. 
Is there a way to have environments (dev->stage->prod or any) and kind 
of promote RPMs to the upper? 
I see some "content environment" in [1] but the idea is different. 
There's copy from one repo to another and again to method to copy 
specific RPMs or latest versions. 

Looks like juicer [2] is trying to solve this. We use it for one client 
and it works. Downside is that I'm stuck with 3rd party tool. 

[1] https://docs.pulpproject.org/plugins/pulp_rpm/user-guide/recipes.html 
[2] https://github.com/juicer/juicer 


___ 
Pulp-list mailing list 
Pulp-list@redhat.com 
https://www.redhat.com/mailman/listinfo/pulp-list 



-- 
-- May the fox be with you ... 
/\ 
(~( 
) ) /\_/\ 
(_=---_(@ @) 
( \ / 
/|/\|\ V 
" " " " 



-- 
Vladimir Vasilev
Senior Systems Administrator
PnT DevOps - System Operations
Red Hat Czech s.r.o., Purkynova 99, 612 00 Brno, Czech Republic
Work: +420 532-294-569
Cell: +420 737-080-404 

___
Pulp-list mailing list
Pulp-list@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-list

___
Pulp-list mailing list
Pulp-list@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-list