[Pulp-list] Sept 23 Triage delayed
Due to a conflict with another meeting, tomorrow's triage session (or today's, depending on where you are) is going to happen a little bit later in the day. Triage is normally scheduled for 10:30 AM, Eastern Time. The conflicting meeting ends at 11, so we'll give folks time to get settled after the meeting and most likely start triage an hour later, at 11:30 AM in the usual place (#pulp-dev on freenode). Apologies for any inconvenience. If that time doesn't work out, I'll update here with the details. signature.asc Description: OpenPGP digital signature ___ Pulp-list mailing list Pulp-list@redhat.com https://www.redhat.com/mailman/listinfo/pulp-list
Re: [Pulp-list] Tasks stuck in waiting state
When using Pulp with Qpid (default broker) there is a hard-to-reproduce deadlocking bug [0]. The bug is in Qpid not Pulp, but we are very interested in seeing it resolved. In terms of clearing out your task operations, this will happen naturally if all the pulp workers are killed and restarted. If it's really bad you could consider running `sudo kill -9 -f celery` which kills all pulp workers. You could also issue cancel for all outstanding tasks with pulp-admin and then kill+restart at which point your system will be empty when processes finish starting. Note that deadlocked workers usually need to be killed with SIGKILL before being restarted. Many users never experience this problem. A few users do experience it and usually they experience it again. Several devs have tried to reproduce this but we have not been able to. The Qpid project is aware and investigating. I believe they have some rpms that provide a new version of python-qpid which is specifically patched for this issue. I'm waiting for them to produce rpms for the different distros so that affected users can evaluate if it resolves their issue. One other option to be aware of is that Pulp does support rabbitMQ and has not experienced this deadlocking issue. See the docs and server.conf for more info. FYI Pulp currently only tests the releases against Qpid. [0]: https://issues.apache.org/jira/browse/QPID-7317 -Brian On 09/21/2016 09:00 PM, Erinn Looney-Triggs wrote: I have 52 tasks that are stuck in a waiting state with nothing in a running state. I don't know much about pulp at this point, I am just fighting my way through satellite in an attempt to make it stable, but this looks a bit odd to me: pulp-admin -u admin -p tasks list | grep -i waiting | wc -l 52 pulp-admin -u admin -p tasks list --state running +--+ Tasks +--+ No tasks found The tasks, with the exception of one are all unit_update operations, the remaining one is a sync operation. I have done many restarts of the pulp processes with no luck in clearing these out, I can kill them off of course, but I would prefer to know what is going on here. Also chances are very good this will happen again. Thanks, -Erinn The technical details: RHEL 7.2 rpm -qa | grep pulp pulp-katello-1.0.1-1.el7sat.noarch rubygem-smart_proxy_pulp-1.2.2-1.el7sat.noarch python-pulp-repoauth-2.8.3.4-1.el7sat.noarch python-pulp-client-lib-2.8.3.4-1.el7sat.noarch pulp-docker-plugins-2.0.1.1-1.el7sat.noarch pulp-selinux-2.8.3.4-1.el7sat.noarch pulp-server-2.8.3.4-1.el7sat.noarch pulp-client-1.0-1.noarch python-pulp-common-2.8.3.4-1.el7sat.noarch pulp-rpm-admin-extensions-2.8.3.5-1.el7sat.noarch python-pulp-docker-common-2.0.1.1-1.el7sat.noarch pulp-ostree-plugins-1.1.1-2.el7sat.noarch pulp-puppet-plugins-2.8.3.3-1.el7sat.noarch python-pulp-bindings-2.8.3.4-1.el7sat.noarch python-isodate-0.5.0-4.pulp.el7sat.noarch python-pulp-streamer-2.8.3.4-1.el7sat.noarch python-pulp-oid_validation-2.8.3.4-1.el7sat.noarch python-pulp-agent-lib-2.8.3.4-1.el7sat.noarch python-pulp-ostree-common-1.1.1-2.el7sat.noarch pulp-rpm-handlers-2.8.3.5-1.el7sat.noarch pulp-admin-client-2.8.3.4-1.el7sat.noarch pulp-rpm-plugins-2.8.3.5-1.el7sat.noarch python-pulp-rpm-common-2.8.3.5-1.el7sat.noarch python-pulp-puppet-common-2.8.3.3-1.el7sat.noarch pulp-puppet-tools-2.8.3.3-1.el7sat.noarch ps -awfux | grep celery root 65959 0.0 0.0 112648 972 pts/0S+ 18:57 0:00 | \_ grep --color=auto celery apache52282 0.1 0.0 685240 63396 ?Ssl 18:46 0:00 /usr/bin/python /usr/bin/celery worker -A pulp.server.async.app -n resource_manager@%h -Q resource_manager -c 1 --events --umask 18 --pidfile=/var/run/pulp/resource_manager.pid --heartbeat-interval=30 apache52406 0.0 0.0 595524 53092 ?S18:46 0:00 \_ /usr/bin/python /usr/bin/celery worker -A pulp.server.async.app -n resource_manager@%h -Q resource_manager -c 1 --events --umask 18 --pidfile=/var/run/pulp/resource_manager.pid --heartbeat-interval=30 apache52424 0.1 0.0 685240 63272 ?Ssl 18:46 0:01 /usr/bin/python /usr/bin/celery worker -n reserved_resource_worker-0@%h -A pulp.server.async.app -c 1 --events --umask 18 --pidfile=/var/run/pulp/reserved_resource_worker-0.pid --heartbeat-interval=30 apache52692 0.0 0.0 610828 56536 ?Sl 18:46 0:00 \_ /usr/bin/python /usr/bin/celery worker -n reserved_resource_worker-0@%h -A pulp.server.async.app -c 1 --events --umask 18 --pidfile=/var/run/pulp/reserved_resource_worker-0.pid --heartbeat-interval=30 apache52426 0.1 0.0 684664 63404 ?Ssl 18:46 0:01 /usr/bin/python /usr/bin/celery worker -n reserved_resource_worker-1@%h -A pulp.server.async.app -c 1 --events --umask 18 --pidfile=/var/run/pulp/reserved_resource_worker-1.pid --heartbeat-interval=30 a
Re: [Pulp-list] Using Pulp to serve RHEL 6, 7, and Centos 6, 7
While I'm sure there are different ways of doing this, and more complex setups, but what you ask is not outside of what pulp can do for you. Currently, our site has a single pulp server which serves up RHEL6/7, CENTOS 6/7 and OracleLinux7 repositories along with a few other custom/non-custom repos. The thing about RHEL is they are licensed repositories, so you need a valid certificate to mirror them. So if you don't have any valid RHEL machines, then you are SOL. If you do then all you need to do is use the certs from a valid RHEL6/7 box and use them when you setup your pulp repo, and periodically RedHat will update those certs so you will need to update your repo config as well.. For example: (Modify to suit, we also serve up http so that's why it's explicitly enabled, and we only care about 64bit) Setup Epel6 cd /etc/pki/rpm-gpg wget https://dl.fedoraproject.org/pub/epel/RPM-GPG-KEY-EPEL-6 pulp-admin rpm repo create --repo-id=epel-6 --serve-http=True --feed=https://dl.fedoraproject.org/pub/epel/6/x86_64/ --gpg-key=/etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-6 --relative-url=epel-6 Setup Epel7 cd /etc/pki/rpm-gpg wget https://dl.fedoraproject.org/pub/epel/RPM-GPG-KEY-EPEL-7 pulp-admin rpm repo create --repo-id=epel-7 --serve-http=True --feed=https://dl.fedoraproject.org/pub/epel/7/x86_64/ --gpg-key=/etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-7 --relative-url=epel-7 Setup RHEL6 repos pulp-admin rpm repo create --repo-id=rhel-6-server-rpms --feed=https://cdn.redhat.com/content/dist/rhel/server/6/6Server/x86_64/os --feed-ca-cert=/etc/rhsm/ca/redhat-uep.pem --feed-key=/etc/pki/entitlement/xx-key.pem-rhel6 --feed-cert=/etc/pki/entitlement/xx.pem-rhel6 --relative-url=rhel-6-server-rpms --gpg-key=/etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release --serve-http=True pulp-admin rpm repo create --repo-id=rhel-6-server-thirdparty-oracle-java-rpms --feed=https://cdn.redhat.com/content/dist/rhel/server/6/6Server/x86_64/oracle-java/os --feed-ca-cert=/etc/rhsm/ca/redhat-uep.pem --feed-key=/etc/pki/entitlement/x-key.pem-rhel6 --feed-cert=/etc/pki/entitlement/x.pem-rhel6 --relative-url=rhel-6-server-thirdparty-oracle-java-rpms --gpg-key=/etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release --serve-http=True pulp-admin rpm repo create --repo-id=rhel-6-server-optional-rpms --feed=https://cdn.redhat.com/content/dist/rhel/server/6/6Server/x86_64/optional/os --feed-ca-cert=/etc/rhsm/ca/redhat-uep.pem --feed-key=/etc/pki/entitlement/x-key.pem-rhel6 --feed-cert=/etc/pki/entitlement/x.pem-rhel6 --relative-url=rhel-6-server-optional-rpms --gpg-key=/etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release --serve-http=True Setup RHEL7 repos pulp-admin rpm repo create --repo-id=rhel-7-server-rpms --feed=https://cdn.redhat.com/content/dist/rhel/server/7/7Server/x86_64/os --feed-ca-cert=/etc/rhsm/ca/redhat-uep.pem --feed-key=/etc/pki/entitlement/x-key.pem-rhel7 --feed-cert=/etc/pki/entitlement/x.pem-rhel7 --relative-url=rhel-7-server-rpms --gpg-key=/etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release --serve-http=True pulp-admin rpm repo create --repo-id=rhel-7-server-extras-rpms --feed=https://cdn.redhat.com/content/dist/rhel/server/7/7Server/x86_64/extras/os --feed-ca-cert=/etc/rhsm/ca/redhat-uep.pem --feed-key=/etc/pki/entitlement/x-key.pem-rhel7 --feed-cert=/etc/pki/entitlement/x.pem-rhel7 --relative-url=rhel-7-server-extras-rpms --gpg-key=/etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release --serve-http=True pulp-admin rpm repo create --repo-id=rhel-7-server-optional-rpms --feed=https://cdn.redhat.com/content/dist/rhel/server/7/7Server/x86_64/optional/os --feed-ca-cert=/etc/rhsm/ca/redhat-uep.pem --feed-key=/etc/pki/entitlement/x-key.pem-rhel7 --feed-cert=/etc/pki/entitlement/x.pem-rhel7 --relative-url=rhel-7-server-optional-rpms --gpg-key=/etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release --serve-http=True pulp-admin rpm repo create --repo-id=rhel-7-server-thirdparty-oracle-java-rpms --feed=https://cdn.redhat.com/content/dist/rhel/server/7/7Server/x86_64/oracle-java/os --feed-ca-cert=/etc/rhsm/ca/redhat-uep.pem --feed-key=/etc/pki/entitlement/x-key.pem-rhel7 --feed-cert=/etc/pki/entitlement/x.pem-rhel7 --relative-url=rhel-7-server-thirdparty-oracle-java-rpms --gpg-key=/etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release --serve-http=True Setup Centos7 Repos pulp-admin rpm repo create --repo-id=CentOS-7-base --feed=http://mirror.centos.org/centos/7/os/x86_64/ --relative-url= CentOS-7-base --serve-http=True pulp-admin rpm repo create --repo-id=CentOS-7-updates --feed=http://mirror.centos.org/centos/7/updates/x86_64/ --relative-url= CentOS-7-updates --serve-http=True pulp-admin rpm repo create --repo-id=CentOS-7-extras --feed=http://mirror.centos.org/centos/7/extras/x86_64/ --relative-url= CentOS-7-extras --serve-http=True Does this help? Jonathan Kaufman | DevOps Engineer | Foot Locker Inc. (414-357-4062) | jkauf...@footlocker.com From: Donald
Re: [Pulp-list] Using Pulp to serve RHEL 6, 7, and Centos 6, 7
You just need to use your entitlement certificates for RHEL to sync the RHEL 6/7 repositories. It doesn't matter if your Pulp server is running CentOS or not. You can get your entitlement certificate by logging in to RHN. Once you have that, copy it to your Pulp servers and create the RHEL repos like so: $ pulp-admin rpm repo create -repo-id=rhel6-server-x86_64 -feed=https://cdn.redhat.com/./../../ --feed-ca-cert=/etc/rhsm/ca/redhat-uep.pem -feed-cert=/etc/pki/entitlement/yourrhncert.pem -feed-key=/etc/pki/entitlement/yourrhncert.pem This enables you to sync 'protected' RHEL repositories. To determine what repositories your certificate entitles you to download you can use 'rct cat-ert /etc/pki/entitlement/yourrhncert.pem' You obviously won't need entitlement certificates to sync CentOS repositories. Hope this helps. Josh From: pulp-list-boun...@redhat.com [mailto:pulp-list-boun...@redhat.com] On Behalf Of Donald Wolfe Sent: Thursday, September 22, 2016 9:38 AM To: pulp-list@redhat.com Subject: [Pulp-list] Using Pulp to serve RHEL 6, 7, and Centos 6, 7 Hi, I am trying to setup a central package repo for clients to include RHEL 6 and 7, and CentOS 6 and 7. I have found lots of simple examples documented where folks are using Pulp to serve either RHEL, or CentOS, or other single distros, but none for how to configure a more complex setup with multiple distros/versions. The issue I see is around being able to download RHEL packages from a Pulp server built on a different distro like CentOS, or version RHEL 6 or 7. We have RHEL 6 and 7 systems which are all licensed properly, but we use CentOS 6 or 7 for less critical servers and just want to be able to perform periodic updates on them all by updating dev/test environments, and waiting a month (for verifying application integrity) before updating from the same set of packages in production environments. Is there a way to do this? I can't even seem to tell if I can do this with Satellite Server, but would prefer not to have to buy it as it would be overkill in our environment? Would someone please point me in the right direction? Would love to see an example config of something similar. Thank you, and best regards, Don Wolfe Disclaimer: This communication and any attachments contain private, confidential, privileged and/or proprietary information intended solely for the Recipient(s) named above. If you are not the intended Recipient, any use, dissemination, distribution or copying of the communication is strictly prohibited. If received in error, we apologize and ask that you please notify the Sender by returning this e-mail and permanently deleting this communication from your computer, including destruction of any printed copies. Any views expressed herein are not necessarily those of the Company represented by this e-mail source. No contracts, agreements or legally binding understandings may be entered into solely by an e-mail communication. ___ Pulp-list mailing list Pulp-list@redhat.com https://www.redhat.com/mailman/listinfo/pulp-list
[Pulp-list] Using Pulp to serve RHEL 6, 7, and Centos 6, 7
Hi, I am trying to setup a central package repo for clients to include RHEL 6 and 7, and CentOS 6 and 7. I have found lots of simple examples documented where folks are using Pulp to serve either RHEL, or CentOS, or other single distros, but none for how to configure a more complex setup with multiple distros/versions. The issue I see is around being able to download RHEL packages from a Pulp server built on a different distro like CentOS, or version RHEL 6 or 7. We have RHEL 6 and 7 systems which are all licensed properly, but we use CentOS 6 or 7 for less critical servers and just want to be able to perform periodic updates on them all by updating dev/test environments, and waiting a month (for verifying application integrity) before updating from the same set of packages in production environments. Is there a way to do this? I can't even seem to tell if I can do this with Satellite Server, but would prefer not to have to buy it as it would be overkill in our environment? Would someone please point me in the right direction? Would love to see an example config of something similar. Thank you, and best regards, Don Wolfe Disclaimer: This communication and any attachments contain private, confidential, privileged and/or proprietary information intended solely for the Recipient(s) named above. If you are not the intended Recipient, any use, dissemination, distribution or copying of the communication is strictly prohibited. If received in error, we apologize and ask that you please notify the Sender by returning this e-mail and permanently deleting this communication from your computer, including destruction of any printed copies. Any views expressed herein are not necessarily those of the Company represented by this e-mail source. No contracts, agreements or legally binding understandings may be entered into solely by an e-mail communication. ___ Pulp-list mailing list Pulp-list@redhat.com https://www.redhat.com/mailman/listinfo/pulp-list
