Re: [Pulp-list] qpid on el6 fails to install

2016-04-05 Thread Michael Hrivnak 
This issue has been resolved by the qpid project. Please let us know if you
see any further problems with that copr repo.

Michael

On Tue, Mar 29, 2016 at 10:49 AM, Michael Hrivnak 
wrote:

> Some users have discovered that when trying to install qpid (one of the
> message brokers we support) on el6, using the copr repo that the qpid
> project has provided [0], there is a problem with dependency resolution. It
> appears that qpid-proton-c was updated in EPEL to 0.12, and the special el6
> build of qpid needs to be re-built with that newer version.
>
> For now, if this is blocking you from installing on el6, you have these
> options:
>
> 1) el7 is great; you should check it out! Ok more seriously...
>
> 2) You can install the old 0.10 proton packages manually from here:
> http://koji.fedoraproject.org/koji/buildinfo?buildID=689093
>
> 3) You can use rabbitmq as your message broker.
>
> Michael
>
> [0] http://qpid.apache.org/packages.html#epel
>
___
Pulp-list mailing list
Pulp-list@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-list

[Pulp-list] Pulp 2.8.1 has been released!

2016-04-05 Thread Sean Myers 
Pulp 2.8.1 has been published to the stable repositories:

https://repos.fedorapeople.org/repos/pulp/pulp/stable/2.8/

Changes
===

The following Redmine (pulp.plan.io) issues are addressed in 2.8.1:

  Pulp
1734The Pulp streamer logs a traceback when a ConnectionError 
occurs.
1764SELinux denial on Celery attempting to read resolv.conf
1737The checksum/checksum_type is not being included in the lazy 
catalog
1735Downloading the CentOS base repository fails with lazy and 
--verify-all
1776sync_complete response has changed

  Puppet Support
1607Puppet install distributor fails when deleting repository if 
not published

  RPM Support
1138Install applicable errata consumer task does not report kernel 
packages were updated
1366Advisory package list doesn't match packages in the repository
1548published errata contain packages not in repo
1789Sync fills up /var/cache/pulp/ with rpms

More details regarding these issues can be found in Redmine at the following 
URL:

http://bit.ly/1LHtwWp

Aside from rpm version numbers and the addition of release notes,
no changes to 2.8.1 took place between the 2.8.1 beta release and
this release candidate.

RHEL/CentOS 6 Users
===

The issue with the el6 qpid COPR repository that impacted qpid
installations mentioned in the 2.8.1 release candidate announcemnet
has been resolved. The qpid COPR repository for el6 has since been
rebuilt, so el6 installations of Pulp should proceed normally again.





signature.asc
Description: OpenPGP digital signature
___
Pulp-list mailing list
Pulp-list@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-list

Re: [Pulp-list] goferd on consumers cannot connect to pulp server's queue after 2.7 -> 2.8 upgrade

2016-04-05 Thread Jeff Ortel 
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256



On 04/05/2016 02:40 PM, Konstantin M. Khankin wrote:
> On pulp-server:
> 
> [root@nms pulp]# LANG=C ls -l total 16 -rw-r-. 1 root   apache 1021 *Mar 
> 24* 23:47 ca.crt -rw-r-.
> 1 root   apache 1675 *Mar 24* 23:47 ca.key drwxr-xr-x. 2 apache apache   33 
> Mar 15 08:32 content 
> -rw-r-. 1 root   apache 1675 *Mar 24* 23:47 rsa.key -rw-r--r--. 1 root   
> apache  451 *Mar 24* 23:47
> rsa_pub.key
> 
> [root@nms pulp]# grep pulp /var/log/yum.log Mar 24 23:45:53 Erased:
> pulp-puppet-plugins-2.7.1-1.el7.noarch Mar 24 23:45:54 Erased: 
> pulp-rpm-plugins-2.7.1-1.el7.noarch Mar 24
> 23:46:13 Erased: pulp-server-2.7.1-1.el7.noarch Mar 24 23:47:00 Updated:
> python-pulp-common-2.8.0-1.el7.noarch Mar 24 23:47:01 Updated: 
> python-pulp-repoauth-2.8.0-1.el7.noarch 
> Mar 24 23:47:03 Updated: python-pulp-rpm-common-2.8.0-1.el7.noarch Mar 24 
> 23:47:03 Updated:
> python-pulp-client-lib-2.8.0-1.el7.noarch Mar 24 23:47:04 Updated:
> python-pulp-puppet-common-2.8.0-1.el7.noarch Mar 24 23:47:04 Installed:
> python-pulp-docker-common-2.0.0-1.el7.noarch Mar 24 23:47:05 Updated:
> python-pulp-bindings-2.8.0-1.el7.noarch Mar 24 23:47:06 Updated: 
> pulp-admin-client-2.8.0-1.el7.noarch 
> *Mar 24 23:47:14 Installed: pulp-server-2.8.0-1.el7.noarch* Mar 24 23:47:15 
> Installed:
> pulp-docker-plugins-2.0.0-1.el7.noarch Mar 24 23:47:16 Updated:
> pulp-rpm-admin-extensions-2.8.0-1.el7.noarch Mar 24 23:47:17 Updated:
> pulp-puppet-admin-extensions-2.8.0-1.el7.noarch Mar 24 23:47:18 Updated:
> python-pulp-oid_validation-2.8.0-1.el7.noarch Mar 24 23:49:01 Updated: 
> pulp-selinux-2.8.0-1.el7.noarch 
> Mar 25 00:01:40 Installed: pulp-rpm-plugins-2.8.0-1.el7.noarch Mar 25 
> 04:31:03 Обновлено:
> python-kombu.noarch 1:3.0.33-4.pulp.el7 Mar 25 04:31:13 Установлено: 
> pulp-docker-admin-extensions.noarch
> 2.0.0-1.el7
> 
> On pulp-consumer key was not changed since pulp-consumer installation time 
> (June 2015)
> 
> I think the root cause for this is here ->
> http://pulp.readthedocs.org/en/latest/user-guide/release-notes/2.8.x.html#upgrade-instructions-for-2-7-x-2-8
- -x.
>
> 
Removing python-semantic-version package causes removal of pulp-server package. 
Later on when I reinstalled
> server, certificates were updated as well

Yikes.  We should be advising users to do:

$ rpm -e --nodeps python-semantic-version

instead of using yum/dnf for just this reason when working around this upgrade 
problem.

> 
> Discussion thread is here -> 
> https://www.redhat.com/archives/pulp-list/2016-March/msg00078.html
> 
> 
> 2016-04-05 16:32 GMT+03:00 Jeff Ortel  >:
> 
> 
> 
> On 04/04/2016 09:53 AM, Konstantin M. Khankin wrote:
>> Hi Jeff!
> 
>> [root@drone ~]# pulp-consumer -v history 
>> +--+ 
>> Consumer History [drone] 
>> +--+
> 
>> 2016-04-04 19:46:18,164 - ERROR - Client-side exception occurred Traceback 
>> (most recent call last):
>> File "/usr/lib/python2.7/site-packages/pulp/client/extensions/core.py", line 
>> 474, in run exit_code = 
>> Cli.run(self, args) File "/usr/lib/python2.7/site-packages/okaara/cli.py", 
>> line 974, in run exit_code
>> = command_or_section.execute(self.prompt, remaining_args) File 
>> "/usr/lib/python2.7/site-packages/pulp/client/extensions/extensions.py", 
>> line 210, in execute return 
>> self.method(*arg_list, **clean_kwargs) File 
>> "/usr/lib/python2.7/site-packages/pulp/client/consumer/cli.py", line 367, in 
>> history 
>> kwargs['start-date'], kwargs['end-date']).response_body File 
>> "/usr/lib/python2.7/site-packages/pulp/bindings/consumer.py", line 199, in 
>> history return 
>> self.server.GET(path, queries) File 
>> "/usr/lib/python2.7/site-packages/pulp/bindings/server.py", line
>> 92, in GET return self._request('GET', path, queries, 
>> ignore_prefix=ignore_prefix) File 
>> "/usr/lib/python2.7/site-packages/pulp/bindings/server.py", line 152, in 
>> _request response_code, 
>> response_body = self.server_wrapper.request(method, url, body) File 
>> "/usr/lib/python2.7/site-packages/pulp/bindings/server.py", line 349, in 
>> request raise 
>> exceptions.ConnectionException(None, str(err), None) ConnectionException: 
>> (None, 'tlsv1 alert decrypt 
>> error', None) An error occurred attempting to contact the server. More 
>> information may be found using
>> the -v flag.
> 
>> /etc/pulp/consumer/consumer.conf looks correct (correct server name and 
>> verify_ssl: False). Checked on
>> a server - httpd's configs pulp.conf and pulp.conf.rpmsave have no 
>> differences in SSL section. 
>> .pulp/consumer.log does not contain any recent records
> 
>> I just think that maybe I need to re-register my consumers... But then again 
>> why could it have
>> happened?
> 
> I wonder if the /etc/pki/pulp/ca.* got updated somehow during the upgrade.  
> Can you check the date on
> those files

Re: [Pulp-list] goferd on consumers cannot connect to pulp server's queue after 2.7 -> 2.8 upgrade

2016-04-05 Thread Konstantin M. Khankin 
On pulp-server:

[root@nms pulp]# LANG=C ls -l
total 16
-rw-r-. 1 root   apache 1021 *Mar 24* 23:47 ca.crt
-rw-r-. 1 root   apache 1675 *Mar 24* 23:47 ca.key
drwxr-xr-x. 2 apache apache   33 Mar 15 08:32 content
-rw-r-. 1 root   apache 1675 *Mar 24* 23:47 rsa.key
-rw-r--r--. 1 root   apache  451 *Mar 24* 23:47 rsa_pub.key

[root@nms pulp]# grep pulp /var/log/yum.log
Mar 24 23:45:53 Erased: pulp-puppet-plugins-2.7.1-1.el7.noarch
Mar 24 23:45:54 Erased: pulp-rpm-plugins-2.7.1-1.el7.noarch
Mar 24 23:46:13 Erased: pulp-server-2.7.1-1.el7.noarch
Mar 24 23:47:00 Updated: python-pulp-common-2.8.0-1.el7.noarch
Mar 24 23:47:01 Updated: python-pulp-repoauth-2.8.0-1.el7.noarch
Mar 24 23:47:03 Updated: python-pulp-rpm-common-2.8.0-1.el7.noarch
Mar 24 23:47:03 Updated: python-pulp-client-lib-2.8.0-1.el7.noarch
Mar 24 23:47:04 Updated: python-pulp-puppet-common-2.8.0-1.el7.noarch
Mar 24 23:47:04 Installed: python-pulp-docker-common-2.0.0-1.el7.noarch
Mar 24 23:47:05 Updated: python-pulp-bindings-2.8.0-1.el7.noarch
Mar 24 23:47:06 Updated: pulp-admin-client-2.8.0-1.el7.noarch
*Mar 24 23:47:14 Installed: pulp-server-2.8.0-1.el7.noarch*
Mar 24 23:47:15 Installed: pulp-docker-plugins-2.0.0-1.el7.noarch
Mar 24 23:47:16 Updated: pulp-rpm-admin-extensions-2.8.0-1.el7.noarch
Mar 24 23:47:17 Updated: pulp-puppet-admin-extensions-2.8.0-1.el7.noarch
Mar 24 23:47:18 Updated: python-pulp-oid_validation-2.8.0-1.el7.noarch
Mar 24 23:49:01 Updated: pulp-selinux-2.8.0-1.el7.noarch
Mar 25 00:01:40 Installed: pulp-rpm-plugins-2.8.0-1.el7.noarch
Mar 25 04:31:03 Обновлено: python-kombu.noarch 1:3.0.33-4.pulp.el7
Mar 25 04:31:13 Установлено: pulp-docker-admin-extensions.noarch 2.0.0-1.el7

On pulp-consumer key was not changed since pulp-consumer installation time
(June 2015)

I think the root cause for this is here ->
http://pulp.readthedocs.org/en/latest/user-guide/release-notes/2.8.x.html#upgrade-instructions-for-2-7-x-2-8-x.
Removing python-semantic-version package causes removal of pulp-server
package. Later on when I reinstalled server, certificates were updated as
well

Discussion thread is here ->
https://www.redhat.com/archives/pulp-list/2016-March/msg00078.html


2016-04-05 16:32 GMT+03:00 Jeff Ortel :

> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA256
>
>
>
> On 04/04/2016 09:53 AM, Konstantin M. Khankin wrote:
> > Hi Jeff!
> >
> > [root@drone ~]# pulp-consumer -v history
> > +--+
> Consumer History [drone]
> > +--+
> >
> > 2016-04-04 19:46:18,164 - ERROR - Client-side exception occurred
> Traceback (most recent call last): File
> > "/usr/lib/python2.7/site-packages/pulp/client/extensions/core.py", line
> 474, in run exit_code =
> > Cli.run(self, args) File
> "/usr/lib/python2.7/site-packages/okaara/cli.py", line 974, in run
> exit_code =
> > command_or_section.execute(self.prompt, remaining_args) File
> > "/usr/lib/python2.7/site-packages/pulp/client/extensions/extensions.py",
> line 210, in execute return
> > self.method(*arg_list, **clean_kwargs) File
> > "/usr/lib/python2.7/site-packages/pulp/client/consumer/cli.py", line
> 367, in history
> > kwargs['start-date'], kwargs['end-date']).response_body File
> > "/usr/lib/python2.7/site-packages/pulp/bindings/consumer.py", line 199,
> in history return
> > self.server.GET(path, queries) File
> "/usr/lib/python2.7/site-packages/pulp/bindings/server.py", line 92,
> > in GET return self._request('GET', path, queries,
> ignore_prefix=ignore_prefix) File
> > "/usr/lib/python2.7/site-packages/pulp/bindings/server.py", line 152, in
> _request response_code,
> > response_body = self.server_wrapper.request(method, url, body) File
> > "/usr/lib/python2.7/site-packages/pulp/bindings/server.py", line 349, in
> request raise
> > exceptions.ConnectionException(None, str(err), None)
> ConnectionException: (None, 'tlsv1 alert decrypt
> > error', None) An error occurred attempting to contact the server. More
> information may be found using the
> > -v flag.
> >
> > /etc/pulp/consumer/consumer.conf looks correct (correct server name and
> verify_ssl: False). Checked on a
> > server - httpd's configs pulp.conf and pulp.conf.rpmsave have no
> differences in SSL section.
> > .pulp/consumer.log does not contain any recent records
> >
> > I just think that maybe I need to re-register my consumers... But then
> again why could it have happened?
>
> I wonder if the /etc/pki/pulp/ca.* got updated somehow during the
> upgrade.  Can you check the date on those
> files?
>
> >
> > 2016-04-04 17:21 GMT+03:00 Jeff Ortel  jor...@redhat.com>>:
> >
> > Konstantin,
> >
> > The agent validates registration by making a REST API call to the server
> using the consumer certificate.
> > On the consumer, can you try running a pulp-consumer command?
> >
> > For example:
> >
> > $ pulp-consumer history
> >
> > Thanks,
> >
> > Jeff
> >
> > On 04/02/2016 04:51 PM, Konstantin M. Kh

Re: [Pulp-list] Pulp 2.8.0 stops running on CentOS 6.7

2016-04-05 Thread Preethi Thomas 
You should be able to upgrade to the new kombu package 

http://koji.katello.org/koji/buildinfo?buildID=30314 

Thanks 
Preethi 

- Original Message -





Yes, thanks, that looks like it precisely. Is there a workaround, a new kombu 
package to upgrade to, or to regress to? 

-Alan 






From: Preethi Thomas [mailto:ptho...@redhat.com] 
Sent: Tuesday, April 05, 2016 10:52 AM 
To: Sparks, Alan  
Cc: pulp-list@redhat.com 
Subject: Re: [Pulp-list] Pulp 2.8.0 stops running on CentOS 6.7 





Looks like you may be hitting this issue 





https://pulp.plan.io/issues/1801 



- Original Message -





The only thing I’ve found in the syslogs, before all goes south is: 

Apr 1 18:28:19 oa-ftc-repo0001 pulp: pulp.server.async.scheduler:ERROR: Worker 
'reserved_resource_worke...@oa-ftc-repo0001.oa.ftc.hpelabs.net' has gone 
missing, removing from list of workers 

Apr 1 18:28:19 oa-ftc-repo0001 pulp: pulp.server.async.tasks:ERROR: The worker 
named reserved_resource_worke...@oa-ftc-repo0001.oa.ftc.hpelabs.net is missing. 
Canceling the tasks in its queue. 

Apr 1 18:28:19 oa-ftc-repo0001 pulp: pulp.server.async.scheduler:ERROR: Worker 
'reserved_resource_worke...@oa-ftc-repo0001.oa.ftc.hpelabs.net' has gone 
missing, removing from list of workers 

Apr 1 18:28:19 oa-ftc-repo0001 pulp: pulp.server.async.tasks:ERROR: The worker 
named reserved_resource_worke...@oa-ftc-repo0001.oa.ftc.hpelabs.net is missing. 
Canceling the tasks in its queue. 

Apr 1 18:28:19 oa-ftc-repo0001 pulp: pulp.server.async.scheduler:ERROR: Worker 
'reserved_resource_worke...@oa-ftc-repo0001.oa.ftc.hpelabs.net' has gone 
missing, removing from list of workers 

Apr 1 18:28:19 oa-ftc-repo0001 pulp: pulp.server.async.tasks:ERROR: The worker 
named reserved_resource_worke...@oa-ftc-repo0001.oa.ftc.hpelabs.net is missing. 
Canceling the tasks in its queue. 

Apr 1 18:28:19 oa-ftc-repo0001 pulp: pulp.server.async.scheduler:ERROR: Worker 
'reserved_resource_worke...@oa-ftc-repo0001.oa.ftc.hpelabs.net' has gone 
missing, removing from list of workers 



No idea what is happening at that time, that would cause all the processes to 
“disconnect”. They seem to still be running… qpid or mongo going bad somehow? 
Just vanilla stock installs of those. 

-Alan 






From: pulp-list-boun...@redhat.com [ mailto:pulp-list-boun...@redhat.com ] On 
Behalf Of Sparks, Alan 
Sent: Tuesday, April 05, 2016 10:19 AM 
To: pulp-list@redhat.com 
Subject: [Pulp-list] Pulp 2.8.0 stops running on CentOS 6.7 




I have a 2.8.0 install running on a patched-up CentOS 6.7 machine, running all 
from the pulp-stable distribution. QPID as MQ, and Mongo server 
mongodb-server-2.4.14-1.el6.x86_64. 



I can’t seem to keep it running more than a week before it falls over, tasks 
stop running, and following repeated in the syslog: 

Apr 5 16:09:22 oa-ftc-repo0001 pulp: pulp.server.async.scheduler:ERROR: There 
are 0 pulp_resource_manager processes running. Pulp will not operate correctly 
without at least one pulp_resource_mananger process running. 

Apr 5 16:09:22 oa-ftc-repo0001 pulp: pulp.server.async.scheduler:ERROR: There 
are 0 pulp_celerybeat processes running. Pulp will not operate correctly 
without at least one pulp_celerybeat process running. 





I do see this running in process list: 

10606 ? Sl 7:57 /usr/bin/python /usr/bin/celery beat 
--app=pulp.server.async.celery_instance.celery 
--scheduler=pulp.server.async.scheduler.Scheduler --workdir=/var/run/pulp/ -f 
/var/log/pulp/celerybeat.log -l INFO --detach 
--pidfile=/var/run/pulp/celerybeat.pid 



If I attempt to stop celerybeat: 

# service pulp_celerybeat stop 

celery init v10.0. 

Using configuration: /etc/default/pulp_workers, /etc/default/pulp_celerybeat 

Stopping pulp_celerybeat... ERROR 

Timed out while stopping (30s) 



I’m not sure how to determine what it dying… If I hard stop and start 
everything, or if I reboot, I can get out of the issue for a few days, before 
it recurs. 

Does anyone have advice on what to look for? The Pulp logs basically says 
everything is logging to syslog, but have not found a smoking gun to indicate 
what fell over. 

-Alan 




___ 
Pulp-list mailing list 
Pulp-list@redhat.com 
https://www.redhat.com/mailman/listinfo/pulp-list 









___
Pulp-list mailing list
Pulp-list@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-list

Re: [Pulp-list] Pulp 2.8.0 stops running on CentOS 6.7

2016-04-05 Thread Sparks, Alan 
Yes, thanks, that looks like it precisely.   Is there a workaround, a new kombu 
package to upgrade to, or to regress to?
-Alan


From: Preethi Thomas [mailto:ptho...@redhat.com]
Sent: Tuesday, April 05, 2016 10:52 AM
To: Sparks, Alan 
Cc: pulp-list@redhat.com
Subject: Re: [Pulp-list] Pulp 2.8.0 stops running on CentOS 6.7

Looks like you may be hitting this issue

https://pulp.plan.io/issues/1801


The only thing I’ve found in the syslogs, before all goes south is:
Apr  1 18:28:19 oa-ftc-repo0001 pulp: pulp.server.async.scheduler:ERROR: Worker 
'reserved_resource_worke...@oa-ftc-repo0001.oa.ftc.hpelabs.net' has gone 
missing, removing from list of workers
Apr  1 18:28:19 oa-ftc-repo0001 pulp: pulp.server.async.tasks:ERROR: The worker 
named 
reserved_resource_worke...@oa-ftc-repo0001.oa.ftc.hpelabs.net
 is missing. Canceling the tasks in its queue.
Apr  1 18:28:19 oa-ftc-repo0001 pulp: pulp.server.async.scheduler:ERROR: Worker 
'reserved_resource_worke...@oa-ftc-repo0001.oa.ftc.hpelabs.net' has gone 
missing, removing from list of workers
Apr  1 18:28:19 oa-ftc-repo0001 pulp: pulp.server.async.tasks:ERROR: The worker 
named 
reserved_resource_worke...@oa-ftc-repo0001.oa.ftc.hpelabs.net
 is missing. Canceling the tasks in its queue.
Apr  1 18:28:19 oa-ftc-repo0001 pulp: pulp.server.async.scheduler:ERROR: Worker 
'reserved_resource_worke...@oa-ftc-repo0001.oa.ftc.hpelabs.net' has gone 
missing, removing from list of workers
Apr  1 18:28:19 oa-ftc-repo0001 pulp: pulp.server.async.tasks:ERROR: The worker 
named 
reserved_resource_worke...@oa-ftc-repo0001.oa.ftc.hpelabs.net
 is missing. Canceling the tasks in its queue.
Apr  1 18:28:19 oa-ftc-repo0001 pulp: pulp.server.async.scheduler:ERROR: Worker 
'reserved_resource_worke...@oa-ftc-repo0001.oa.ftc.hpelabs.net' has gone 
missing, removing from list of workers

No idea what is happening at that time, that would cause all the processes to 
“disconnect”.   They seem to still be running… qpid or mongo going bad somehow? 
 Just vanilla stock installs of those.
-Alan


From: pulp-list-boun...@redhat.com 
[mailto:pulp-list-boun...@redhat.com] On Behalf Of Sparks, Alan
Sent: Tuesday, April 05, 2016 10:19 AM
To: pulp-list@redhat.com
Subject: [Pulp-list] Pulp 2.8.0 stops running on CentOS 6.7

I have a 2.8.0 install running on a patched-up CentOS 6.7 machine, running all 
from the pulp-stable distribution.   QPID as MQ, and Mongo server 
mongodb-server-2.4.14-1.el6.x86_64.

I can’t seem to keep it running more than a week before it falls over, tasks 
stop running, and following repeated in the syslog:
Apr  5 16:09:22 oa-ftc-repo0001 pulp: pulp.server.async.scheduler:ERROR: There 
are 0 pulp_resource_manager processes running. Pulp will not operate correctly 
without at least one pulp_resource_mananger process running.
Apr  5 16:09:22 oa-ftc-repo0001 pulp: pulp.server.async.scheduler:ERROR: There 
are 0 pulp_celerybeat processes running. Pulp will not operate correctly 
without at least one pulp_celerybeat process running.


I do see this running in process list:
10606 ?Sl 7:57 /usr/bin/python /usr/bin/celery beat 
--app=pulp.server.async.celery_instance.celery 
--scheduler=pulp.server.async.scheduler.Scheduler --workdir=/var/run/pulp/ -f 
/var/log/pulp/celerybeat.log -l INFO --detach 
--pidfile=/var/run/pulp/celerybeat.pid

If I attempt to stop celerybeat:
# service pulp_celerybeat stop
celery init v10.0.
Using configuration: /etc/default/pulp_workers, /etc/default/pulp_celerybeat
Stopping pulp_celerybeat... ERROR
Timed out while stopping (30s)

I’m not sure how to determine what it dying… If I hard stop and start 
everything, or if I reboot, I can get out of the issue for a few days, before 
it recurs.
Does anyone have advice on what to look for?   The Pulp logs basically says 
everything is logging to syslog, but have not found a smoking gun to indicate 
what fell over.
-Alan


___
Pulp-list mailing list
Pulp-list@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-list

___
Pulp-list mailing list
Pulp-list@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-list

Re: [Pulp-list] Pulp 2.8.0 stops running on CentOS 6.7

2016-04-05 Thread Preethi Thomas 
Looks like you may be hitting this issue 

https://pulp.plan.io/issues/1801 

- Original Message -

> The only thing I’ve found in the syslogs, before all goes south is:

> Apr 1 18:28:19 oa-ftc-repo0001 pulp: pulp.server.async.scheduler:ERROR:
> Worker 'reserved_resource_worke...@oa-ftc-repo0001.oa.ftc.hpelabs.net' has
> gone missing, removing from list of workers

> Apr 1 18:28:19 oa-ftc-repo0001 pulp: pulp.server.async.tasks:ERROR: The
> worker named reserved_resource_worke...@oa-ftc-repo0001.oa.ftc.hpelabs.net
> is missing. Canceling the tasks in its queue.

> Apr 1 18:28:19 oa-ftc-repo0001 pulp: pulp.server.async.scheduler:ERROR:
> Worker 'reserved_resource_worke...@oa-ftc-repo0001.oa.ftc.hpelabs.net' has
> gone missing, removing from list of workers

> Apr 1 18:28:19 oa-ftc-repo0001 pulp: pulp.server.async.tasks:ERROR: The
> worker named reserved_resource_worke...@oa-ftc-repo0001.oa.ftc.hpelabs.net
> is missing. Canceling the tasks in its queue.

> Apr 1 18:28:19 oa-ftc-repo0001 pulp: pulp.server.async.scheduler:ERROR:
> Worker 'reserved_resource_worke...@oa-ftc-repo0001.oa.ftc.hpelabs.net' has
> gone missing, removing from list of workers

> Apr 1 18:28:19 oa-ftc-repo0001 pulp: pulp.server.async.tasks:ERROR: The
> worker named reserved_resource_worke...@oa-ftc-repo0001.oa.ftc.hpelabs.net
> is missing. Canceling the tasks in its queue.

> Apr 1 18:28:19 oa-ftc-repo0001 pulp: pulp.server.async.scheduler:ERROR:
> Worker 'reserved_resource_worke...@oa-ftc-repo0001.oa.ftc.hpelabs.net' has
> gone missing, removing from list of workers

> No idea what is happening at that time, that would cause all the processes to
> “disconnect”. They seem to still be running… qpid or mongo going bad
> somehow? Just vanilla stock installs of those.

> -Alan

> From: pulp-list-boun...@redhat.com [mailto:pulp-list-boun...@redhat.com] On
> Behalf Of Sparks, Alan
> Sent: Tuesday, April 05, 2016 10:19 AM
> To: pulp-list@redhat.com
> Subject: [Pulp-list] Pulp 2.8.0 stops running on CentOS 6.7

> I have a 2.8.0 install running on a patched-up CentOS 6.7 machine, running
> all from the pulp-stable distribution. QPID as MQ, and Mongo server
> mongodb-server-2.4.14-1.el6.x86_64.

> I can’t seem to keep it running more than a week before it falls over, tasks
> stop running, and following repeated in the syslog:

> Apr 5 16:09:22 oa-ftc-repo0001 pulp: pulp.server.async.scheduler:ERROR: There
> are 0 pulp_resource_manager processes running. Pulp will not operate
> correctly without at least one pulp_resource_mananger process running.

> Apr 5 16:09:22 oa-ftc-repo0001 pulp: pulp.server.async.scheduler:ERROR: There
> are 0 pulp_celerybeat processes running. Pulp will not operate correctly
> without at least one pulp_celerybeat process running.

> I do see this running in process list:

> 10606 ? Sl 7:57 /usr/bin/python /usr/bin/celery beat
> --app=pulp.server.async.celery_instance.celery
> --scheduler=pulp.server.async.scheduler.Scheduler --workdir=/var/run/pulp/
> -f /var/log/pulp/celerybeat.log -l INFO --detach
> --pidfile=/var/run/pulp/celerybeat.pid

> If I attempt to stop celerybeat:

> # service pulp_celerybeat stop

> celery init v10.0.

> Using configuration: /etc/default/pulp_workers, /etc/default/pulp_celerybeat

> Stopping pulp_celerybeat... ERROR

> Timed out while stopping (30s)

> I’m not sure how to determine what it dying… If I hard stop and start
> everything, or if I reboot, I can get out of the issue for a few days,
> before it recurs.

> Does anyone have advice on what to look for? The Pulp logs basically says
> everything is logging to syslog, but have not found a smoking gun to
> indicate what fell over.

> -Alan

> ___
> Pulp-list mailing list
> Pulp-list@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-list___
Pulp-list mailing list
Pulp-list@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-list

Re: [Pulp-list] Pulp 2.8.0 stops running on CentOS 6.7

2016-04-05 Thread Sparks, Alan 
The only thing I've found in the syslogs, before all goes south is:
Apr  1 18:28:19 oa-ftc-repo0001 pulp: pulp.server.async.scheduler:ERROR: Worker 
'reserved_resource_worke...@oa-ftc-repo0001.oa.ftc.hpelabs.net' has gone 
missing, removing from list of workers
Apr  1 18:28:19 oa-ftc-repo0001 pulp: pulp.server.async.tasks:ERROR: The worker 
named reserved_resource_worke...@oa-ftc-repo0001.oa.ftc.hpelabs.net is missing. 
Canceling the tasks in its queue.
Apr  1 18:28:19 oa-ftc-repo0001 pulp: pulp.server.async.scheduler:ERROR: Worker 
'reserved_resource_worke...@oa-ftc-repo0001.oa.ftc.hpelabs.net' has gone 
missing, removing from list of workers
Apr  1 18:28:19 oa-ftc-repo0001 pulp: pulp.server.async.tasks:ERROR: The worker 
named reserved_resource_worke...@oa-ftc-repo0001.oa.ftc.hpelabs.net is missing. 
Canceling the tasks in its queue.
Apr  1 18:28:19 oa-ftc-repo0001 pulp: pulp.server.async.scheduler:ERROR: Worker 
'reserved_resource_worke...@oa-ftc-repo0001.oa.ftc.hpelabs.net' has gone 
missing, removing from list of workers
Apr  1 18:28:19 oa-ftc-repo0001 pulp: pulp.server.async.tasks:ERROR: The worker 
named reserved_resource_worke...@oa-ftc-repo0001.oa.ftc.hpelabs.net is missing. 
Canceling the tasks in its queue.
Apr  1 18:28:19 oa-ftc-repo0001 pulp: pulp.server.async.scheduler:ERROR: Worker 
'reserved_resource_worke...@oa-ftc-repo0001.oa.ftc.hpelabs.net' has gone 
missing, removing from list of workers

No idea what is happening at that time, that would cause all the processes to 
"disconnect".   They seem to still be running... qpid or mongo going bad 
somehow?  Just vanilla stock installs of those.
-Alan


From: pulp-list-boun...@redhat.com [mailto:pulp-list-boun...@redhat.com] On 
Behalf Of Sparks, Alan
Sent: Tuesday, April 05, 2016 10:19 AM
To: pulp-list@redhat.com
Subject: [Pulp-list] Pulp 2.8.0 stops running on CentOS 6.7

I have a 2.8.0 install running on a patched-up CentOS 6.7 machine, running all 
from the pulp-stable distribution.   QPID as MQ, and Mongo server 
mongodb-server-2.4.14-1.el6.x86_64.

I can't seem to keep it running more than a week before it falls over, tasks 
stop running, and following repeated in the syslog:
Apr  5 16:09:22 oa-ftc-repo0001 pulp: pulp.server.async.scheduler:ERROR: There 
are 0 pulp_resource_manager processes running. Pulp will not operate correctly 
without at least one pulp_resource_mananger process running.
Apr  5 16:09:22 oa-ftc-repo0001 pulp: pulp.server.async.scheduler:ERROR: There 
are 0 pulp_celerybeat processes running. Pulp will not operate correctly 
without at least one pulp_celerybeat process running.


I do see this running in process list:
10606 ?Sl 7:57 /usr/bin/python /usr/bin/celery beat 
--app=pulp.server.async.celery_instance.celery 
--scheduler=pulp.server.async.scheduler.Scheduler --workdir=/var/run/pulp/ -f 
/var/log/pulp/celerybeat.log -l INFO --detach 
--pidfile=/var/run/pulp/celerybeat.pid

If I attempt to stop celerybeat:
# service pulp_celerybeat stop
celery init v10.0.
Using configuration: /etc/default/pulp_workers, /etc/default/pulp_celerybeat
Stopping pulp_celerybeat... ERROR
Timed out while stopping (30s)

I'm not sure how to determine what it dying... If I hard stop and start 
everything, or if I reboot, I can get out of the issue for a few days, before 
it recurs.
Does anyone have advice on what to look for?   The Pulp logs basically says 
everything is logging to syslog, but have not found a smoking gun to indicate 
what fell over.
-Alan

___
Pulp-list mailing list
Pulp-list@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-list

[Pulp-list] Pulp 2.8.0 stops running on CentOS 6.7

2016-04-05 Thread Sparks, Alan 
I have a 2.8.0 install running on a patched-up CentOS 6.7 machine, running all 
from the pulp-stable distribution.   QPID as MQ, and Mongo server 
mongodb-server-2.4.14-1.el6.x86_64.

I can't seem to keep it running more than a week before it falls over, tasks 
stop running, and following repeated in the syslog:
Apr  5 16:09:22 oa-ftc-repo0001 pulp: pulp.server.async.scheduler:ERROR: There 
are 0 pulp_resource_manager processes running. Pulp will not operate correctly 
without at least one pulp_resource_mananger process running.
Apr  5 16:09:22 oa-ftc-repo0001 pulp: pulp.server.async.scheduler:ERROR: There 
are 0 pulp_celerybeat processes running. Pulp will not operate correctly 
without at least one pulp_celerybeat process running.


I do see this running in process list:
10606 ?Sl 7:57 /usr/bin/python /usr/bin/celery beat 
--app=pulp.server.async.celery_instance.celery 
--scheduler=pulp.server.async.scheduler.Scheduler --workdir=/var/run/pulp/ -f 
/var/log/pulp/celerybeat.log -l INFO --detach 
--pidfile=/var/run/pulp/celerybeat.pid

If I attempt to stop celerybeat:
# service pulp_celerybeat stop
celery init v10.0.
Using configuration: /etc/default/pulp_workers, /etc/default/pulp_celerybeat
Stopping pulp_celerybeat... ERROR
Timed out while stopping (30s)

I'm not sure how to determine what it dying... If I hard stop and start 
everything, or if I reboot, I can get out of the issue for a few days, before 
it recurs.
Does anyone have advice on what to look for?   The Pulp logs basically says 
everything is logging to syslog, but have not found a smoking gun to indicate 
what fell over.
-Alan

___
Pulp-list mailing list
Pulp-list@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-list

Re: [Pulp-list] goferd on consumers cannot connect to pulp server's queue after 2.7 -> 2.8 upgrade

2016-04-05 Thread Jeff Ortel 
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256



On 04/04/2016 09:53 AM, Konstantin M. Khankin wrote:
> Hi Jeff!
> 
> [root@drone ~]# pulp-consumer -v history 
> +--+ 
> Consumer History [drone] 
> +--+
> 
> 2016-04-04 19:46:18,164 - ERROR - Client-side exception occurred Traceback 
> (most recent call last): File
> "/usr/lib/python2.7/site-packages/pulp/client/extensions/core.py", line 474, 
> in run exit_code =
> Cli.run(self, args) File "/usr/lib/python2.7/site-packages/okaara/cli.py", 
> line 974, in run exit_code =
> command_or_section.execute(self.prompt, remaining_args) File
> "/usr/lib/python2.7/site-packages/pulp/client/extensions/extensions.py", line 
> 210, in execute return
> self.method(*arg_list, **clean_kwargs) File
> "/usr/lib/python2.7/site-packages/pulp/client/consumer/cli.py", line 367, in 
> history 
> kwargs['start-date'], kwargs['end-date']).response_body File
> "/usr/lib/python2.7/site-packages/pulp/bindings/consumer.py", line 199, in 
> history return
> self.server.GET(path, queries) File 
> "/usr/lib/python2.7/site-packages/pulp/bindings/server.py", line 92,
> in GET return self._request('GET', path, queries, 
> ignore_prefix=ignore_prefix) File
> "/usr/lib/python2.7/site-packages/pulp/bindings/server.py", line 152, in 
> _request response_code,
> response_body = self.server_wrapper.request(method, url, body) File
> "/usr/lib/python2.7/site-packages/pulp/bindings/server.py", line 349, in 
> request raise
> exceptions.ConnectionException(None, str(err), None) ConnectionException: 
> (None, 'tlsv1 alert decrypt
> error', None) An error occurred attempting to contact the server. More 
> information may be found using the
> -v flag.
> 
> /etc/pulp/consumer/consumer.conf looks correct (correct server name and 
> verify_ssl: False). Checked on a 
> server - httpd's configs pulp.conf and pulp.conf.rpmsave have no differences 
> in SSL section. 
> .pulp/consumer.log does not contain any recent records
> 
> I just think that maybe I need to re-register my consumers... But then again 
> why could it have happened?

I wonder if the /etc/pki/pulp/ca.* got updated somehow during the upgrade.  Can 
you check the date on those
files?

> 
> 2016-04-04 17:21 GMT+03:00 Jeff Ortel  >:
> 
> Konstantin,
> 
> The agent validates registration by making a REST API call to the server 
> using the consumer certificate.
> On the consumer, can you try running a pulp-consumer command?
> 
> For example:
> 
> $ pulp-consumer history
> 
> Thanks,
> 
> Jeff
> 
> On 04/02/2016 04:51 PM, Konstantin M. Khankin wrote:
>> Hello!
> 
>> After 2.7->2.8 upgrade all consumers cannot talk to pulp server - goferd 
>> returns the next error on any 
>> operation: [WARNING][MainThread] pulp.agent.gofer.pulpplugin:107 - validate 
>> registration failed:
>> (None, 'tlsv1 alert decrypt error', None)
> 
>> I didn't change any keys or configs. Logs also do not help. What could went 
>> wrong?
> 
>> Thanks!
> 
>> -- Konstantin Khankin
> 
> 
>> ___ Pulp-list mailing list 
>> Pulp-list@redhat.com
> 
>> https://www.redhat.com/mailman/listinfo/pulp-list
> 
> 
> 
> 
> 
> -- Ханкин Константин
-BEGIN PGP SIGNATURE-
Version: GnuPG v2

iQIcBAEBCAAGBQJXA75YAAoJEM9lW7UwFwZXOmEP/1u9qM6qS5waUjOZFPKwdRet
ZGuqECMY2Eu/PSFB9TSfU/RHvIdUcJce73WhvIKdg8lZ576WmEJQCPLRH4ocFDo+
8jP22voFuEkxfg2pAfeRSNle354vn4l1XMD7vlqscD6Z8opnQSkWNowJ2jb1tWSI
lY8zyAt/MbMAwsKI0NaAQhI4ORz99TjVpATEXGvFxsFZeYdHB3EW9o5iAgTl0mqo
vSp8yaAiCROjWqCjfYpnBk9z8T0sPctCEOtu81fYLJOzYehl51PjwRF0lOt/LYTl
bvQ2KfG1yThFz168eP7UROm63SLxv+B27K/QVIy5LfX9Zzcam6WudZ0rukWWGYur
OMQrXIlZhxjbk6uA9QYwS359S9e5DREtZTVgX9ZTUZ+3EWlCsAIn7ei9LGwoVTwC
uuQbdS0tcKYNLAHRlttGBPHeC2sQp90H3Pdl2LxyyANZ6mR7prBmVQxkWpb7ujSC
FXuUgEXnQz5aRRaETEQEUicYBfnHXYeqVLvxLlyNw9lDFk/tOg/7FAunXZi1sqVL
YfLaJICRvtihcTH3+CpLQ+S8sD4IG7Akg/v4aCUm4livOXoDN8jV0ysTzvNnBk5n
rvKM0py6wKLGd6FIbjWmb1hXwGsLmuvEkuRJJbMQdIySEv9lLBFwxpSIUfZVO/ie
IzggUNc0+mXTbLzMpBaj
=Pfx5
-END PGP SIGNATURE-

___
Pulp-list mailing list
Pulp-list@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-list