[Expired for maas (Ubuntu) because there has been no activity for 60
days.]
** Changed in: maas (Ubuntu)
Status: Incomplete = Expired
--
You received this bug notification because you are a member of Ubuntu
Server Team, which is subscribed to maas in Ubuntu.
[Expired for MAAS because there has been no activity for 60 days.]
** Changed in: maas
Status: Incomplete = Expired
--
You received this bug notification because you are a member of Ubuntu
Server Team, which is subscribed to maas in Ubuntu.
https://bugs.launchpad.net/bugs/1153077
Title:
I've recently seen a somewhat similar situation:
- Celery went down for a while.
- Celerybeat jobs accumulated in RabbitMQ.
- When Celery came back up, it spun manically, going through hundreds of
old jobs.
However, neither Celery nor RabbitMQ showed signs of falling over. They
were just busy.
Maybe RabbitMQ or Celery have a mechanism for ignoring old jobs. If not, we
can add a timestamp check to the job
implementations themselves.
That's a good point, there is support in Celery for this:
http://celery.github.io/celery/userguide/executing.html#expiration
--
You received this bug
On Tuesday 25 Feb 2014 10:17:25 you wrote:
- Celery went down for a while.
- Celerybeat jobs accumulated in RabbitMQ.
That's weird, the beat stuff is integrated into the celeryd isn't it?
It's a separate bug, but we should ensure that Celerybeat jobs have a
very short lifetime. Maybe
I'm going to close this bug unless anyone can confirm it's still a
problem? Nobody I know has seen this happen for at least the whole of
the last development cycle.
** Changed in: maas
Status: Triaged = Incomplete
** Changed in: maas (Ubuntu)
Status: Confirmed = Incomplete
--
Downgrading - nobody I know has experienced this lately.
** Changed in: maas
Importance: Critical = High
--
You received this bug notification because you are a member of Ubuntu
Server Team, which is subscribed to maas in Ubuntu.
https://bugs.launchpad.net/bugs/1153077
Title:
rabbitmq
I haven't used MAAS for about 6 months precisely because of this
issue. It may have been fixed in the interim.
On Tue, Oct 15, 2013 at 6:43 AM, Julian Edwards
1153...@bugs.launchpad.net wrote:
Have any you seen this again lately?
--
You received this bug notification because you are
On Tuesday 15 Oct 2013 08:48:08 you wrote:
I haven't used MAAS for about 6 months precisely because of this
issue. It may have been fixed in the interim.
Are you able to try it again? I have not seen the problem in a long time,
although it may have been a RabbitMQ bug in precise.
--
You
For the moment at least the system I had tried to install MAAS on is
in use. I may look at it again during the winter break.
On Tue, Oct 15, 2013 at 10:01 AM, Julian Edwards
1153...@bugs.launchpad.net wrote:
On Tuesday 15 Oct 2013 08:48:08 you wrote:
I haven't used MAAS for about 6 months
Erick, thanks for uploading your leases file. I've improved the code so
that it parses it in less than a couple of seconds, and there's plenty
of scope for improving that. I'll try to polish it and land it later
this week.
** Branch linked: lp:~allenap/maas/dhcp-leases-parsing
--
You received
Have any you seen this again lately?
--
You received this bug notification because you are a member of Ubuntu
Server Team, which is subscribed to maas in Ubuntu.
https://bugs.launchpad.net/bugs/1153077
Title:
rabbitmq queue fills up and celery stops executing tasks when
upload_dhcp_leases
I've just hit this again overnight. Logs are attached, command output is
below.
$ sudo rabbitmqctl list_queues -p /maas_workers name messages_unacknowledged
Listing queues ...
c1d352af-8060-4bf4-b820-f468b22a97fb4
celery 0
maas-juju.celeryd.pidbox0
master 0
...done.
$ sudo service
That maas.log from jhujhiti is packed with OAuthUnauthorized exceptions.
Right at the end of celery.log there's:
[2013-05-29 10:33:56,514: INFO/MainProcess] Got task from broker:
provisioningserver.tasks.refresh_secrets[842cc6d7-7e55-4098-a31e-
221b87b0b545]
and a little time later:
jhujhiti told me that the celery worker is eating up all the cpu and
most of the memory.
In celery.log, I see: [2013-05-29 10:41:28,386: INFO/MainProcess] Task
provisioningserver.tasks.upload_dhcp_leases[11a33416-5a33-4530-8fdf-
e0825b62d616] succeeded in *451.886018991s*: None
I thought that it
If someone can recreate this problem, it would be good to see in the
celery logs (/var/log/maas/*celery*.log) if there isn't an indication of
what's wrong with the workers.
--
You received this bug notification because you are a member of Ubuntu
Server Team, which is subscribed to maas in
Adding a TTL to the celery queues will avoid the out of memory error.
But if we want to fix the core problem we need to understand why the
tasks are pilling up without being processed by the workers.
--
You received this bug notification because you are a member of Ubuntu
Server Team, which is
I suspect this is caused by task messages getting generated before a
celeryd listener is connected/registered. I saw it once myself (but only
once).
Did the tasks start piling up right from the start or did the system
work for a while, then stopped working with the tasks piling up?
--
You
This is pretty difficult to diagnose without access to a system that
shows the problem… If you're seeing this bug, could you please run a
couple of commands and paste the results:
sudo rabbitmqctl list_queues -p /maas_workers name messages_unacknowledged
messages memory
service
Unfortunately, due in part to this bug, we aren't currently using MAAS.
--
You received this bug notification because you are a member of Ubuntu
Server Team, which is subscribed to maas in Ubuntu.
https://bugs.launchpad.net/bugs/1153077
Title:
rabbitmq queue fills up and celery stops
I suspect this is caused by task messages getting generated before a
celeryd listener is connected/registered. I saw it once myself (but
only once).
** Changed in: maas
Status: New = Triaged
** Changed in: maas
Importance: High = Critical
--
You received this bug notification
Status changed to 'Confirmed' because the bug affects multiple users.
** Changed in: maas (Ubuntu)
Status: New = Confirmed
--
You received this bug notification because you are a member of Ubuntu
Server Team, which is subscribed to maas in Ubuntu.
https://bugs.launchpad.net/bugs/1153077
I think I'm seeing the same issue. I have a celeryd process that appears
to be stuck in a loop and a queue in the vhost /maas_workers named
c1d352af-8060-4bf4-b820-f468b22a97fb. One thing that stands out to me is
that the queue has 4 unacknowledged messages at all times. New ones seem
to come in
** Changed in: maas (Ubuntu)
Importance: Undecided = Critical
** Changed in: maas
Importance: Undecided = High
** Changed in: maas (Ubuntu)
Importance: Critical = High
--
You received this bug notification because you are a member of Ubuntu
Server Team, which is subscribed to maas in
** Summary changed:
- rabbitmq queue fills up and salary stops executing tasks when
upload_dhcp_leases is done every minute (by default)
+ rabbitmq queue fills up and celery stops executing tasks when
upload_dhcp_leases is done every minute (by default)
** Also affects: maas
Importance:
25 matches
Mail list logo
