[Expired for MAAS because there has been no activity for 60 days.]
** Changed in: maas
Status: Incomplete => Expired
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1153077
Title:
rabbitmq que
[Expired for maas (Ubuntu) because there has been no activity for 60
days.]
** Changed in: maas (Ubuntu)
Status: Incomplete => Expired
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1153077
Tit
On Tuesday 25 Feb 2014 10:17:25 you wrote:
> - Celery went down for a while.
>
> - Celerybeat jobs accumulated in RabbitMQ.
That's weird, the beat stuff is integrated into the celeryd isn't it?
> It's a separate bug, but we should ensure that Celerybeat jobs have a
> very short lifetime. Maybe R
> Maybe RabbitMQ or Celery have a mechanism for ignoring old jobs. If not, we
> can add a timestamp check to the job
> implementations themselves.
That's a good point, there is support in Celery for this:
http://celery.github.io/celery/userguide/executing.html#expiration
--
You received this bu
I've recently seen a somewhat similar situation:
- Celery went down for a while.
- Celerybeat jobs accumulated in RabbitMQ.
- When Celery came back up, it spun manically, going through hundreds of
old jobs.
However, neither Celery nor RabbitMQ showed signs of falling over. They
were just busy.
I'm going to close this bug unless anyone can confirm it's still a
problem? Nobody I know has seen this happen for at least the whole of
the last development cycle.
** Changed in: maas
Status: Triaged => Incomplete
** Changed in: maas (Ubuntu)
Status: Confirmed => Incomplete
--
Y
Downgrading - nobody I know has experienced this lately.
** Changed in: maas
Importance: Critical => High
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1153077
Title:
rabbitmq queue fills up and
Erick, thanks for uploading your leases file. I've improved the code so
that it parses it in less than a couple of seconds, and there's plenty
of scope for improving that. I'll try to polish it and land it later
this week.
** Branch linked: lp:~allenap/maas/dhcp-leases-parsing
--
You received th
For the moment at least the system I had tried to install MAAS on is
in use. I may look at it again during the winter break.
On Tue, Oct 15, 2013 at 10:01 AM, Julian Edwards
<1153...@bugs.launchpad.net> wrote:
> On Tuesday 15 Oct 2013 08:48:08 you wrote:
>> I haven't used MAAS for about 6 months p
On Tuesday 15 Oct 2013 08:48:08 you wrote:
> I haven't used MAAS for about 6 months precisely because of this
> issue. It may have been fixed in the interim.
Are you able to try it again? I have not seen the problem in a long time,
although it may have been a RabbitMQ bug in precise.
--
You re
I haven't used MAAS for about 6 months precisely because of this
issue. It may have been fixed in the interim.
On Tue, Oct 15, 2013 at 6:43 AM, Julian Edwards
<1153...@bugs.launchpad.net> wrote:
> Have any you seen this again lately?
>
> --
> You received this bug notification because you are subs
Have any you seen this again lately?
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1153077
Title:
rabbitmq queue fills up and celery stops executing tasks when
upload_dhcp_leases is done every min
Leases file is attached. I am working on figuring out why there are so
many abandoned leases, but hopefully MAAS can be made to not choke on
such a large file.
** Attachment added: "dhcpd.leases.gz"
https://bugs.launchpad.net/ubuntu/+source/maas/+bug/1153077/+attachment/3689992/+files/dhcpd.le
jhujhiti told me that the celery worker is eating up all the cpu and
most of the memory.
In celery.log, I see: [2013-05-29 10:41:28,386: INFO/MainProcess] Task
provisioningserver.tasks.upload_dhcp_leases[11a33416-5a33-4530-8fdf-
e0825b62d616] succeeded in *451.886018991s*: None
I thought that it
That maas.log from jhujhiti is packed with OAuthUnauthorized exceptions.
Right at the end of celery.log there's:
[2013-05-29 10:33:56,514: INFO/MainProcess] Got task from broker:
provisioningserver.tasks.refresh_secrets[842cc6d7-7e55-4098-a31e-
221b87b0b545]
and a little time later:
[2013-05-29
I've just hit this again overnight. Logs are attached, command output is
below.
$ sudo rabbitmqctl list_queues -p /maas_workers name messages_unacknowledged
Listing queues ...
c1d352af-8060-4bf4-b820-f468b22a97fb4
celery 0
maas-juju.celeryd.pidbox0
master 0
...done.
$ sudo service ma
Unfortunately, due in part to this bug, we aren't currently using MAAS.
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1153077
Title:
rabbitmq queue fills up and celery stops executing tasks when
u
This is pretty difficult to diagnose without access to a system that
shows the problem… If you're seeing this bug, could you please run a
couple of commands and paste the results:
sudo rabbitmqctl list_queues -p /maas_workers name messages_unacknowledged
messages memory
service maas-cluster-celer
> I suspect this is caused by task messages getting generated before a
celeryd listener is connected/registered. I saw it once myself (but only
once).
Did the tasks start piling up right from the start or did the system
work for a while, then stopped working with the tasks piling up?
--
You rece
If someone can recreate this problem, it would be good to see in the
celery logs (/var/log/maas/*celery*.log) if there isn't an indication of
what's wrong with the workers.
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.l
Adding a TTL to the celery queues will avoid the out of memory error.
But if we want to fix the core problem we need to understand why the
tasks are pilling up without being processed by the workers.
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribe
I suspect this is caused by task messages getting generated before a
celeryd listener is connected/registered. I saw it once myself (but
only once).
** Changed in: maas
Status: New => Triaged
** Changed in: maas
Importance: High => Critical
--
You received this bug notification becau
** Changed in: maas (Ubuntu)
Importance: Undecided => Critical
** Changed in: maas
Importance: Undecided => High
** Changed in: maas (Ubuntu)
Importance: Critical => High
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https
Status changed to 'Confirmed' because the bug affects multiple users.
** Changed in: maas (Ubuntu)
Status: New => Confirmed
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1153077
Title:
rabbi
I think I'm seeing the same issue. I have a celeryd process that appears
to be stuck in a loop and a queue in the vhost /maas_workers named
c1d352af-8060-4bf4-b820-f468b22a97fb. One thing that stands out to me is
that the queue has 4 unacknowledged messages at all times. New ones seem
to come in an
** Summary changed:
- rabbitmq queue fills up and salary stops executing tasks when
upload_dhcp_leases is done every minute (by default)
+ rabbitmq queue fills up and celery stops executing tasks when
upload_dhcp_leases is done every minute (by default)
** Also affects: maas
Importance: Unde
26 matches
Mail list logo