Hello again,
still having problems,
I noticed that the errata-cache gets triggerd when a client refreshes its
repos or searches for updates. So I manually fired a "zypper ref -s" or a
"apt-get update" on every client connected to this spacewalk server. This
seems to have fixed "something". Suddenly, the "errata-cache" task has
finished and I got some backtraces from spacewalk.
But I still have trouble. The UI does not seem to get updated correctly
(on the "system" page, nearly all systems look "up-to-date". But I know,
it's not!). But the updates are available to the clients (searching on the
client).
When I look into the last errata-cache schedules (in Spacewalk "Admin" ->
"Task schedules" -> "errata-cache-bunch") I can see, that the
"errata-cache" tasks is often "SKIPPED"
errata-cache
2016-01-12 08:50:00 MEZ
2016-01-12 08:50:00 MEZ
SKIPPED
errata-cache
2016-01-12 08:49:00 MEZ
2016-01-12 08:49:00 MEZ
SKIPPED
errata-cache
2016-01-12 08:48:00 MEZ
2016-01-12 08:48:00 MEZ
SKIPPED
errata-cache
2016-01-12 08:47:00 MEZ
2016-01-12 08:50:00 MEZ
FINISHED
errata-cache
2016-01-12 08:46:00 MEZ
2016-01-12 08:46:00 MEZ
SKIPPED
errata-cache
2016-01-12 08:45:00 MEZ
2016-01-12 08:45:00 MEZ
SKIPPED
errata-cache
2016-01-12 08:44:00 MEZ
2016-01-12 08:46:00 MEZ
FINISHED
errata-mailer
2016-01-12 08:43:00 MEZ
2016-01-12 08:43:00 MEZ
FINISHED
errata-cache
2016-01-12 08:43:00 MEZ
2016-01-12 08:43:00 MEZ
FINISHED
errata-cache
2016-01-12 08:42:00 MEZ
2016-01-12 08:42:00 MEZ
SKIPPED
errata-cache
2016-01-12 08:41:00 MEZ
2016-01-12 08:41:00 MEZ
SKIPPED
errata-cache
2016-01-12 08:40:00 MEZ
2016-01-12 08:40:00 MEZ
SKIPPED
errata-cache
2016-01-12 08:39:00 MEZ
2016-01-12 08:42:00 MEZ
FINISHED
errata-mailer
2016-01-12 08:38:00 MEZ
2016-01-12 08:38:00 MEZ
FINISHED
errata-cache
2016-01-12 08:38:00 MEZ
2016-01-12 08:38:00 MEZ
FINISHED
errata-mailer
2016-01-12 08:37:00 MEZ
2016-01-12 08:37:00 MEZ
FINISHED
errata-cache
2016-01-12 08:37:00 MEZ
2016-01-12 08:37:00 MEZ
FINISHED
errata-cache
2016-01-12 08:36:00 MEZ
2016-01-12 08:36:00 MEZ
SKIPPED
errata-cache
2016-01-12 08:35:00 MEZ
2016-01-12 08:35:00 MEZ
SKIPPED
errata-cache
2016-01-12 08:34:00 MEZ
2016-01-12 08:36:00 MEZ
FINISHED
errata-mailer
2016-01-12 08:33:00 MEZ
2016-01-12 08:33:00 MEZ
FINISHED
errata-cache
2016-01-12 08:33:00 MEZ
2016-01-12 08:33:00 MEZ
FINISHED
errata-mailer
2016-01-12 08:32:00 MEZ
2016-01-12 08:32:00 MEZ
FINISHED
errata-cache
2016-01-12 08:32:00 MEZ
2016-01-12 08:32:00 MEZ
FINISHED
errata-mailer
2016-01-12 08:31:00 MEZ
2016-01-12 08:31:00 MEZ
FINISHED
errata-cache
2016-01-12 08:31:00 MEZ
2016-01-12 08:31:00 MEZ
FINISHED
errata-mailer
2016-01-12 08:30:00 MEZ
2016-01-12 08:30:00 MEZ
FINISHED
errata-cache
2016-01-12 08:30:00 MEZ
2016-01-12 08:30:00 MEZ
FINISHED
errata-cache
2016-01-12 08:29:00 MEZ
2016-01-12 08:29:00 MEZ
SKIPPED
And also I noticed that the "errata-mailer" tasks seems to get only
started, when the "errata-cache" tasks finished within one minute. (Just
look above). As soon as the errata-cache tasks runs longer than one
minute, the "errata-mailer" is not started. I'm also not sure, if
something went wrong, while I was shutting down the spacewalk server for
the update.
This is what it looked before (on spacewalk 2.2). I started the update on
2016-01-04 abount 08:30.
errata-cache
2016-01-04 10:10:00 MEZ
2016-01-04 10:10:00 MEZ
SKIPPED
errata-cache
2016-01-04 10:00:00 MEZ
2016-01-04 10:00:00 MEZ
SKIPPED
errata-cache
2016-01-04 09:50:00 MEZ
2016-01-04 09:50:00 MEZ
SKIPPED
errata-cache
2016-01-04 09:40:00 MEZ
2016-01-04 09:40:00 MEZ
SKIPPED
errata-cache
2016-01-04 09:30:00 MEZ
2016-01-04 09:30:00 MEZ
SKIPPED
errata-cache
2016-01-04 09:20:00 MEZ
2016-01-04 09:20:00 MEZ
SKIPPED
errata-cache
2016-01-04 09:10:00 MEZ
2016-01-04 09:10:00 MEZ
SKIPPED
errata-cache
2016-01-04 09:00:00 MEZ
2016-01-04 09:00:00 MEZ
SKIPPED
errata-cache
2016-01-04 08:50:00 MEZ
2016-01-04 08:50:00 MEZ
SKIPPED
errata-cache
2016-01-04 08:40:00 MEZ
2016-01-04 08:40:00 MEZ
SKIPPED
errata-cache
2016-01-04 08:30:00 MEZ
2016-01-05 08:59:59 MEZ
INTERRUPTED
errata-mailer
2016-01-04 08:30:00 MEZ
2016-01-04 08:29:57 MEZ
INTERRUPTED
errata-mailer
2016-01-04 08:00:00 MEZ
2016-01-04 08:00:00 MEZ
FINISHED
errata-cache
2016-01-04 08:00:00 MEZ
2016-01-04 08:00:00 MEZ
FINISHED
errata-mailer
2016-01-04 07:50:00 MEZ
2016-01-04 07:50:00 MEZ
FINISHED
errata-cache
2016-01-04 07:50:00 MEZ
2016-01-04 07:50:00 MEZ
FINISHED
I would be glad, if someone could give me further hints, where to look for
the root cause to this.
Kind regards,
Robert
Mit freundlichen Grüßen
Robert Paschedag
Netlution GmbH
Landteilstr. 33
68163 Mannheim
im Auftrag des
SWR
Südwestrundfunk
Informations- und Kommunikationssysteme
Neckarstraße 230
70190 Stuttgart
Telefon +49 (0)711 /929-12654 oder
Telefon +49 (0)711 /929-13714
paschedag.netlut...@swr.de
swr.de
Von: Robert Paschedag <robert.pasche...@web.de>
An: paschedag.netlut...@swr.de
Kopie: spacewalk-list@redhat.com
Datum: 08.01.2016 21:53
Betreff: Re: [Spacewalk-list] Errata Cache bunch running forever on
Spacewalk 2.4
Hi again,
just some further information.
Because I already activated debug logging of taskomatic, searching for
server errata cache tasks within the rhn_taskomatic_daemon logfiles with
grep 'Updating errata cache for server' rhn_taskomatic_daemon.log* | awk
'{print $NF; }' | sort | uniq -c
i get
14 [1000010274]
14 [1000010299]
4 [1000010504]
14 [1000010512]
14 [1000010516]
14 [1000010517]
14 [1000010518]
14 [1000010519]
14 [1000010520]
14 [1000010521]
14 [1000010522]
14 [1000010523]
14 [1000010524]
14 [1000010527]
14 [1000010528]
14 [1000010529]
14 [1000010530]
14 [1000010531]
14 [1000010532]
14 [1000010564]
14 [1000010570]
15 [1000010573]
15 [1000010593]
13 [1000010639]
14 [1000010640]
15 [1000010641]
13 [1000010642]
14 [1000010647]
13 [1000010648]
14 [1000010649]
15 [1000010650]
14 [1000010651]
14 [1000010652]
6 [1000010672]
15 [1000010683]
14 [1000010684]
14 [1000010685]
14 [1000010687]
14 [1000010688]
14 [1000010689]
14 [1000010690]
14 [1000010691]
14 [1000010692]
14 [1000010693]
14 [1000010694]
15 [1000010695]
14 [1000010697]
14 [1000010698]
2 [1000010702]
14 [1000010705]
14 [1000010710]
2 [1000010715]
On this server, there are more than 400 servers registered, so this
looks like, something is "hanging" on these servers and not going any
further (i think).
But right now, I don't know - exactly - where to look for this "error"
or within which DB table to further track this down.
Regards,
Robert
Am 08.01.2016 um 19:31 schrieb Robert Paschedag:
> Can anybody give a hint, in which db table to search for these
errata-cache tasks, that are started over and over again? I just don't get
it myself right now . It also looks like that it is always the same prox.
50 servers that this occurs.
>
> Regards
> Robert
> Am 07.01.2016 10:11 vorm. schrieb paschedag.netlut...@swr.de:
>>
>> Hi all,
>>
>> it looks like one (or more?) jobs have been interrupted, while the
spacewalk service had shut down for upgrade. I upgraded my spacewalk
server from 2.2 -> 2.3 -> 2.4.
>>
>> Now it looks like the errata-cache bunch tasks is running forever and
the errata notification mail is not started anymore. I think it gets
called from errata-cache, when this has finished.
>>
>> Errata Cache:
>> 2016-01-07 10:00:00 MEZ
>> RUNNING
>> Errata Notification Mail:
>> 2016-01-04 08:30:00 MEZ
>> INTERRUPTED
>>
>>
>> When I look into the database, I can see several tasks in the
rhntaskqueue but I'm pretty sure, that these tasks do not get cleared
correctly, so the task is run for every server over and over again. I'm
not sure! This is just my thinking.
>>
>> rhnschema=# select * from rhntaskqueue;
>> org_id | task_name | task_data | priority |
earliest
>>
--------+----------------------------+------------+----------+-------------------------------
>> 1 | update_server_errata_cache | 1000010701 | 0 |
2016-01-07 10:00:45.963006+01
>> 1 | update_server_errata_cache | 1000010274 | 0 |
2016-01-07 10:00:53.198001+01
>> 1 | update_server_errata_cache | 1000010527 | 0 |
2016-01-07 10:01:03.768592+01
>> 1 | update_server_errata_cache | 1000010689 | 0 |
2016-01-07 10:01:56.915296+01
>> 1 | update_server_errata_cache | 1000010641 | 0 |
2016-01-07 10:02:41.775593+01
>> 1 | update_server_errata_cache | 1000010687 | 0 |
2016-01-07 10:05:01.471365+01
>> 1 | update_server_errata_cache | 1000010650 | 0 |
2016-01-07 10:05:26.492405+01
>> 1 | update_server_errata_cache | 1000010520 | 0 |
2016-01-07 10:06:38.052352+01
>> (8 Zeilen)
>>
>> rhnschema=#
>>
>> Any help will be appreciated to debug this further.
>>
>> Regards,
>> Robert
>>
>>
>> Mit freundlichen Grüßen
>>
>> Robert Paschedag
>> Netlution GmbH
>> Landteilstr. 33
>> 68163 Mannheim
>>
>> im Auftrag des
>> SWR
>> Südwestrundfunk
>> Informations- und Kommunikationssysteme
>> Neckarstraße 230
>> 70190 Stuttgart
>>
>> Telefon +49 (0)711 /929-12654 oder
>> Telefon +49 (0)711 /929-13714
>> paschedag.netlut...@swr.de
>>
>> swr.de
>>
>
> _______________________________________________
> Spacewalk-list mailing list
> Spacewalk-list@redhat.com
> https://www.redhat.com/mailman/listinfo/spacewalk-list
>
_______________________________________________
Spacewalk-list mailing list
Spacewalk-list@redhat.com
https://www.redhat.com/mailman/listinfo/spacewalk-list
