On 12.1.2016 09:08 paschedag.netlut...@swr.de wrote:
Hello again,
still having problems,
I noticed that the errata-cache gets triggerd when a client refreshes
its repos or searches for updates. So I manually fired a "zypper ref -s"
or a "apt-get update" on every client connected to this spacewalk
server. This seems to have fixed "something". Suddenly, the
"errata-cache" task has finished and I got some backtraces from spacewalk.
But I still have trouble. The UI does not seem to get updated correctly
(on the "system" page, nearly all systems look "up-to-date". But I know,
it's not!). But the updates are available to the clients (searching on
the client).
When I look into the last errata-cache schedules (in Spacewalk "Admin"
-> "Task schedules" -> "errata-cache-bunch") I can see, that the
"errata-cache" tasks is often "SKIPPED"
errata-cache _2016-01-12 08:50:00 MEZ _
<https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=132>
2016-01-12 08:50:00 MEZ SKIPPED
errata-cache _2016-01-12 08:49:00 MEZ _
<https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=132>
2016-01-12 08:49:00 MEZ SKIPPED
errata-cache _2016-01-12 08:48:00 MEZ _
<https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=132>
2016-01-12 08:48:00 MEZ SKIPPED
errata-cache _2016-01-12 08:47:00 MEZ _
<https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=132>
2016-01-12 08:50:00 MEZ FINISHED
errata-cache _2016-01-12 08:46:00 MEZ _
<https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=132>
2016-01-12 08:46:00 MEZ SKIPPED
errata-cache _2016-01-12 08:45:00 MEZ _
<https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=132>
2016-01-12 08:45:00 MEZ SKIPPED
errata-cache _2016-01-12 08:44:00 MEZ _
<https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=132>
2016-01-12 08:46:00 MEZ FINISHED
errata-mailer _2016-01-12 08:43:00 MEZ _
<https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=132>
2016-01-12 08:43:00 MEZ FINISHED
errata-cache _2016-01-12 08:43:00 MEZ _
<https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=132>
2016-01-12 08:43:00 MEZ FINISHED
errata-cache _2016-01-12 08:42:00 MEZ _
<https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=132>
2016-01-12 08:42:00 MEZ SKIPPED
errata-cache _2016-01-12 08:41:00 MEZ _
<https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=132>
2016-01-12 08:41:00 MEZ SKIPPED
errata-cache _2016-01-12 08:40:00 MEZ _
<https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=132>
2016-01-12 08:40:00 MEZ SKIPPED
errata-cache _2016-01-12 08:39:00 MEZ _
<https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=132>
2016-01-12 08:42:00 MEZ FINISHED
errata-mailer _2016-01-12 08:38:00 MEZ _
<https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=132>
2016-01-12 08:38:00 MEZ FINISHED
errata-cache _2016-01-12 08:38:00 MEZ _
<https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=132>
2016-01-12 08:38:00 MEZ FINISHED
errata-mailer _2016-01-12 08:37:00 MEZ _
<https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=132>
2016-01-12 08:37:00 MEZ FINISHED
errata-cache _2016-01-12 08:37:00 MEZ _
<https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=132>
2016-01-12 08:37:00 MEZ FINISHED
errata-cache _2016-01-12 08:36:00 MEZ _
<https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=132>
2016-01-12 08:36:00 MEZ SKIPPED
errata-cache _2016-01-12 08:35:00 MEZ _
<https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=132>
2016-01-12 08:35:00 MEZ SKIPPED
errata-cache _2016-01-12 08:34:00 MEZ _
<https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=132>
2016-01-12 08:36:00 MEZ FINISHED
errata-mailer _2016-01-12 08:33:00 MEZ _
<https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=132>
2016-01-12 08:33:00 MEZ FINISHED
errata-cache _2016-01-12 08:33:00 MEZ _
<https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=132>
2016-01-12 08:33:00 MEZ FINISHED
errata-mailer _2016-01-12 08:32:00 MEZ _
<https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=132>
2016-01-12 08:32:00 MEZ FINISHED
errata-cache _2016-01-12 08:32:00 MEZ _
<https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=132>
2016-01-12 08:32:00 MEZ FINISHED
errata-mailer _2016-01-12 08:31:00 MEZ _
<https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=132>
2016-01-12 08:31:00 MEZ FINISHED
errata-cache _2016-01-12 08:31:00 MEZ _
<https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=132>
2016-01-12 08:31:00 MEZ FINISHED
errata-mailer _2016-01-12 08:30:00 MEZ _
<https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=132>
2016-01-12 08:30:00 MEZ FINISHED
errata-cache _2016-01-12 08:30:00 MEZ _
<https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=132>
2016-01-12 08:30:00 MEZ FINISHED
errata-cache _2016-01-12 08:29:00 MEZ _
<https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=132>
2016-01-12 08:29:00 MEZ SKIPPED
And also I noticed that the "errata-mailer" tasks seems to get only
started, when the "errata-cache" tasks finished within one minute. (Just
look above). As soon as the errata-cache tasks runs longer than one
minute, the "errata-mailer" is not started. I'm also not sure, if
something went wrong, while I was shutting down the spacewalk server for
the update.
This is what it looked before (on spacewalk 2.2). I started the update
on 2016-01-04 abount 08:30.
errata-cache _2016-01-04 10:10:00 MEZ _
<https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=7>
2016-01-04 10:10:00 MEZ SKIPPED
errata-cache _2016-01-04 10:00:00 MEZ _
<https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=7>
2016-01-04 10:00:00 MEZ SKIPPED
errata-cache _2016-01-04 09:50:00 MEZ _
<https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=7>
2016-01-04 09:50:00 MEZ SKIPPED
errata-cache _2016-01-04 09:40:00 MEZ _
<https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=7>
2016-01-04 09:40:00 MEZ SKIPPED
errata-cache _2016-01-04 09:30:00 MEZ _
<https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=7>
2016-01-04 09:30:00 MEZ SKIPPED
errata-cache _2016-01-04 09:20:00 MEZ _
<https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=7>
2016-01-04 09:20:00 MEZ SKIPPED
errata-cache _2016-01-04 09:10:00 MEZ _
<https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=7>
2016-01-04 09:10:00 MEZ SKIPPED
errata-cache _2016-01-04 09:00:00 MEZ _
<https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=7>
2016-01-04 09:00:00 MEZ SKIPPED
errata-cache _2016-01-04 08:50:00 MEZ _
<https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=7>
2016-01-04 08:50:00 MEZ SKIPPED
errata-cache _2016-01-04 08:40:00 MEZ _
<https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=7>
2016-01-04 08:40:00 MEZ SKIPPED
errata-cache _2016-01-04 08:30:00 MEZ _
<https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=7>
2016-01-05 08:59:59 MEZ INTERRUPTED
errata-mailer _2016-01-04 08:30:00 MEZ _
<https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=7>
2016-01-04 08:29:57 MEZ INTERRUPTED
errata-mailer _2016-01-04 08:00:00 MEZ _
<https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=7>
2016-01-04 08:00:00 MEZ FINISHED
errata-cache _2016-01-04 08:00:00 MEZ _
<https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=7>
2016-01-04 08:00:00 MEZ FINISHED
errata-mailer _2016-01-04 07:50:00 MEZ _
<https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=7>
2016-01-04 07:50:00 MEZ FINISHED
errata-cache _2016-01-04 07:50:00 MEZ _
<https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=7>
2016-01-04 07:50:00 MEZ FINISHED
I would be glad, if someone could give me further hints, where to look
for the root cause to this.
Kind regards,
Robert
Mit freundlichen Grüßen
Robert Paschedag
Netlution GmbH
Landteilstr. 33
68163 Mannheim
im Auftrag des
SWR
Südwestrundfunk
Informations- und Kommunikationssysteme
Neckarstraße 230
70190 Stuttgart
Telefon +49 (0)711 /929-12654 oder
Telefon +49 (0)711 /929-13714
paschedag.netlut...@swr.de
swr.de
Von: Robert Paschedag <robert.pasche...@web.de>
An: paschedag.netlut...@swr.de
Kopie: spacewalk-list@redhat.com
Datum: 08.01.2016 21:53
Betreff: Re: [Spacewalk-list] Errata Cache bunch running forever on
Spacewalk 2.4
------------------------------------------------------------------------
Hi again,
just some further information.
Because I already activated debug logging of taskomatic, searching for
server errata cache tasks within the rhn_taskomatic_daemon logfiles with
grep 'Updating errata cache for server' rhn_taskomatic_daemon.log* | awk
'{print $NF; }' | sort | uniq -c
i get
14 [1000010274]
14 [1000010299]
4 [1000010504]
14 [1000010512]
14 [1000010516]
14 [1000010517]
14 [1000010518]
14 [1000010519]
14 [1000010520]
14 [1000010521]
14 [1000010522]
14 [1000010523]
14 [1000010524]
14 [1000010527]
14 [1000010528]
14 [1000010529]
14 [1000010530]
14 [1000010531]
14 [1000010532]
14 [1000010564]
14 [1000010570]
15 [1000010573]
15 [1000010593]
13 [1000010639]
14 [1000010640]
15 [1000010641]
13 [1000010642]
14 [1000010647]
13 [1000010648]
14 [1000010649]
15 [1000010650]
14 [1000010651]
14 [1000010652]
6 [1000010672]
15 [1000010683]
14 [1000010684]
14 [1000010685]
14 [1000010687]
14 [1000010688]
14 [1000010689]
14 [1000010690]
14 [1000010691]
14 [1000010692]
14 [1000010693]
14 [1000010694]
15 [1000010695]
14 [1000010697]
14 [1000010698]
2 [1000010702]
14 [1000010705]
14 [1000010710]
2 [1000010715]
On this server, there are more than 400 servers registered, so this
looks like, something is "hanging" on these servers and not going any
further (i think).
But right now, I don't know - exactly - where to look for this "error"
or within which DB table to further track this down.
Regards,
Robert
Am 08.01.2016 um 19:31 schrieb Robert Paschedag:
> Can anybody give a hint, in which db table to search for these
errata-cache tasks, that are started over and over again? I just don't
get it myself right now . It also looks like that it is always the same
prox. 50 servers that this occurs.
>
> Regards
> Robert
> Am 07.01.2016 10:11 vorm. schrieb paschedag.netlut...@swr.de:
>>
>> Hi all,
>>
>> it looks like one (or more?) jobs have been interrupted, while the
spacewalk service had shut down for upgrade. I upgraded my spacewalk
server from 2.2 -> 2.3 -> 2.4.
>>
>> Now it looks like the errata-cache bunch tasks is running forever
and the errata notification mail is not started anymore. I think it gets
called from errata-cache, when this has finished.
>>
>> Errata Cache:
>> 2016-01-07 10:00:00 MEZ
>> RUNNING
>> Errata Notification Mail:
>> 2016-01-04 08:30:00 MEZ
>> INTERRUPTED
>>
>>
>> When I look into the database, I can see several tasks in the
rhntaskqueue but I'm pretty sure, that these tasks do not get cleared
correctly, so the task is run for every server over and over again. I'm
not sure! This is just my thinking.
>>
>> rhnschema=# select * from rhntaskqueue;
>> org_id | task_name | task_data | priority | earliest
>>
--------+----------------------------+------------+----------+-------------------------------
>> 1 | update_server_errata_cache | 1000010701 | 0 |
2016-01-07 10:00:45.963006+01
>> 1 | update_server_errata_cache | 1000010274 | 0 |
2016-01-07 10:00:53.198001+01
>> 1 | update_server_errata_cache | 1000010527 | 0 |
2016-01-07 10:01:03.768592+01
>> 1 | update_server_errata_cache | 1000010689 | 0 |
2016-01-07 10:01:56.915296+01
>> 1 | update_server_errata_cache | 1000010641 | 0 |
2016-01-07 10:02:41.775593+01
>> 1 | update_server_errata_cache | 1000010687 | 0 |
2016-01-07 10:05:01.471365+01
>> 1 | update_server_errata_cache | 1000010650 | 0 |
2016-01-07 10:05:26.492405+01
>> 1 | update_server_errata_cache | 1000010520 | 0 |
2016-01-07 10:06:38.052352+01
>> (8 Zeilen)
>>
>> rhnschema=#
>>
>> Any help will be appreciated to debug this further.
>>
>> Regards,
>> Robert
>>
>>
>> Mit freundlichen Grüßen
>>
>> Robert Paschedag
>> Netlution GmbH
>> Landteilstr. 33
>> 68163 Mannheim
>>
>> im Auftrag des
>> SWR
>> Südwestrundfunk
>> Informations- und Kommunikationssysteme
>> Neckarstraße 230
>> 70190 Stuttgart
>>
>> Telefon +49 (0)711 /929-12654 oder
>> Telefon +49 (0)711 /929-13714
>> paschedag.netlut...@swr.de
>>
>> swr.de
>>
>
> _______________________________________________
> Spacewalk-list mailing list
> Spacewalk-list@redhat.com
> https://www.redhat.com/mailman/listinfo/spacewalk-list
>
Hello,
since Spacewalk 2.3 this type of task is running every minute instead of
every ten minutes before. However any of these tasks should not take so
long. These tasks are picked by taskomatic daemon and processed. Are
there any exceptions in /var/log/rhn/rhn_taskomatic_daemon.log ? Try to
restart taskomatic.
Regards,
--
Jan Dobes
Satellite Engineering, Red Hat
_______________________________________________
Spacewalk-list mailing list
Spacewalk-list@redhat.com
https://www.redhat.com/mailman/listinfo/spacewalk-list
