Re: [Spacewalk-list] Check "task queue" for client on server within database? SOLVED!
Already solved it Regards Robert Am 09.12.2015 5:08 nachm. schrieb Jan Dobes : > > On 8.12.2015 10:23 paschedag.netlut...@swr.de wrote: > > Hi everyone, > > > > I just noticed, that several of my clients are not picking up remote > > commands from the server. Although all "remote commands" are allowed on > > the client, scheduling a task stays in "pending" state forever. > > > > On the client, nothing gets logged anymore. It looks, like the client > > does not "find" anything to do in its queue on the server. When I debug > > the "rhn_check" command, this code here fails > > > > action = self.server.queue.get(up2dateAuth.getSystemId(), > > 94 ACTION_VERSION, status_report) > > 95 > > 96 return action > > > > "action" is emtpy every time now. > > > > Any help will be appreciated. > > > > Regards, > > Robert > > > > > > Mit freundlichen Grüßen > > > > Robert Paschedag > > Netlution GmbH > > Landteilstr. 33 > > 68163 Mannheim > > > > im Auftrag des > > SWR > > Südwestrundfunk > > Informations- und Kommunikationssysteme > > Neckarstraße 230 > > 70190 Stuttgart > > > > Telefon +49 (0)711 /929-12654 oder > > Telefon +49 (0)711 /929-13714 > > paschedag.netlut...@swr.de > > > > swr.de > > > > Hello, > > try to run rhn_check -vvv to display more output. Also check what's > happening on server side in /var/log/rhn/rhn_server_xmlrpc.log . > > Regards, > -- > Jan Dobes > Satellite Engineering, Red Hat > > ___ > Spacewalk-list mailing list > Spacewalk-list@redhat.com > https://www.redhat.com/mailman/listinfo/spacewalk-list ___ Spacewalk-list mailing list Spacewalk-list@redhat.com https://www.redhat.com/mailman/listinfo/spacewalk-list
Re: [Spacewalk-list] Check "task queue" for client on server within database?
On 8.12.2015 10:23 paschedag.netlut...@swr.de wrote: Hi everyone, I just noticed, that several of my clients are not picking up remote commands from the server. Although all "remote commands" are allowed on the client, scheduling a task stays in "pending" state forever. On the client, nothing gets logged anymore. It looks, like the client does not "find" anything to do in its queue on the server. When I debug the "rhn_check" command, this code here fails action = self.server.queue.get(up2dateAuth.getSystemId(), 94 ACTION_VERSION, status_report) 95 96 return action "action" is emtpy every time now. Any help will be appreciated. Regards, Robert Mit freundlichen Grüßen Robert Paschedag Netlution GmbH Landteilstr. 33 68163 Mannheim im Auftrag des SWR Südwestrundfunk Informations- und Kommunikationssysteme Neckarstraße 230 70190 Stuttgart Telefon +49 (0)711 /929-12654 oder Telefon +49 (0)711 /929-13714 paschedag.netlut...@swr.de swr.de Hello, try to run rhn_check -vvv to display more output. Also check what's happening on server side in /var/log/rhn/rhn_server_xmlrpc.log . Regards, -- Jan Dobes Satellite Engineering, Red Hat ___ Spacewalk-list mailing list Spacewalk-list@redhat.com https://www.redhat.com/mailman/listinfo/spacewalk-list
Re: [Spacewalk-list] Check "task queue" for client on server within database?
You misunderstand... The filesystem was read-only... I had to reboot the spacewalk server then. Afterwards... everything was read write again. Robert Am 09.12.2015 1:24 vorm. schrieb Paul Robert Marino : > > were there any DBI related errors in the logs? > > On Tue, Dec 8, 2015 at 7:23 PM, Paul Robert Marino > wrote: > > Wait I am confused by this bug report, how can you update a read only > > database? > > You should have gotten an error from the database when it could not > > write to the journal. > > Now PostgreSQL would have probably allowed you to keep reading as long > > as the queries fit within the memory boundaries, but you wouldn't be > > able to update the statuses in that case. > > now if the journal was on a different files system than the data, for > > example you have spacewalk in a separate table space. I have seen > > instances where PostgreSQL will commit to the journal but will fail to > > update the data. In those cases you need to fix the file system and > > then restart the PosgresSQL service to force a journal recovery, then > > you would have to bounce the spacewalk services to get them to > > reconnect. > > > > > > > > > > > > On Tue, Dec 8, 2015 at 3:41 PM, Robert Paschedag > > wrote: > >> So... finally... after long digging... I found my error. > >> > >> Situation: > >> > >> Nearly all clients stopped picking up tasks from spacewalk at the same > >> time. > >> > >> Problem was an error within the database because of a suddenly read only > >> filesystem. All problematic clients had an "open" task, that was in status > >> "picked up", but the client had nothing to return anymore. > >> > >> And all the "new" tasks had not been picked up, because within the handler > >> "queue.get", the handler returns without doing anything if there is still > >> an "open" task for the client. > >> > >> So the solution was to change the status of the "open" task within the > >> database. > >> > >> I put all tasks to "fail" (status code 3) with > >> > >> "update rhnServerActions set status = 3, completed_time = > >> CURRENT_TIMESTAMP where status = 1;" > >> > >> Database is PostgreSQL > >> > >> After that, all clients picked up the "remaining" tasks and the server > >> nearly broke down ;-) > >> > >> Regards > >> Robert > >> Am 08.12.2015 10:23 schrieb paschedag.netlut...@swr.de: > >>> > >>> Hi everyone, > >>> > >>> I just noticed, that several of my clients are not picking up remote > >>> commands from the server. Although all "remote commands" are allowed on > >>> the client, scheduling a task stays in "pending" state forever. > >>> > >>> On the client, nothing gets logged anymore. It looks, like the client > >>> does not "find" anything to do in its queue on the server. When I debug > >>> the "rhn_check" command, this code here fails > >>> > >>> action = self.server.queue.get(up2dateAuth.getSystemId(), > >>> 94 ACTION_VERSION, status_report) > >>> 95 > >>> 96 return action > >>> > >>> "action" is emtpy every time now. > >>> > >>> Any help will be appreciated. > >>> > >>> Regards, > >>> Robert > >>> > >>> > >>> Mit freundlichen Grüßen > >>> > >>> Robert Paschedag > >>> Netlution GmbH > >>> Landteilstr. 33 > >>> 68163 Mannheim > >>> > >>> im Auftrag des > >>> SWR > >>> Südwestrundfunk > >>> Informations- und Kommunikationssysteme > >>> Neckarstraße 230 > >>> 70190 Stuttgart > >>> > >>> Telefon +49 (0)711 /929-12654 oder > >>> Telefon +49 (0)711 /929-13714 > >>> paschedag.netlut...@swr.de > >>> > >>> swr.de > >>> > >> > >> ___ > >> Spacewalk-list mailing list > >> Spacewalk-list@redhat.com > >> https://www.redhat.com/mailman/listinfo/spacewalk-list > > ___ > Spacewalk-list mailing list > Spacewalk-list@redhat.com > https://www.redhat.com/mailman/listinfo/spacewalk-list ___ Spacewalk-list mailing list Spacewalk-list@redhat.com https://www.redhat.com/mailman/listinfo/spacewalk-list
Re: [Spacewalk-list] Check "task queue" for client on server within database?
Wait I am confused by this bug report, how can you update a read only database? You should have gotten an error from the database when it could not write to the journal. Now PostgreSQL would have probably allowed you to keep reading as long as the queries fit within the memory boundaries, but you wouldn't be able to update the statuses in that case. now if the journal was on a different files system than the data, for example you have spacewalk in a separate table space. I have seen instances where PostgreSQL will commit to the journal but will fail to update the data. In those cases you need to fix the file system and then restart the PosgresSQL service to force a journal recovery, then you would have to bounce the spacewalk services to get them to reconnect. On Tue, Dec 8, 2015 at 3:41 PM, Robert Paschedag wrote: > So... finally... after long digging... I found my error. > > Situation: > > Nearly all clients stopped picking up tasks from spacewalk at the same time. > > Problem was an error within the database because of a suddenly read only > filesystem. All problematic clients had an "open" task, that was in status > "picked up", but the client had nothing to return anymore. > > And all the "new" tasks had not been picked up, because within the handler > "queue.get", the handler returns without doing anything if there is still an > "open" task for the client. > > So the solution was to change the status of the "open" task within the > database. > > I put all tasks to "fail" (status code 3) with > > "update rhnServerActions set status = 3, completed_time = CURRENT_TIMESTAMP > where status = 1;" > > Database is PostgreSQL > > After that, all clients picked up the "remaining" tasks and the server nearly > broke down ;-) > > Regards > Robert > Am 08.12.2015 10:23 schrieb paschedag.netlut...@swr.de: >> >> Hi everyone, >> >> I just noticed, that several of my clients are not picking up remote >> commands from the server. Although all "remote commands" are allowed on the >> client, scheduling a task stays in "pending" state forever. >> >> On the client, nothing gets logged anymore. It looks, like the client does >> not "find" anything to do in its queue on the server. When I debug the >> "rhn_check" command, this code here fails >> >> action = self.server.queue.get(up2dateAuth.getSystemId(), >> 94 ACTION_VERSION, status_report) >> 95 >> 96 return action >> >> "action" is emtpy every time now. >> >> Any help will be appreciated. >> >> Regards, >> Robert >> >> >> Mit freundlichen Grüßen >> >> Robert Paschedag >> Netlution GmbH >> Landteilstr. 33 >> 68163 Mannheim >> >> im Auftrag des >> SWR >> Südwestrundfunk >> Informations- und Kommunikationssysteme >> Neckarstraße 230 >> 70190 Stuttgart >> >> Telefon +49 (0)711 /929-12654 oder >> Telefon +49 (0)711 /929-13714 >> paschedag.netlut...@swr.de >> >> swr.de >> > > ___ > Spacewalk-list mailing list > Spacewalk-list@redhat.com > https://www.redhat.com/mailman/listinfo/spacewalk-list ___ Spacewalk-list mailing list Spacewalk-list@redhat.com https://www.redhat.com/mailman/listinfo/spacewalk-list
Re: [Spacewalk-list] Check "task queue" for client on server within database?
were there any DBI related errors in the logs? On Tue, Dec 8, 2015 at 7:23 PM, Paul Robert Marino wrote: > Wait I am confused by this bug report, how can you update a read only > database? > You should have gotten an error from the database when it could not > write to the journal. > Now PostgreSQL would have probably allowed you to keep reading as long > as the queries fit within the memory boundaries, but you wouldn't be > able to update the statuses in that case. > now if the journal was on a different files system than the data, for > example you have spacewalk in a separate table space. I have seen > instances where PostgreSQL will commit to the journal but will fail to > update the data. In those cases you need to fix the file system and > then restart the PosgresSQL service to force a journal recovery, then > you would have to bounce the spacewalk services to get them to > reconnect. > > > > > > On Tue, Dec 8, 2015 at 3:41 PM, Robert Paschedag > wrote: >> So... finally... after long digging... I found my error. >> >> Situation: >> >> Nearly all clients stopped picking up tasks from spacewalk at the same time. >> >> Problem was an error within the database because of a suddenly read only >> filesystem. All problematic clients had an "open" task, that was in status >> "picked up", but the client had nothing to return anymore. >> >> And all the "new" tasks had not been picked up, because within the handler >> "queue.get", the handler returns without doing anything if there is still an >> "open" task for the client. >> >> So the solution was to change the status of the "open" task within the >> database. >> >> I put all tasks to "fail" (status code 3) with >> >> "update rhnServerActions set status = 3, completed_time = CURRENT_TIMESTAMP >> where status = 1;" >> >> Database is PostgreSQL >> >> After that, all clients picked up the "remaining" tasks and the server >> nearly broke down ;-) >> >> Regards >> Robert >> Am 08.12.2015 10:23 schrieb paschedag.netlut...@swr.de: >>> >>> Hi everyone, >>> >>> I just noticed, that several of my clients are not picking up remote >>> commands from the server. Although all "remote commands" are allowed on the >>> client, scheduling a task stays in "pending" state forever. >>> >>> On the client, nothing gets logged anymore. It looks, like the client does >>> not "find" anything to do in its queue on the server. When I debug the >>> "rhn_check" command, this code here fails >>> >>> action = self.server.queue.get(up2dateAuth.getSystemId(), >>> 94 ACTION_VERSION, status_report) >>> 95 >>> 96 return action >>> >>> "action" is emtpy every time now. >>> >>> Any help will be appreciated. >>> >>> Regards, >>> Robert >>> >>> >>> Mit freundlichen Grüßen >>> >>> Robert Paschedag >>> Netlution GmbH >>> Landteilstr. 33 >>> 68163 Mannheim >>> >>> im Auftrag des >>> SWR >>> Südwestrundfunk >>> Informations- und Kommunikationssysteme >>> Neckarstraße 230 >>> 70190 Stuttgart >>> >>> Telefon +49 (0)711 /929-12654 oder >>> Telefon +49 (0)711 /929-13714 >>> paschedag.netlut...@swr.de >>> >>> swr.de >>> >> >> ___ >> Spacewalk-list mailing list >> Spacewalk-list@redhat.com >> https://www.redhat.com/mailman/listinfo/spacewalk-list ___ Spacewalk-list mailing list Spacewalk-list@redhat.com https://www.redhat.com/mailman/listinfo/spacewalk-list
Re: [Spacewalk-list] Check "task queue" for client on server within database?
So... finally... after long digging... I found my error. Situation: Nearly all clients stopped picking up tasks from spacewalk at the same time. Problem was an error within the database because of a suddenly read only filesystem. All problematic clients had an "open" task, that was in status "picked up", but the client had nothing to return anymore. And all the "new" tasks had not been picked up, because within the handler "queue.get", the handler returns without doing anything if there is still an "open" task for the client. So the solution was to change the status of the "open" task within the database. I put all tasks to "fail" (status code 3) with "update rhnServerActions set status = 3, completed_time = CURRENT_TIMESTAMP where status = 1;" Database is PostgreSQL After that, all clients picked up the "remaining" tasks and the server nearly broke down ;-) Regards Robert Am 08.12.2015 10:23 schrieb paschedag.netlut...@swr.de: > > Hi everyone, > > I just noticed, that several of my clients are not picking up remote commands > from the server. Although all "remote commands" are allowed on the client, > scheduling a task stays in "pending" state forever. > > On the client, nothing gets logged anymore. It looks, like the client does > not "find" anything to do in its queue on the server. When I debug the > "rhn_check" command, this code here fails > > action = self.server.queue.get(up2dateAuth.getSystemId(), > 94 ACTION_VERSION, status_report) > 95 > 96 return action > > "action" is emtpy every time now. > > Any help will be appreciated. > > Regards, > Robert > > > Mit freundlichen Grüßen > > Robert Paschedag > Netlution GmbH > Landteilstr. 33 > 68163 Mannheim > > im Auftrag des > SWR > Südwestrundfunk > Informations- und Kommunikationssysteme > Neckarstraße 230 > 70190 Stuttgart > > Telefon +49 (0)711 /929-12654 oder > Telefon +49 (0)711 /929-13714 > paschedag.netlut...@swr.de > > swr.de > ___ Spacewalk-list mailing list Spacewalk-list@redhat.com https://www.redhat.com/mailman/listinfo/spacewalk-list
[Spacewalk-list] Check "task queue" for client on server within database?
Hi everyone, I just noticed, that several of my clients are not picking up remote commands from the server. Although all "remote commands" are allowed on the client, scheduling a task stays in "pending" state forever. On the client, nothing gets logged anymore. It looks, like the client does not "find" anything to do in its queue on the server. When I debug the "rhn_check" command, this code here fails action = self.server.queue.get(up2dateAuth.getSystemId(), 94 ACTION_VERSION, status_report) 95 96 return action "action" is emtpy every time now. Any help will be appreciated. Regards, Robert Mit freundlichen Grüßen Robert Paschedag Netlution GmbH Landteilstr. 33 68163 Mannheim im Auftrag des SWR Südwestrundfunk Informations- und Kommunikationssysteme Neckarstraße 230 70190 Stuttgart Telefon +49 (0)711 /929-12654 oder Telefon +49 (0)711 /929-13714 paschedag.netlut...@swr.de swr.de ___ Spacewalk-list mailing list Spacewalk-list@redhat.com https://www.redhat.com/mailman/listinfo/spacewalk-list
