Re: [Spacewalk-list] ? Failures Cloning Channels with spacewalk-manage-channel-lifecycle

[Boyd, Robert]

I'm happy to report that after waiting a few days and letting the monthly 
cleanup /tuning jobs run on my spacewalk master server, the cloning process is 
working for all of the channels that were failing.

I suspect that a previous monthly job didn't complete properly due to running 
out of memory and left something problematic behind somewhere.  

I managed to clone 3 channel groups this morning without a hiccough much to my 
relief.   Just in time for the beginning of our patch cycle this quarter.


Robert Boyd
Sr. Systems Engineer 
PeopleFluent
e. robert.b...@peoplefluent.com


Click here to experience the power of the new PeopleFluent Mirror Suite ™
Visit: www.peoplefluent.com | Read: PeopleFluent Blog | Follow: @PeopleFluent



-Original Message-
From: spacewalk-list-boun...@redhat.com 
[mailto:spacewalk-list-boun...@redhat.com] On Behalf Of Boyd, Robert
Sent: Friday, January 29, 2016 2:45 PM
To: spacewalk-list@redhat.com
Subject: Re: [Spacewalk-list] ? Failures Cloning Channels with 
spacewalk-manage-channel-lifecycle

Here's some additional info about one of the failure modes I'm seeing.  I kept 
cranking up the memory limits until I now have the limits at default =4192 
max=16388.   Some channels are handled ok by this tool now.   

Others fail pretty quickly while clearing the first target channel.



___
Spacewalk-list mailing list
Spacewalk-list@redhat.com
https://www.redhat.com/mailman/listinfo/spacewalk-list

Re: [Spacewalk-list] ? Failures Cloning Channels with spacewalk-manage-channel-lifecycle

[Boyd, Robert]

Here's some additional info about one of the failure modes I'm seeing.  I kept 
cranking up the memory limits until I now have the limits at default =4192 
max=16388.   Some channels are handled ok by this tool now.   

Others fail pretty quickly while clearing the first target channel.

One thing I see that concerns me are processes I see running Java that don't 
have the increased memory allocations.   Like this one:

root  3595  3349  6 14:32 ?00:00:13 
/usr/bin/java -Dibm.dst.compatibility=true -Xms1024m -Xmx4096m 
-Djava.library.path=/usr/lib:/usr/lib64:/usr/lib/oracle/11.2/client64/lib:/usr/lib/oracle/11.2/client/lib
 
-classpath 
/usr/share/java/tanukiwrapper.jar:/usr/share/rhn/classes:/usr/share/java/struts.jar:/usr/share/java/jfreechart.jar:/usr/share/java/jpam.jar:/usr/share/java/javamail.jar:/usr/share/java/axis/axis-ant.jar:/usr/share/java/quartz.jar:/usr/share/java/commons-codec.jar:/usr/share/java/commons-beanutils.jar:/usr/share/java/ojdbc14.jar:/usr/share/java/jta.jar:/usr/share/java/concurrent.jar:/usr/share/rhn/lib/spacewalk-asm.jar:/usr/share/java/axis/jaxrpc.jar:/usr/share/java/commons-collections.jar:/usr/share/java/taglibs-standard.jar:/usr/share/java/axis/axis.jar:/usr/share/java/xalan-j2.jar:/usr/share/java/commons-validator.jar:/usr/share/java/asm/asm-attrs.jar:/usr/share/java/jaf.jar:/usr/share/java/jdom.jar:/usr/share/java/oro.jar:/usr/share/rhn/lib/rhn.jar:/usr/share/java/redstone-xmlrpc.jar:/usr/share/java/oscache.jar:/usr/share/java/log4j.jar:/usr/share/java/wsdl4j.jar:/usr/share/java/jcommon.jar:/usr/share/java/commons-el.jar:/usr/share/java/taglibs-core.jar:/usr/share/java/commons-lang.jar:/usr/share/java/commons-digester.jar:/usr/share/java/jasper5-runtime.jar:/usr/share/java/jspapi.jar:/usr/share/java/c3p0.jar:/usr/share/java/sitemesh.jar:/usr/share/java/jasper5-compiler.jar:/usr/share/java/axis/saaj.jar:/usr/share/java/commons-logging.jar:/usr/share/java/commons-discovery.jar:/usr/share/java/xml-commons-apis.jar:/usr/share/java/axis/jaxrpc.jar:/usr/share/java/commons-cli.jar:/usr/share/java/bcel.jar:/usr/share/java/antlr.jar:/usr/share/java/xerces-j2.jar:/usr/share/java/hibernate3.jar:/usr/share/java/cglib.jar:/usr/share/java/dom4j.jar:/usr/share/rhn/lib/java-branding.jar:/usr/share/java/slf4j/api.jar:/usr/share/java/slf4j/jcl.jar:/usr/share/java/simple-core.jar:/usr/share/java/commons-dbcp.jar:/usr/share/java/commons-pool.jar:/usr/share/java/quartz-oracle.jar:/usr/share/java/postgresql-jdbc.jar:/usr/share/java/mchange-commons.jar:/usr/share/java/commons-io.jar
 
-Dwrapper.key=Ny4YUY7QmPOi432B 
-Dwrapper.port=32001 
-Dwrapper.jvm.port.min=31000 
-Dwrapper.jvm.port.max=31999 
-Dwrapper.pid=3349 -Dwrapper.version=3.2.3 
-Dwrapper.native_library=wrapper 
-Dwrapper.service=TRUE 
-Dwrapper.cpu.timeout=10 
-Dwrapper.jvmid=3
com.redhat.rhn.taskomatic.core.TaskomaticDaemon

What is the configuration file that controls the memory allocation for this 
process?

I already modified /usr/share/rhn/config-defaults/rhn_taskomatic_daemon.conf:

# Initial Java Heap Size (in MB)
wrapper.java.initmemory=4196

# Maximum Java Heap Size (in MB)
wrapper.java.maxmemory=16388

If the process I show is supposed to be controlled by this file, the memory 
allocation specified is being ignored, or applied in some way that doesn't 
match with the process shown.


: spacewalk-list-boun...@redhat.com [mailto:spacewalk-list-boun...@redhat.com] 
On Behalf Of Boyd, Robert
Sent: Friday, January 29, 2016 12:27 PM
To: spacewalk-list@redhat.com
Subject: Re: [Spacewalk-list] ? Failures Cloning Channels with 
spacewalk-manage-channel-lifecycle

Thanks Avi,

I tried what you suggested and got some minor improvement -- the tool runs 
longer before things blow up.   I tried increasing the memory limits even more 
-- but that doesn't seem to make any difference.   It's still eventually 
blowing up the same way, and the logs you suggested looking at don't show 
anything interesting around the time of the failures.

Any other suggestions?   I could keep increasing the memory limits even 
farther, but I'm not sure that's a good idea.

Thanks,
Robert

-Original Message-
From: spacewalk-list-boun...@redhat.com 
[mailto:spacewalk-list-boun...@redhat.com] On Behalf Of Avi Miller
Sent: Thursday, January 28, 2016 6:12 PM
To: spacewalk-list@redhat.com
Subject: Re: [Spacewalk-list] ? Failures Cloning Channels with 
spacewalk-manage-channel-lifecycle

Hey,

> On 29 Jan 2016, at 9:56 AM, Boyd, Robert  wrote:
> 
> I have tried poking at log files, but I haven’t found where there seems to be 
> an indication of a problem.   Can someone point me to a prioritized list of 
> logs I should be looking at to find the nature of what’s dying on the back 
> end?  I suspect there is a java resource failure or some other error and 
> would like to track it down and fix it rather than resort to manual/scripted 
> channel by channel cloning( which might 

Re: [Spacewalk-list] ? Failures Cloning Channels with spacewalk-manage-channel-lifecycle

[Boyd, Robert]

Thanks Avi,

I tried what you suggested and got some minor improvement -- the tool runs 
longer before things blow up.   I tried increasing the memory limits even more 
-- but that doesn't seem to make any difference.   It's still eventually 
blowing up the same way, and the logs you suggested looking at don't show 
anything interesting around the time of the failures.

Any other suggestions?   I could keep increasing the memory limits even 
farther, but I'm not sure that's a good idea.

Thanks,
Robert

-Original Message-
From: spacewalk-list-boun...@redhat.com 
[mailto:spacewalk-list-boun...@redhat.com] On Behalf Of Avi Miller
Sent: Thursday, January 28, 2016 6:12 PM
To: spacewalk-list@redhat.com
Subject: Re: [Spacewalk-list] ? Failures Cloning Channels with 
spacewalk-manage-channel-lifecycle

Hey,

> On 29 Jan 2016, at 9:56 AM, Boyd, Robert  wrote:
> 
> I have tried poking at log files, but I haven’t found where there seems to be 
> an indication of a problem.   Can someone point me to a prioritized list of 
> logs I should be looking at to find the nature of what’s dying on the back 
> end?  I suspect there is a java resource failure or some other error and 
> would like to track it down and fix it rather than resort to manual/scripted 
> channel by channel cloning( which might fail in the same way).

You may want to increase the memory available to Tomcat and Taskomatic: 

http://docs.oracle.com/cd/E37670_01/E58680/html/sw22-issues-memory.html

This feels like a Tomcat issue to me, the logs are in /var/log/tomcat6 on the 
Spacewalk server. The Taskomatic logs are in 
/var/log/rhn/rhn_taskomatic_daemon.log

Cheers,
Avi

--
Oracle <http://www.oracle.com>
Avi Miller | Product Management Director | +61 (3) 8616 3496 Oracle Linux and 
Virtualization
417 St Kilda Road, Melbourne, Victoria 3004 Australia


___
Spacewalk-list mailing list
Spacewalk-list@redhat.com
https://www.redhat.com/mailman/listinfo/spacewalk-list

___
Spacewalk-list mailing list
Spacewalk-list@redhat.com
https://www.redhat.com/mailman/listinfo/spacewalk-list

Re: [Spacewalk-list] ? Failures Cloning Channels with spacewalk-manage-channel-lifecycle

[Avi Miller]

Hey,

> On 29 Jan 2016, at 9:56 AM, Boyd, Robert  wrote:
> 
> I have tried poking at log files, but I haven’t found where there seems to be 
> an indication of a problem.   Can someone point me to a prioritized list of 
> logs I should be looking at to find the nature of what’s dying on the back 
> end?  I suspect there is a java resource failure or some other error and 
> would like to track it down and fix it rather than resort to manual/scripted 
> channel by channel cloning( which might fail in the same way).

You may want to increase the memory available to Tomcat and Taskomatic: 

http://docs.oracle.com/cd/E37670_01/E58680/html/sw22-issues-memory.html

This feels like a Tomcat issue to me, the logs are in /var/log/tomcat6 on the 
Spacewalk server. The Taskomatic logs are in 
/var/log/rhn/rhn_taskomatic_daemon.log

Cheers,
Avi

--
Oracle 
Avi Miller | Product Management Director | +61 (3) 8616 3496
Oracle Linux and Virtualization
417 St Kilda Road, Melbourne, Victoria 3004 Australia


___
Spacewalk-list mailing list
Spacewalk-list@redhat.com
https://www.redhat.com/mailman/listinfo/spacewalk-list

[Spacewalk-list] ? Failures Cloning Channels with spacewalk-manage-channel-lifecycle

[Boyd, Robert]

I've been using the spacewalk-manage-channel-lifecycle tool for a while.   
Lately I'm having a problem where the back end of spacewalk is failing and 
causing the tool to blow up with this error:

spacewalk-manage-channel-lifecycle -c oraclelinux6-x86_64 --promote 
--workflow=oracle --clear-channel ; date
Thu Jan 28 17:33:58 EST 2016
INFO: Parent Source: oraclelinux6-x86_64, destination: 
ora-preprod-oraclelinux6-x86_64
INFO: Clearing all errata from ora-preprod-oraclelinux6-x86_64
Traceback (most recent call last):
  File "/usr/local/bin/spacewalk-manage-channel-lifecycle", line 784, in 

merge_channels(parent_source, parent_dest)
  File "/usr/local/bin/spacewalk-manage-channel-lifecycle", line 420, in 
merge_channels
clear_channel(dest_label)
  File "/usr/local/bin/spacewalk-manage-channel-lifecycle", line 500, in 
clear_channel
False)
  File "/usr/lib64/python2.6/xmlrpclib.py", line 1199, in __call__
return self.__send(self.__name, args)
  File "/usr/lib64/python2.6/xmlrpclib.py", line 1489, in __request
verbose=self.__verbose
  File "/usr/lib64/python2.6/xmlrpclib.py", line 1243, in request
Headers
xmlrpclib.ProtocolError: 

I have tried poking at log files, but I haven't found where there seems to be 
an indication of a problem.   Can someone point me to a prioritized list of 
logs I should be looking at to find the nature of what's dying on the back end? 
 I suspect there is a java resource failure or some other error and would like 
to track it down and fix it rather than resort to manual/scripted channel by 
channel cloning( which might fail in the same way).


Thanks,


Robert Boyd
Sr. Systems Engineer
PeopleFluent
p. 919-645-2972 | c. 919-306-4681
e. robert.b...@peoplefluent.com

[http://mktg.peoplefluent.com/rs/peopleclick/images/140410_PF4colorLOGOx150.png]
Click here to experience the power of the new 
PeopleFluent Mirror Suite (tm)
Visit: www.peoplefluent.com | Read: PeopleFluent 
Blog | Follow: 
@PeopleFluent


___
Spacewalk-list mailing list
Spacewalk-list@redhat.com
https://www.redhat.com/mailman/listinfo/spacewalk-list