Right now, I have no more ideas :-(
Maybe you should start tcpdump for further investigation.
Regards
Robert
Am 25.05.2016 12:01 schrieb Lachlan Musicman <data...@gmail.com>:
Also, I've triedThat is the same command on both machines (the one that works, the one that doesn't).How would be make it https?The command we are using (I'm not at work and don't have access to the servers atm, so this may not be 100% correct) isrhnpush --server localhost -u admin --channel slurm_16_05_rc2 *rpm --nosig
--server=https://localhost
--server=http://localhost
--server=http://<fqdn>
--server=https://<fqdn>--server=http://<ip>
--server=https://<ip>Although admittedly I've not personally tried those combinations since last Friday.CheersL.------
The most dangerous phrase in the language is, "We've always done it this way."
- Grace HopperOn 25 May 2016 at 19:50, Robert Paschedag <robert.paschedag@web.de> wrote:Are you trying to push to http or https?
If you are using http, maybe the problem is a redirect to https. Because you get the 302 codes.
Regards
Robert
Am 25.05.2016 10:42 schrieb Lachlan Musicman <datakid@gmail.com>:Yes and yes and yes. I've run the command at least 10 times a day on each server. They both have the same user and pw. I've passed the pw on the CLI via -p and I've typed it in. I've confirmed it's correct via the strace.L.------
The most dangerous phrase in the language is, "We've always done it this way."
- Grace HopperOn 25 May 2016 at 18:37, Robert Paschedag <robert.paschedag@web.de> wrote:So the command line looks the same on both systems... both systems are configured the same way?
Stupid question, but credentials are OK?
Robert
Am 25.05.2016 08:32 schrieb Lachlan Musicman <datakid@gmail.com>:
>
> Thanks for answering Robert, I appreciate it. I think that those log msgs were me casting about wildly for *anything*. I don't think they point to the problem.
>
> The one thing that I see that stands out more than anything else, is the lines
>
> <ipaddress> - - [20/May/2016:13:58:00 +1000] "GET /rhn/errors/500.jsp HTTP/1.1" 302 -
> <ipaddress> - - [20/May/2016:13:58:00 +1000] "POST /rhn/errors/500.jsp HTTP/1.1" 302 -
> <ipaddress> - - [20/May/2016:13:58:00 +1000] "POST /rhn/Login.do?url_bounce=/rhn/errors/500.jsp&request_method=POST HTTP/1.1" 200 5967
>
> These are seen on the server that isn't accepting rhnpush, in /var/log/tomcat/localhost_access_<date>.txt
>
> I do not see these on the working server, at all.
>
> Today I did some straces - on each server - and while it's verbose and hard to pick apart, it looks like the error is similar to the above. ie, that the two straces appear very similar until just after the password is entered, and then the server that wont import files looks like this. Note that the post returns HTTP code 302:
>
> sendto(3, "POST /APP HTTP/1.1\r\nHost: localh"..., 581, 0, NULL, 0) = 581
> poll([{fd=3<socket:[254525]>, events=POLLIN}], 1, 120000) = 1 ([{fd=3, revents=POLLIN}])
> recvfrom(3, "H", 1, 0, NULL, NULL) = 1
> poll([{fd=3<socket:[254525]>, events=POLLIN}], 1, 120000) = 1 ([{fd=3, revents=POLLIN}])
> recvfrom(3, "T", 1, 0, NULL, NULL) = 1
> poll([{fd=3<socket:[254525]>, events=POLLIN}], 1, 120000) = 1 ([{fd=3, revents=POLLIN}])
> recvfrom(3, "T", 1, 0, NULL, NULL) = 1
> poll([{fd=3<socket:[254525]>, events=POLLIN}], 1, 120000) = 1 ([{fd=3, revents=POLLIN}])
> recvfrom(3, "P", 1, 0, NULL, NULL) = 1
> poll([{fd=3<socket:[254525]>, events=POLLIN}], 1, 120000) = 1 ([{fd=3, revents=POLLIN}])
> recvfrom(3, "/", 1, 0, NULL, NULL) = 1
> poll([{fd=3<socket:[254525]>, events=POLLIN}], 1, 120000) = 1 ([{fd=3, revents=POLLIN}])
> recvfrom(3, "1", 1, 0, NULL, NULL) = 1
> poll([{fd=3<socket:[254525]>, events=POLLIN}], 1, 120000) = 1 ([{fd=3, revents=POLLIN}])
> recvfrom(3, ".", 1, 0, NULL, NULL) = 1
> poll([{fd=3<socket:[254525]>, events=POLLIN}], 1, 120000) = 1 ([{fd=3, revents=POLLIN}])
> recvfrom(3, "1", 1, 0, NULL, NULL) = 1
> poll([{fd=3<socket:[254525]>, events=POLLIN}], 1, 120000) = 1 ([{fd=3, revents=POLLIN}])
> recvfrom(3, " ", 1, 0, NULL, NULL) = 1
> poll([{fd=3<socket:[254525]>, events=POLLIN}], 1, 120000) = 1 ([{fd=3, revents=POLLIN}])
> recvfrom(3, "3", 1, 0, NULL, NULL) = 1
> poll([{fd=3<socket:[254525]>, events=POLLIN}], 1, 120000) = 1 ([{fd=3, revents=POLLIN}])
> recvfrom(3, "0", 1, 0, NULL, NULL) = 1
> poll([{fd=3<socket:[254525]>, events=POLLIN}], 1, 120000) = 1 ([{fd=3, revents=POLLIN}])
> recvfrom(3, "2", 1, 0, NULL, NULL) = 1
>
>
>
> On the working server it looks like this. Note that it returns HTTP code 200:
>
>
> sendto(3, "POST /APP HTTP/1.1\r\nHost: localh"..., 586, 0, NULL, 0) = 586
> poll([{fd=3<socket:[964921]>, events=POLLIN}], 1, 120000) = 1 ([{fd=3, revents=POLLIN}])
> recvfrom(3, "H", 1, 0, NULL, NULL) = 1
> poll([{fd=3<socket:[964921]>, events=POLLIN}], 1, 120000) = 1 ([{fd=3, revents=POLLIN}])
> recvfrom(3, "T", 1, 0, NULL, NULL) = 1
> poll([{fd=3<socket:[964921]>, events=POLLIN}], 1, 120000) = 1 ([{fd=3, revents=POLLIN}])
> recvfrom(3, "T", 1, 0, NULL, NULL) = 1
> poll([{fd=3<socket:[964921]>, events=POLLIN}], 1, 120000) = 1 ([{fd=3, revents=POLLIN}])
> recvfrom(3, "P", 1, 0, NULL, NULL) = 1
> poll([{fd=3<socket:[964921]>, events=POLLIN}], 1, 120000) = 1 ([{fd=3, revents=POLLIN}])
> recvfrom(3, "/", 1, 0, NULL, NULL) = 1
> poll([{fd=3<socket:[964921]>, events=POLLIN}], 1, 120000) = 1 ([{fd=3, revents=POLLIN}])
> recvfrom(3, "1", 1, 0, NULL, NULL) = 1
> poll([{fd=3<socket:[964921]>, events=POLLIN}], 1, 120000) = 1 ([{fd=3, revents=POLLIN}])
> recvfrom(3, ".", 1, 0, NULL, NULL) = 1
> poll([{fd=3<socket:[964921]>, events=POLLIN}], 1, 120000) = 1 ([{fd=3, revents=POLLIN}])
> recvfrom(3, "1", 1, 0, NULL, NULL) = 1
> poll([{fd=3<socket:[964921]>, events=POLLIN}], 1, 120000) = 1 ([{fd=3, revents=POLLIN}])
> recvfrom(3, " ", 1, 0, NULL, NULL) = 1
> poll([{fd=3<socket:[964921]>, events=POLLIN}], 1, 120000) = 1 ([{fd=3, revents=POLLIN}])
> recvfrom(3, "2", 1, 0, NULL, NULL) = 1
> poll([{fd=3<socket:[964921]>, events=POLLIN}], 1, 120000) = 1 ([{fd=3, revents=POLLIN}])
> recvfrom(3, "0", 1, 0, NULL, NULL) = 1
> poll([{fd=3<socket:[964921]>, events=POLLIN}], 1, 120000) = 1 ([{fd=3, revents=POLLIN}])
> recvfrom(3, "0", 1, 0, NULL, NULL) = 1
>
>
>
> Why is that? The configurations are as identical as possible. Idiot checked for me by two others.
>
>
>
>
> ------
> The most dangerous phrase in the language is, "We've always done it this way."
>
> - Grace Hopper
>
> On 24 May 2016 at 00:25, <Paschedag.Netlution@swr.de> wrote:
>>
>> The library looks the same to me, but the URL is different.
>>
>> Regards,
>> Robert
>>
>> Mit freundlichen Grüßen
>>
>> Robert Paschedag
>> Netlution GmbH
>> Landteilstr. 33
>> 68163 Mannheim
>>
>> im Auftrag des
>> SWR
>> Südwestrundfunk
>> Informations- und Kommunikationssysteme
>> Neckarstraße 230
>> 70190 Stuttgart
>>
>> Telefon +49 (0)711 /929-12654 oder
>> Telefon +49 (0)711 /929-13714
>> paschedag.netlution@swr.de
>>
>> swr.de
>>
>>
>>
>>
>>
>> Von: Lachlan Musicman <datakid@gmail.com>
>> An: spacewalk-list@redhat.com
>> Datum: 23.05.2016 04:53
>> Betreff: Re: [Spacewalk-list] rhnpush fails with uninformative error msg
>> Gesendet von: spacewalk-list-bounces@redhat.com
>> ________________________________
>>
>>
>>
>> Ignore that, I've got dispatcher working again. That wasn't the problem apparently.
>>
>> On the working server, when I successfully import via rhnpush, I see this in /var/log/httpd/access_log:
>>
>> 112.12.4.12 - - [23/May/2016:11:39:33 +1000] "POST /XMLRPC HTTP/1.1" 200 11932 "-" "rhn.rpclib.py/2.5.77-1.el7"
>>
>> When I do it on the server that doesn't work, I get this:
>>
>> 10.124.28.172 - - [23/May/2016:12:31:58 +1000] "POST /APP HTTP/1.1" 302 - "-" "rhn.rpclib.py/2.5.77-1.el7"
>>
>> So I'm getting a different HTTP codes in my apache server for this rhn.rpclib.py....
>>
>> L.
>>
>> ------
>> The most dangerous phrase in the language is, "We've always done it this way."
>>
>> - Grace Hopper
>>
>> On 23 May 2016 at 10:53, Lachlan Musicman <datakid@gmail.com> wrote:
>> Hmm. It looks like my osa-dispatcher isn't starting. Where are the osa-dispatcher config files?
>>
>> cheers
>> L.
>>
>> ------
>> The most dangerous phrase in the language is, "We've always done it this way."
>>
>> - Grace Hopper
>>
>> On 23 May 2016 at 10:29, Lachlan Musicman <datakid@gmail.com> wrote:
>> No one has seen this error before? No one can even give me pointers on where to start looking for a solution?
>>
>> It almost certainly seems to be a tomcat issue - that either the rhnpush isn't getting through a proxy (??! wtf) or it is an is failing on the other side of the proxy because some other reason?
>>
>> I've check the tomcat conf on both servers - they are the same (as necessary). The rhn is set up as identically as possible, the server OS is the same revision...I don't understand. The only discernable difference is that one is a VMWare VM and one is a stand alone installation.
>>
>> Cheers
>>
>> L.
>>
>> ------
>> The most dangerous phrase in the language is, "We've always done it this way."
>>
>> - Grace Hopper
>>
>> On 20 May 2016 at 16:32, Lachlan Musicman <datakid@gmail.com> wrote:
>> In centos 7.2, using
>>
>> [root@vmpr-res-utils 16.05-rc2]# yum whatprovides rhnpush
>>
>> rhnpush-5.5.89-1.el7.noarch : Package uploader for the Spacewalk or Red Hat Satellite Server
>> Repo : spacewalk
>>
>> rhnpush-5.5.89-1.el7.noarch : Package uploader for the Spacewalk or Red Hat Satellite Server
>> Repo : @spacewalk
>>
>>
>> [root@vmpr-res-utils 16.05-rc2]# rhnpush --server localhost -u admin --channel slurm_16_05_rc2 *rpm --nosig
>> Password:
>>
>> ERROR: unhandled exception occurred: (mismatched tag: line 164, column 2).
>>
>> No logs are created at all (I've deleted them all and checked).
>>
>> There is a log in /var/log/tomcat/localhost_access_{date}.txt
>>
>> ::1 - - [20/May/2016:13:29:49 +1000] "GET /rhn/errors/500.jsp HTTP/1.1" 302 -
>> ::1 - - [20/May/2016:13:29:49 +1000] "POST /rhn/errors/500.jsp HTTP/1.1" 302 -
>> ::1 - - [20/May/2016:13:29:50 +1000] "POST /rhn/Login.do?url_bounce=/rhn/errors/500.jsp&request_method=POST HTTP/1.1" 200 5967
>>
>>
>> Note here that it's defaulting to IPv6 address.
>>
>> If I use the fqdn/hostname (same thing) I get it using the ipv4, same problem:
>>
>> 10.126.18.72 - - [20/May/2016:13:58:00 +1000] "GET /rhn/errors/500.jsp HTTP/1.1" 302 -
>> 10.126.18.72 - - [20/May/2016:13:58:00 +1000] "POST /rhn/errors/500.jsp HTTP/1.1" 302 -
>> 10.126.18.72 - - [20/May/2016:13:58:00 +1000] "POST /rhn/Login.do?url_bounce=/rhn/errors/500.jsp&request_method=POST HTTP/1.1" 200 5967
>>
>>
>> Note that the username and password are correct - I've tested 100 times, and I'm also logged into the website, *using the same credentials*.
>>
>> Further frustration is caused by the fact that I have it working on another server (and another domain/ip range), which was set up identically, runs the same software, and it works.
>>
>> ....
>>
>> OK, I tried to add the server to itself as a system with
>>
>> rhnreg_ks --serverUrl=https://server/XMLRPC --sslCACert=/usr/share/rhn/RHN-ORG-TRUSTED-SSL-CERT --activationkey=1-x
>>
>> and that also failed.
>>
>> Then I went into /var/log/up2date and noticed that the error was the same:
>> Traceback (most recent call last):
>> File "/usr/sbin/rhnreg_ks", line 216, in <module>
>> cli.run()
>> File "/usr/share/rhn/up2date_client/rhncli.py", line 96, in run
>> sys.exit(self.main() or 0)
>> File "/usr/sbin/rhnreg_ks", line 91, in main
>> rhnreg.getCaps()
>> File "/usr/share/rhn/up2date_client/rhnreg.py", line 248, in getCaps
>> s.capabilities.validate()
>> File "/usr/share/rhn/up2date_client/rhnserver.py", line 178, in __get_capabilities
>> self.registration.welcome_message()
>> File "/usr/share/rhn/up2date_client/rhnserver.py", line 63, in __call__
>> return rpcServer.doCall(method, *args, **kwargs)
>> File "/usr/share/rhn/up2date_client/rpcServer.py", line 203, in doCall
>> ret = method(*args, **kwargs)
>> File "/usr/lib64/python2.7/xmlrpclib.py", line 1233, in __call__
>> return self.__send(self.__name, args)
>> File "/usr/share/rhn/up2date_client/rpcServer.py", line 37, in _request1
>> ret = self._request(methodname, params)
>> File "/usr/lib/python2.7/site-packages/rhn/rpclib.py", line 384, in _request
>> self._handler, request, verbose=self._verbose)
>> File "/usr/lib/python2.7/site-packages/rhn/transports.py", line 187, in request
>> return self._process_response(fd, connection)
>> File "/usr/lib/python2.7/site-packages/rhn/transports.py", line 215, in _process_response
>> return self.parse_response(fd)
>> File "/usr/lib/python2.7/site-packages/rhn/transports.py", line 235, in parse_response
>> p.feed(response)
>> File "/usr/lib64/python2.7/xmlrpclib.py", line 558, in feed
>> self._parser.Parse(data, 0)
>> <class 'xml.parsers.expat.ExpatError'>: mismatched tag: line 164, column 2
>>
>>
>> which lead me to this:
>>
>> https://bugzilla.redhat.com/show_bug.cgi?id=1277623
>>
>> but both of my systems have
>>
>> python-debian-0.1.27-3.el7.noarch and python-six-1.9.0-2.el7.noarch
>>
>> I'm reticent to downgrade since I don't know what else that will affect, and it's *working* on one of the servers without the downgrade?
>>
>> I saw this reddit thread:
>>
>> https://www.reddit.com/r/spacewalk/comments/3vnquh/centos_71_spacewalk_24_no_go/
>>
>> but I wont try the solution(s) in the last comment bc both files in both systems are currently the same, so they are obv not the issue.
>>
>>
>> I tried the patch shown here:
>>
>> https://www.redhat.com/archives/spacewalk-list/2015-November/msg00108.html
>>
>> But it just output a whole bunch of blather - looks like something is being chomped though - either time out or buffer length or...?
>>
>> asdIncoming headers:
>> content-length : 5966
>> set-cookie : JSESSIONID=69C01EA51CB966ABC58B3188A956B119; Path=/rhn/; Secure; HttpOnly; HttpOnly
>> server : Apache
>> x-ua-compatible : IE=edge,chrome=1
>> date : Fri, 20 May 2016 06:17:12 GMT
>> content-type : text/html;charset=UTF-8
>> ('body:', '\'<!DOCTYPE HTML>\\n<html>\\n <head>\\n <!-- enclosing head tags in layout_c.jsp -->\\n <meta http-equiv="X-UA-Compatible" content="IE=edge"/>\\n <!-- HTML5 shim and Respond.js IE8 support of HTML5 elements and media queries -->\\n <!--[if lt IE 9]>\\n <script type=\\\'text/_javascript_\\\' src="" <script type=\\\'text/_javascript_\\\' src="" <![endif]-->\\n \\n <meta http-equiv="content-type" content="text/html;charset=UTF-8"/>\\n <title>\\n Spacewalk\\n \\n \\n - Sign In\\n \\n \\n </title>\\n <link rel="shortcut icon" href="" />\\n\\n <meta name="viewport" content="width=device-width, initial-scale=1.0" />\\n\\n \\n \\n <link rel="stylesheet" href="" />\\n \\n <link rel="stylesheet" href="" />\\n <link rel="stylesheet" href="" />\\n\\n <link rel="stylesheet" href="" />\\n <link rel="stylesheet" href="" /> >> ('body:', '\'/_javascript_/select2/select2-bootstrap.css" />\\n\\n <link rel="stylesheet" href="" />\\n <link rel="stylesheet" href="" />\\n\\n <script src="" <script src="" <script src="" <script src="" <script src="" <script src="" <script src="" <script src="" <script src="" <script src="" <meta name="decorator" content="layout_c" />\\n <script src="" </head>\\n <body _onload_="">\\n <div class="spacewalk-top-wrap">\\n <nav class="navbar navbar-default navbar-pf" role="navigation">\\n \'')
>> ('body:', '\' \\n\\n\\n\\n<!-- header.jsp -->\\n\\n\\n\\n<div class="navbar-header">\\n <button type="button" class="navbar-toggle" data-toggle="collapse" data-target=".navbar-collapse">\\n <span class="sr-only">Toggle navigation</span>\\n <span class="icon-bar"></span>\\n <span class="icon-bar"></span>\\n <span class="icon-bar"></span>\\n </button>\\n <a class="navbar-brand" href="" href="http://www.spacewalkproject.org/">http://www.spacewalkproject.org/" title="Visit http://www.spacewalkproject.org/ for more information">\\n <img src="" alt="Spacewalk project" id="rhLogo" />\\n </a>\\n <a class="navbar-brand" href="" title="Spacewalk homepage">\\n <img src="" alt="Spacewalk" id="rhnLogo" accesskey="2"/>\\n </a>\\n \\n</div>\\n\\n<div class="navbar-collapse collapse">\\n \\n\\n \\n \\n <ul class="nav navbar-nav navbar-primary"><li class="active"><a href="" In</a></li>\\n<li><a href="" \\n</div>\\n<!-- end header.jsp -->\\n\\n </nav>\\n <div class="spacewalk-main-column-layout">\\n \'')
>> ('body:', '\' <aside id="spacewalk-aside" class="navbar-collapse collapse">\\n \\n\\n<nav id="sidenav">\\n \\n \\n \\n \\n</nav>\\n\\n \\n\\n\\n\\n\\n\\n \\n\\n\\n<!-- TODO make this more generic -->\\n\\n\\n\\n </aside>\\n <section id="spacewalk-content">\\n <!-- Alerts and messages -->\\n \\n \\n <div class="text-center">\\n \\n \\n \\n \\n <h1>Welcome to Spacewalk</h1>\\n <p><strong>Spacewalk</strong> is an open source (GPLv2) Linux systems management solution. Below are some links that should prove useful if you have any questions regarding your Spacewalk server. </p>\\n \\n \\n\\n <form name="loginForm" id="loginForm" method="post" action="" class="form-horizontal col-md-6 col-md-offset-3 text-left">\\n <input type="hidden" name="csrf_token" value="7530790251516499651">\\n \\n\\n \\n \\n \\n \\n \\n \\n \\n\\n\\n<div class="form-group">\\n <label for="" class="col-xs-5 control-label">Spacewal\'')
>> ('body:', '\'k Login: </label>\\n <div class="col-xs-7">\\n <input type="text" id="username-field" class="form-control" tabindex="1" name="username" maxlength="63" placeholder="Login" />\\n </div>\\n</div>\\n\\n<div class="form-group">\\n <label for="" class="col-xs-5 control-label">Password: </label>\\n <div class="col-xs-7">\\n <input type="password" id="password-field" class="form-control" tabindex="2" name="password" maxlength="32" placeholder="Password" />\\n </div>\\n</div>\\n\\n<div class="text-right">\\n <input type="hidden" name="login_cb" value="login">\\n <input type="hidden" name="url_bounce" value="/rhn/errors/500.jsp">\\n <input type="hidden" name="request_method" value="GET">\\n <input type="submit" tabindex="3" value="Sign In" id="login" class="btn btn-primary">\\n</div>\\n\\n </form>\\n <div class="col-md-6 col-md-offset-3 text-left">\\n \\n \\n \\n \\n <p>For details on Spacewalk, please visit our web-site at <a href="" href="http://www.spacewalkproject.org/">http://www.spacewalkproject.org/" target="_blank"> http://www.spacewalkprojec\'')
>>
>>
>>
>>
>> Would appreciate any ideas people might have?
>>
>>
>>
>> cheers
>> L.
>>
>> ------
>> The most dangerous phrase in the language is, "We've always done it this way."
>>
>> - Grace Hopper
>>
>>
>> _______________________________________________
>> Spacewalk-list mailing list
>> Spacewalk-list@redhat.com
>> https://www.redhat.com/mailman/listinfo/spacewalk-list
>>
>>
>> _______________________________________________
>> Spacewalk-list mailing list
>> Spacewalk-list@redhat.com
>> https://www.redhat.com/mailman/listinfo/spacewalk-list
>
>
_______________________________________________ Spacewalk-list mailing list Spacewalk-list@redhat.com https://www.redhat.com/mailman/listinfo/spacewalk-list
