Thanks for looking harder into this! The credential encoding in httpcomponents/httpclient has been problem free as far as I have seen, so if you determine that that's the issue I am sure it will be news to a lot of people. But by using the wire logging you should be able to see the headers, including the encoded credentials, and compare/contrast what's working and what's not pretty easily.
Karl On Thu, Dec 27, 2018 at 5:42 AM Erlend Garåsen <e.f.gara...@usit.uio.no> wrote: > > It wasn't necessary to deal with tools like tcpdump etc. Adding the > following to the logging.xml did the trick: > <Logger name="org.apache.http.wire" level="debug" additivity="false"> > <AppenderRef ref="MyFile" /> > </Logger> > > So now I know what's going on. Bad credentials: > > DEBUG 2018-12-27T11:18:41,593 (Thread-1508) - http-outgoing-2 << > "HTTP/1.1 401 Unauthorized[\r][\n]" > > Strange, because connection is working according to the Solr Output > Connector. I'll double-check whether the Solr server has another > password for index writing (path "/solr/uio/update/extract"). Or maybe > we have an encoding issue with the password since it's long and contains > special characters. > > --8<-- > > DEBUG 2018-12-27T11:18:41,591 (Thread-1508) - http-outgoing-2 >> > "</div><!-- container --> [\n]" > DEBUG 2018-12-27T11:18:41,591 (Thread-1508) - http-outgoing-2 >> > "</body> [\n]" > DEBUG 2018-12-27T11:18:41,591 (Thread-1508) - http-outgoing-2 >> > "</html>[\n]" > DEBUG 2018-12-27T11:18:41,591 (Thread-1508) - http-outgoing-2 >> "[\n]" > DEBUG 2018-12-27T11:18:41,591 (Thread-1508) - http-outgoing-2 >> "[\r][\n]" > DEBUG 2018-12-27T11:18:41,592 (Thread-1508) - http-outgoing-2 >> > "2f[\r][\n]" > DEBUG 2018-12-27T11:18:41,592 (Thread-1508) - http-outgoing-2 >> "[\r][\n]" > DEBUG 2018-12-27T11:18:41,592 (Thread-1508) - http-outgoing-2 >> > "******************[\r][\n]" > DEBUG 2018-12-27T11:18:41,592 (Thread-1508) - http-outgoing-2 >> "[\r][\n]" > DEBUG 2018-12-27T11:18:41,592 (Thread-1508) - http-outgoing-2 >> > "0[\r][\n]" > DEBUG 2018-12-27T11:18:41,592 (Thread-1508) - http-outgoing-2 >> "[\r][\n]" > DEBUG 2018-12-27T11:18:41,593 (Thread-1508) - http-outgoing-2 << > "HTTP/1.1 401 Unauthorized[\r][\n]" > DEBUG 2018-12-27T11:18:41,593 (Thread-1508) - http-outgoing-2 << "Date: > Thu, 27 Dec 2018 10:18:41 GMT[\r][\n]" > DEBUG 2018-12-27T11:18:41,593 (Thread-1508) - http-outgoing-2 << > "Server: Apache/2.4.6 (Red Hat Enterprise Linux) > OpenSSL/1.0.2k-fips[\r][\n]" > DEBUG 2018-12-27T11:18:41,593 (Thread-1508) - http-outgoing-2 << > "WWW-Authenticate: Basic realm="Solr"[\r][\n]" > DEBUG 2018-12-27T11:18:41,593 (Thread-1508) - http-outgoing-2 << > "Content-Length: 381[\r][\n]" > DEBUG 2018-12-27T11:18:41,593 (Thread-1508) - http-outgoing-2 << > "Keep-Alive: timeout=10, max=100[\r][\n]" > DEBUG 2018-12-27T11:18:41,593 (Thread-1508) - http-outgoing-2 << > "Connection: Keep-Alive[\r][\n]" > DEBUG 2018-12-27T11:18:41,593 (Thread-1508) - http-outgoing-2 << > "Content-Type: text/html; charset=iso-8859-1[\r][\n]" > DEBUG 2018-12-27T11:18:41,593 (Thread-1508) - http-outgoing-2 << "[\r][\n]" > DEBUG 2018-12-27T11:18:41,594 (Thread-1508) - http-outgoing-2 << > "<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">[\n]" > DEBUG 2018-12-27T11:18:41,594 (Thread-1508) - http-outgoing-2 << > "<html><head>[\n]" > DEBUG 2018-12-27T11:18:41,594 (Thread-1508) - http-outgoing-2 << > "<title>401 Unauthorized</title>[\n]" > DEBUG 2018-12-27T11:18:41,594 (Thread-1508) - http-outgoing-2 << > "</head><body>[\n]" > DEBUG 2018-12-27T11:18:41,594 (Thread-1508) - http-outgoing-2 << > "<h1>Unauthorized</h1>[\n]" > DEBUG 2018-12-27T11:18:41,594 (Thread-1508) - http-outgoing-2 << > "<p>This server could not verify that you[\n]" > DEBUG 2018-12-27T11:18:41,594 (Thread-1508) - http-outgoing-2 << "are > authorized to access the document[\n]" > DEBUG 2018-12-27T11:18:41,594 (Thread-1508) - http-outgoing-2 << > "requested. Either you supplied the wrong[\n]" > DEBUG 2018-12-27T11:18:41,594 (Thread-1508) - http-outgoing-2 << > "credentials (e.g., bad password), or your[\n]" > DEBUG 2018-12-27T11:18:41,594 (Thread-1508) - http-outgoing-2 << > "browser doesn't understand how to supply[\n]" > DEBUG 2018-12-27T11:18:41,594 (Thread-1508) - http-outgoing-2 << "the > credentials required.</p>[\n]" > DEBUG 2018-12-27T11:18:41,594 (Thread-1508) - http-outgoing-2 << > "</body></html>[\n]" > WARN 2018-12-27T11:18:41,599 (Worker thread '48') - IO exception during > indexing https://www.journals.uio.no/index.php/Dialogia: null > org.apache.http.client.ClientProtocolException > > Erlend > > On 21/12/2018 16:39, Karl Wright wrote: > > I'll have a look as time permits, but it won't be for a couple of weeks. > > Usually it's best to debug with http wire debugging, not packet > > captures. I'm not an HTTP expert so it may be better to bring this up > > to the HttpComponents/HttpClient list, not ManifoldCF. > > > > Karl > > > > > > On Fri, Dec 21, 2018 at 8:54 AM Erlend Garåsen <e.f.gara...@usit.uio.no > > <mailto:e.f.gara...@usit.uio.no>> wrote: > > > > > > I tried to configure preemptive authentication by modifying the > > HttpPoster class, but I still get the same errors. Then I ran the > > following command in order to analyze the traffic: > > tcpdump -w /tmp/chatter.dmp -s 0 -i ens192 -X host > > solr-test01.uio.no <http://solr-test01.uio.no> > > > > I'm not an expert in reading such outputs, but I can send you the > dump > > file, Karl. I don't want to let the file be available for others, > even > > though it's from our test environment. > > > > Erlend > > > > On 13/12/2018 15:09, Karl Wright wrote: > > > Ok, thanks, I misunderstood where the SSL error was coming from. > The > > > Solr connection is what is complaining. Do you see "Connection > > working" > > > for your output connection? Please forgive me if you already > answered > > > this; I didn't note it in your response. If you see that, then the > > > connector was able to talk to your Solr "ping" handler, which > > would mean > > > that the SSL configuration is right but there's something else > > about the > > > connection that we would have to figure out and deal with. > > > > > > Karl > > > > > > > > > On Thu, Dec 13, 2018 at 8:58 AM Erlend Garåsen > > <e.f.gara...@usit.uio.no <mailto:e.f.gara...@usit.uio.no> > > > <mailto:e.f.gara...@usit.uio.no <mailto:e.f.gara...@usit.uio.no>>> > > wrote: > > > > > > On 13/12/2018 14:26, Karl Wright wrote: > > > > This is SSL. Did you add the server's cert to the web > > connector's > > > > keystore? Or, if not, add a "trust all" rule? > > > > > > Thanks for the reply, Karl. > > > > > > Yes, the root certificate was added both in the repository > > connection > > > and for the Solr connector. > > > > > > I checked the "trust all" option and restarted the job. Same > > problem. > > > > > > BTW, I get a lot of 200s, so MCF is able to fetch the URLs. > > > > > > I have attached a screenshot of what I can see in the simple > > history > > > report. > > > > > > Erlend > > > > > > >
