Re: [jetty-users] SolrJ/Solr: HTTP protocol violation: Authentication challenge without WWW-Authenticate header

2023-06-04 Thread Shawn Heisey
On 6/4/2023 8:37 AM, Greg Wilkins wrote: Shawn, The other thing to do is to bisect the changes from a jetty-9.4.x that you think works to jetty-9.4.48 that you think does not work. There are not that many changes in jetty 9.4, so it should not be too much work... but it will be some effort

Re: [jetty-users] SolrJ/Solr: HTTP protocol violation: Authentication challenge without WWW-Authenticate header

2023-06-04 Thread Greg Wilkins
Shawn, The other thing to do is to bisect the changes from a jetty-9.4.x that you think works to jetty-9.4.48 that you think does not work. There are not that many changes in jetty 9.4, so it should not be too much work... but it will be some effort which is why we don't offer open source

Re: [jetty-users] SolrJ/Solr: HTTP protocol violation: Authentication challenge without WWW-Authenticate header

2023-06-02 Thread Joakim Erdfelt
On Thu, Jun 1, 2023 at 5:43 PM Shawn Heisey wrote: > On 6/1/23 15:45, Jesse McConnell wrote: > > Your best bet would be to do at Greg said and capture the actual request > > failing on the wire using wireshark or tcpdump, or get a > > HttpChannel.Listener[1] set up in the server so you can see

Re: [jetty-users] SolrJ/Solr: HTTP protocol violation: Authentication challenge without WWW-Authenticate header

2023-06-01 Thread Shawn Heisey
On 6/1/23 15:45, Jesse McConnell wrote: Your best bet would be to do at Greg said and capture the actual request failing on the wire using wireshark or tcpdump, or get a HttpChannel.Listener[1] set up in the server so you can see what is happening there. I do not have access to the private

Re: [jetty-users] SolrJ/Solr: HTTP protocol violation: Authentication challenge without WWW-Authenticate header

2023-06-01 Thread Jesse McConnell
Your best bet would be to do at Greg said and capture the actual request failing on the wire using wireshark or tcpdump, or get a HttpChannel.Listener[1] set up in the server so you can see what is happening there. Ultimately Jetty-9 is community end of life (a year ago today! [2]), and we are

Re: [jetty-users] SolrJ/Solr: HTTP protocol violation: Authentication challenge without WWW-Authenticate header

2023-06-01 Thread Shawn Heisey
On 6/1/23 13:25, Jesse McConnell wrote: Friction points between Jetty and Solr on the server side are possible or, more likely, an issue with the client.  With Jetty, it is never a good idea to double configuration values and see what happens, especially regarding HTTP configurations. You can

Re: [jetty-users] SolrJ/Solr: HTTP protocol violation: Authentication challenge without WWW-Authenticate header

2023-06-01 Thread Jesse McConnell
FWIW, it is unlikely to be an issue with the internals of the Jetty server; that may seem like a large number, but it is minor compared to other jetty usages in the wild. Friction points between Jetty and Solr on the server side are possible or, more likely, an issue with the client. With Jetty,

Re: [jetty-users] SolrJ/Solr: HTTP protocol violation: Authentication challenge without WWW-Authenticate header

2023-06-01 Thread Shawn Heisey
On 6/1/23 01:35, Bill Ross wrote: If you can turn off the sorting for a test, that would prove the assumption. Sorting is absolutely required for this effort. I can't turn it off. I am using Solr's cursorMark functionality. The "standard" Solr pagination functionality simply will not work

Re: [jetty-users] SolrJ/Solr: HTTP protocol violation: Authentication challenge without WWW-Authenticate header

2023-06-01 Thread Bill Ross
I usually look for low-effort changes to test assumptions and tug on the tangle from different directions when I'm stuck. In that spirit: > The queries gathering documents from the source are faster with a filter in > place, so they feed data to the queue faster. I think this is probably >

Re: [jetty-users] SolrJ/Solr: HTTP protocol violation: Authentication challenge without WWW-Authenticate header

2023-05-31 Thread Shawn Heisey
On 5/31/23 17:48, Bill Ross wrote: Can you swap in another httpclient to test? I assume swapping jetty server would be too much, given something works. :-) I can't do anything about the Jetty server without upgrading Solr. I really want to get them upgraded, but it's not up to me. I tried

Re: [jetty-users] SolrJ/Solr: HTTP protocol violation: Authentication challenge without WWW-Authenticate header

2023-05-31 Thread Bill Ross
Can you swap in another httpclient to test? I assume swapping jetty server would be too much, given something works. :-) >From faster result on smaller batch size: are you monitoring memory use? I'd try even smaller, looking at the perf profile for clues. Bill -- Phobrain.com On 2023-05-31

Re: [jetty-users] SolrJ/Solr: HTTP protocol violation: Authentication challenge without WWW-Authenticate header

2023-05-31 Thread Shawn Heisey
On 5/31/23 02:50, Bill Ross wrote: Minimal relevant experience here, but lots of debugging, and since you're stuck: Is this failing only on the last request of a series of any length? It's failing in the middle of a run. My program grabs results from one Solr cluster and indexes them into

Re: [jetty-users] SolrJ/Solr: HTTP protocol violation: Authentication challenge without WWW-Authenticate header

2023-05-31 Thread Shawn Heisey
On 5/30/23 23:31, Greg Wilkins wrote: Can you capture the full headers of the bad 401? Here is a log with org.eclipse.jetty at DEBUG. I deleted everything just before the problem occurred: https://paste.elyograg.org/view/9dbc041f If this is not capturing the data you need, what do I need

Re: [jetty-users] SolrJ/Solr: HTTP protocol violation: Authentication challenge without WWW-Authenticate header

2023-05-31 Thread Bill Ross
Minimal relevant experience here, but lots of debugging, and since you're stuck: Is this failing only on the last request of a series of any length? Bill -- Phobrain.com On 2023-05-31 01:40, Shawn Heisey wrote: > On 5/30/23 21:51, Shawn Heisey wrote: > >> So I am using http/1.1 in

Re: [jetty-users] SolrJ/Solr: HTTP protocol violation: Authentication challenge without WWW-Authenticate header

2023-05-31 Thread Shawn Heisey
On 5/30/23 23:31, Greg Wilkins wrote: This would need for the request to have failed authentication (when many requests with same credentials had passed) and for the resulting 401 response to have been incorrectly generated by jetty.   It seems very unlikely that the one request that has a

Re: [jetty-users] SolrJ/Solr: HTTP protocol violation: Authentication challenge without WWW-Authenticate header

2023-05-31 Thread Shawn Heisey
On 5/30/23 21:51, Shawn Heisey wrote: So I am using http/1.1 in conjunction with that correct option.  Bonus ... the indexing is happening faster than before, because Solr doesn't ever need to forward requests to another server.  I should find out in the next hour or so whether it actually

Re: [jetty-users] SolrJ/Solr: HTTP protocol violation: Authentication challenge without WWW-Authenticate header

2023-05-30 Thread Greg Wilkins
If I understand correctly, the situation is that you are sending many requests, all with credentials encoded in the headers. Occasionally a 401 response is received that does not have the WWW-Authenticate header. This would need for the request to have failed authentication (when many requests

Re: [jetty-users] SolrJ/Solr: HTTP protocol violation: Authentication challenge without WWW-Authenticate header

2023-05-30 Thread Shawn Heisey
On 5/30/23 18:48, Shawn Heisey wrote: I am trying one more thing ... setting the client to use http/1.1. I remember we had some issues with Solr where http2 wasn't working right, and I think the final fix for that was upgrading Jetty. This did not help. It still randomly (though not very

Re: [jetty-users] SolrJ/Solr: HTTP protocol violation: Authentication challenge without WWW-Authenticate header

2023-05-30 Thread Shawn Heisey
On 5/30/23 17:34, Shawn Heisey wrote: I commented the line setting org.eclipse.jetty to WARN, and I'm running it again.  This is going to create a VERY big logfile, because jetty is now logging at DEBUG. I am trying one more thing ... setting the client to use http/1.1. I remember we had

Re: [jetty-users] SolrJ/Solr: HTTP protocol violation: Authentication challenge without WWW-Authenticate header

2023-05-30 Thread Shawn Heisey
On 5/30/23 13:52, Shawn Heisey wrote: Before throwing this exception, the program makes over 25000 successful requests, indexing over 25 million total docs into Solr. On the last run, it only failed to index ONE batch. 30287 successful POST requests and ONE failure. That failure was the

[jetty-users] SolrJ/Solr: HTTP protocol violation: Authentication challenge without WWW-Authenticate header

2023-05-30 Thread Shawn Heisey
I have a program written using SolrJ 9.2.1 talking to Solr 9.1.1 servers. SolrJ 9.2.1 includes Jetty client 10.0.13. Solr 9.1.1 includes Jetty (both server and client) 9.4.48. Here is stacktrace info: https://paste.elyograg.org/view/dd2685a8 The Solr client is configured with basic auth