Can you swap in another httpclient to test? I assume swapping jetty
server would be too much, given something works. :-)
>From faster result on smaller batch size: are you monitoring memory use?
I'd try even smaller, looking at the perf profile for clues.
Bill
--
Phobrain.com
On 2023-05-31 16:30, Shawn Heisey wrote:
> On 5/31/23 02:50, Bill Ross wrote:
>
>> Minimal relevant experience here, but lots of debugging, and since you're
>> stuck:
>>
>> Is this failing only on the last request of a series of any length?
>
> It's failing in the middle of a run. My program grabs results from one Solr
> cluster and indexes them into another Solr cluster. 30.2 million documents
> total, it's failing after about 20-25 million docs.
>
> I broke up the migration into six smaller chunks that are all run separately
> (the biggest of which is 9.3 million docs) and now it completes with no
> problem.
>
> I did not see any usable error messages in the Solr server logs.
>
> The documents are indexed in batches of 1000 docs, so it's just slightly
> north of 30000 total requests for the full run.
>
> The way the program works is that I have a query thread querying the source
> cluster and pushing the documents one by one onto a queue. Then the indexing
> threads pull documents off the queue, organize them into batches, and send
> the batches to the target cluster.
>
> It's being done with SolrJ -- the Java Solr client. The specfic clients I am
> using utilize the Jetty httpclient, version 10.0.13.
>
> The Solr servers are running a slightly older release that fires up Jetty
> 9.4.48 to start the Solr webapp.
>
> There is a Solr upgrade available that upgrades the Jetty server to 10.0.13,
> but the client is resisting an upgrade.
>
> At least I found a way to get the work done by breaking it into six smaller
> runs instead of doing the whole job at once. Big bonus ... it also runs
> about three times as fast for some reason. I don't know why, but I'll take
> it!
>
> I'm trying to find out whether the problem is in Jetty (client or server),
> SolrJ, the Solr servers, or my usage. The particulars make me think its
> something in Jetty, but that could be completely wrong.
>
> Thanks,
> Shawn
> _______________________________________________
> jetty-users mailing list
> jetty-users@eclipse.org
> To unsubscribe from this list, visit
> https://www.eclipse.org/mailman/listinfo/jetty-users
_______________________________________________
jetty-users mailing list
jetty-users@eclipse.org
To unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/jetty-users
