First off, just reporting this: I wound up with approx 58% few documents having submitted via ConcurrentUpdateSolrServer. I went back and changed the code to use HttpSolrServer and had 100%
This was a long running test, approx 12 hours, with gigabytes of data, so conveniently shared / reproducible, but I at least wanted to email around, in part to get it "on the record", and second to see if anybody else has seen this? I didn't see anything in JIRA. I realize that Concurrent update is asynchronous and I'm giving up the ability to monitor things, but since it works using the old server, there's nothing glaringly wrong at least. Here's a few more details: * Approx 2 M docs, submitted 1,000 at a time. * Solr 4.0.0 on Windows Server 2008 * Solr server JVM configured with 4 Gigs of RAM * Submitting client JVM (SolrJ) configured with 10 Gigs of RAM * Did didn't see any OOM (Out Of Memory) errors on the asynchronous / ConcurrentUpdateSolrServer run. However, I didn't capture the entire log. Usually with OOM it's just before the run crashes, and the end of the log on the screen looked fine. * I also didn't think there was OOM issues on the Solr server side, for the same reason * When submitting the same data synchronously (via HttpSolrServer) it didn't have any problems Questions: The async client certainly finished faster, and since the underlying Solr server presumably didn't do the real work any faster, presumably a backlog built up somewhere. Agreed? I'm guessing this backlog had something to do with the failure. Or are there other areas to think about? Which process would get backlogged, the SolrJ client or the Solr server? I'd guess the server? And if async submits are accumulated in the Solr server, is there some mechanism to queue them onto disk, or does it try to hold them all in RAM? And *if* the backlog caused an OOM condition, wouldn't that JVM have mostly crashed (if not completely)? Any guesses on the mostly likely failure point, and where to look? Thanks, Mark -- Mark Bennett / New Idea Engineering, Inc. / mbenn...@ideaeng.com Direct: 408-733-0387 / Main: 866-IDEA-ENG / Cell: 408-829-6513
