Hello Thomas Corthals, Thank you very much for your valuable reply. I am trying to use cursors, but for the first query also its taking so much time to get the results, and next query I am getting heap memory error in my java application.
On Fri, Apr 5, 2024 at 2:41 PM Thomas Corthals <tho...@klascement.net> wrote: > Hi Prasad, > > This is expected with "deep paging": > > https://solr.apache.org/guide/solr/latest/query-guide/pagination-of-results.html#performance-problems-with-deep-paging > > Have a look at cursors instead, that should solve your problem: > > https://solr.apache.org/guide/solr/latest/query-guide/pagination-of-results.html#fetching-a-large-number-of-sorted-results-cursors > > Thomas > > Op vr 5 apr 2024 om 10:36 schreef prasad bezavada < > prasadbezav...@gmail.com > >: > > > Dear Team, > > > > I'm currently using Solr version 8.11.3, configured with RAM resources > (125 > > GB physical memory, 64 GB heap memory). The collection comprises 4 shards > > within the same node. Through our Java application ( SolrJ), > > indexed approximately 8 million records from an RDBMS table into Solr. > > > > Presently, my task is to query this indexes and exporting the results (5 > > million records fetched with my solr query) to PDF format via our Java > > application. To avoid potential heap memory issues, I've implemented > > pagination (3 lakhs) in the query using start and setrows parameters. > > > > However, I've encountered an issue where the response time for subsequent > > queries to fetch the next set of results (e.g., 3 to 6 lakhs, 6 to 9 > lakhs) > > progressively increases, leading to socket timeout exceptions. > > Additionally, Solr's physical memory consumption exceeds 90%, without > > releasing it. > > > > I have several queries regarding this situation: > > > > Why does the query time in Solr increase with each pagination query? > > What causes Solr to occupy over 90% of physical memory and fail to > release > > it? > > What would be the optimal approach for retrieving 5 million records from > > our Java application and exporting them to PDF or other file formats? > > Your insights and suggestions on resolving these issues would be greatly > > appreciated. > > > > > > -- > > *Thanks&Regards* > > > > *Prasad Bezavada* > > > -- *Thanks&Regards* *Prasad Bezavada*
