Hi Prasad, Have you tried with a smaller page size? Just how many documents you can fetch in one page with the given memory depends on the size of the documents. You'll have to try out what works for you.
Regardless of page size a cursor will still be the way to go to page through a large set of documents. Thomas Op vr 5 apr 2024 om 12:10 schreef prasad bezavada <prasadbezav...@gmail.com >: > Hello Thomas Corthals, > > Thank you very much for your valuable reply. I am trying to use cursors, > but for the first query also its taking so much time to get the results, > and next query I am getting heap memory error in my java application. > > On Fri, Apr 5, 2024 at 2:41 PM Thomas Corthals <tho...@klascement.net> > wrote: > > > Hi Prasad, > > > > This is expected with "deep paging": > > > > > https://solr.apache.org/guide/solr/latest/query-guide/pagination-of-results.html#performance-problems-with-deep-paging > > > > Have a look at cursors instead, that should solve your problem: > > > > > https://solr.apache.org/guide/solr/latest/query-guide/pagination-of-results.html#fetching-a-large-number-of-sorted-results-cursors > > > > Thomas > > > > Op vr 5 apr 2024 om 10:36 schreef prasad bezavada < > > prasadbezav...@gmail.com > > >: > > > > > Dear Team, > > > > > > I'm currently using Solr version 8.11.3, configured with RAM resources > > (125 > > > GB physical memory, 64 GB heap memory). The collection comprises 4 > shards > > > within the same node. Through our Java application ( SolrJ), > > > indexed approximately 8 million records from an RDBMS table into Solr. > > > > > > Presently, my task is to query this indexes and exporting the results > (5 > > > million records fetched with my solr query) to PDF format via our Java > > > application. To avoid potential heap memory issues, I've implemented > > > pagination (3 lakhs) in the query using start and setrows parameters. > > > > > > However, I've encountered an issue where the response time for > subsequent > > > queries to fetch the next set of results (e.g., 3 to 6 lakhs, 6 to 9 > > lakhs) > > > progressively increases, leading to socket timeout exceptions. > > > Additionally, Solr's physical memory consumption exceeds 90%, without > > > releasing it. > > > > > > I have several queries regarding this situation: > > > > > > Why does the query time in Solr increase with each pagination query? > > > What causes Solr to occupy over 90% of physical memory and fail to > > release > > > it? > > > What would be the optimal approach for retrieving 5 million records > from > > > our Java application and exporting them to PDF or other file formats? > > > Your insights and suggestions on resolving these issues would be > greatly > > > appreciated. > > > > > > > > > -- > > > *Thanks&Regards* > > > > > > *Prasad Bezavada* > > > > > > > > -- > *Thanks&Regards* > > *Prasad Bezavada* >
