[ https://issues.apache.org/jira/browse/SOLR-14645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17189774#comment-17189774 ]
Ishan Chattopadhyaya commented on SOLR-14645: --------------------------------------------- Thanks [~mbolis], this was merged in the community package repo: https://github.com/rohitbemax/dataimporthandler/pull/3 > Make FileListEntityProcessor streaming to prevent OOM > ----------------------------------------------------- > > Key: SOLR-14645 > URL: https://issues.apache.org/jira/browse/SOLR-14645 > Project: Solr > Issue Type: Improvement > Components: contrib - DataImportHandler > Reporter: Marco Bolis > Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > FileListEntityProcessor currently works by keeping the whole list of files to > process in memory and iterating over it. > If the list is very large (think some 100s of Ks) Solr goes OOM and crashes. > Using Java 8 Streams and NIO2 it is possible to entirely avoid this behavior. -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org