DIH streams 1 row at a time. DIH is just a component in Solr. Solr indexing also takes a lot of memory
On Tue, Apr 14, 2009 at 12:02 PM, Mani Kumar <manikumarchau...@gmail.com> wrote: > Yes its throwing the same OOM error and from same place... > yes i will try increasing the size ... just curious : how this dataimport > works? > > Does it loads the whole table into memory? > > Is there any estimate about how much memory it needs to create index for 1GB > of data. > > thx > mani > > On Tue, Apr 14, 2009 at 11:48 AM, Shalin Shekhar Mangar < > shalinman...@gmail.com> wrote: > >> On Tue, Apr 14, 2009 at 11:36 AM, Mani Kumar <manikumarchau...@gmail.com >> >wrote: >> >> > Hi Shalin: >> > yes i tried with batchSize="-1" parameter as well >> > >> > here the config i tried with >> > >> > <dataConfig> >> > >> > <dataSource type="JdbcDataSource" batchSize="-1" name="sp" >> > driver="com.mysql.jdbc.Driver" >> > url="jdbc:mysql://localhost/mydb_development" >> > user="root" password="******" /> >> > >> > >> > I hope i have used batchSize parameter @ right place. >> > >> > >> Yes that is correct. Did it still throw OOM from the same place? >> >> I'd suggest you increase the heap and see what works for you. Also try >> -server on the jvm. >> >> -- >> Regards, >> Shalin Shekhar Mangar. >> > -- --Noble Paul
