Hi Another werid problem. When we setup the autocommit properties, we suppose that the index fille will created every commited.So that the size of the index files will be large enough. We do not want to keep too many small files as [1].
How to control the size of the index files. [1]---------------------------------------------------------------------------- ...omited .... 548KB index/_28w_Lucene41_0.doc 289KB index/_28w_Lucene41_0.pos 1.1M index/_28w_Lucene41_0.tim 24K index/_28w_Lucene41_0.tip 2.1M index/_28w.fdt 766B index/_28w.fdx 5KB index/_28w.fnm 40K index/_28w.nvd 79K index/_28w.nvm 364B index/_28w.si 518KB index/_28x_Lucene41_0.doc 290KB index/_28x_Lucene41_0.pos 1.2M index/_28x_Lucene41_0.tim 28K index/_28x_Lucene41_0.tip 2.1M index/_28x.fdt 843B index/_28x.fdx 5KB index/_28x.fnm 40K index/_28x.nvd 79K index/_28x.nvm 386B index/_28x.si ...omited .... ------------------------------------------------------------------------- 2013/9/17 YouPeng Yang <yypvsxf19870...@gmail.com> > Hi Shawn > > Thank your very much for your reponse. > > I lauch the full-import task on the web page of solr/admin . And I do > check the commit option. > The new docs would be committed after the operation. > The commit option is defferent with the autocommit,right? If the import > datasets are too large that leads to poor performance or > other problems ,such as [1]. > > The exception that indicate that -Too many open files-,we thought is > because of the ulimit. > > > > > > [1]---------------------------------------------------------------------------- > java.io.FileNotFoundException: > /data/apache-tomcat/webapps/solr/collection1/data/index/_149d.fdx (Too many > open files) > > java.io.FileNotFoundException: > /data/apache-tomcat/webapps/solr/collection1/data/index/_149e.fdx (Too many > open files) > > java.io.FileNotFoundException: > /data/apache-tomcat/webapps/solr/collection1/data/index/_149f.fdx (Too many > open files) > > java.io.FileNotFoundException: > /data/apache-tomcat/webapps/solr/collection1/data/index/_149g.fdx (Too many > open files) > > java.io.FileNotFoundException: > /data/apache-tomcat/webapps/solr/collection1/data/index/_149h.fdx (Too many > open files) > > java.io.FileNotFoundException: > /data/apache-tomcat/webapps/solr/collection1/data/index/_149i.fdx (Too many > open files) > > java.io.FileNotFoundException: > /data/apache-tomcat/webapps/solr/collection1/data/index/_149j.fdx (Too many > open files) > > java.io.FileNotFoundException: > /data/apache-tomcat/webapps/solr/collection1/data/index/_149k.fdx (Too many > open files) > > java.io.FileNotFoundException: > /data/apache-tomcat/webapps/solr/collection1/data/index/_149l.fdx (Too many > open files) > > java.io.FileNotFoundException: > /data/apache-tomcat/webapps/solr/collection1/data/index/_149m.fdx (Too many > open files) > > java.io.FileNotFoundException: > /data/apache-tomcat/webapps/solr/collection1/data/index/_149n.fdx (Too many > open files) > > java.io.FileNotFoundException: > /data/apache-tomcat/webapps/solr/collection1/data/index/_149o.fdx (Too many > open files) > > java.io.FileNotFoundException: > /data/apache-tomcat/webapps/solr/collection1/data/index/_149p.fdx (Too many > open files) > > java.io.FileNotFoundException: > /data/apache-tomcat/webapps/solr/collection1/data/index/_149q.fdx (Too many > open files) > > java.io.FileNotFoundException: > /data/apache-tomcat/webapps/solr/collection1/data/index/_149r.fdx (Too many > open files) > > java.io.FileNotFoundException: > /data/apache-tomcat/webapps/solr/collection1/data/index/_149s.fdx (Too many > open files) > > > > 2013/9/17 Shawn Heisey <s...@elyograg.org> > >> On 9/16/2013 8:26 PM, YouPeng Yang wrote: >> > I'm using the DIH to import data from oracle database with Solr4.4 >> > Finally I get 2.7GB index data and 4.1GB tlog data.And the number of >> > docs was 10900000. >> > >> > At first, I move the 2.7GB index data to another new Solr Server in >> > tomcat7. After I start the tomcat ,I find the total number of docs was >> just >> > half of the orginal number. >> > So I thought that maybe the left docs were not commited to index >> > files,and the tlog needed to be replayed . >> >> You need to turn on autoCommit in your solrconfig.xml so that there are >> hard commits happening on a regular basis that flush all indexed data to >> disk and start new transaction log files. I will give you a link with >> some information about that below. >> >> > Sequently , I moved the 2.7GB index data and 4.1GB tlog data to the >> new >> > Solr Server in tomcat7. >> > After I start the tomcat,an exception comes up as [1]. >> > Then it halts.I can not access the tomcat server URL. >> > I noticed that the CPU utilization was high by using the comand: >> top >> > -d 1 | grep tomcatPid. >> > I thought solr was replaying the updatelog.And I wait a long time and it >> > still was replaying. As results ,I give up. >> >> I don't know what the exception was about, but it is likely that it WAS >> replaying the log. With 4.1GB of transaction log, that's going to take >> a LONG time, during which Solr will be unavailable. It always replays >> the entire transaction log. The key, as mentioned above, is in keeping >> that log small. >> >> Here's a wiki page about the slow startup problem and an example of how >> to configure autoCommit to deal with it: >> >> http://wiki.apache.org/solr/SolrPerformanceProblems#Slow_startup >> >> There's a lot of other good information on that page. >> >> Thanks, >> Shawn >> >> >
