> The very first thing I'd ask is "how much free space is on your disk > when this occurs?" Is it possible that you're simply filling up your > disk?
no, i've checked that already. all disks have plenty of space (they have a capacity of 2TB, and are currently filled up to 20%. > > do note that an optimize may require up to 2X the size of your index > if/when it occurs. Are you sure you aren't optimizing as you add > items to your index? > index size is not a problem in our case. Our index currently has about 3GB. What do you mean with "optimizing as you add items to your index"? > But I've never heard of Solr causing hard disk crashes, neither did we, and google is the same opinion. One thing that i've found is the mergeFactor value: http://wiki.apache.org/solr/SolrPerformanceFactors#mergeFactor Our sysadmin speculates that maybe the chunk size of our raid/harddisks and the segment size of the lucene index does not play well together. Does the lucene segment size affect how the data is written to the disk? thanks for your help. -robert > > Best > Erick > > 2010/12/2 Robert Gründler <rob...@dubture.com> > >> Hi, >> >> we have a serious harddisk problem, and it's definitely related to a >> full-import from a relational >> database into a solr index. >> >> The first time it happened on our development server, where the >> raidcontroller crashed during a full-import >> of ~ 8 Million documents. This happened 2 weeks ago, and in this period 2 >> of the harddisks where the solr >> index files are located stopped working (we needed to replace them). >> >> After the crash of the raid controller, we decided to move the development >> of solr/index related stuff to our >> local development machines. >> >> Yesterday i was running another full-import of ~10 Million documents on my >> local development machine, >> and during the import, a harddisk failure occurred. Since this failure, my >> harddisk activity seems to >> be around 100% all the time, even if no solr server is running at all. >> >> I've been googling the last 2 days to find some info about solr related >> harddisk problems, but i didn't find anything >> useful. >> >> Are there any steps we need to take care of in respect to harddisk failures >> when doing a full-import? Right now, >> our steps look like this: >> >> 1. Delete the current index >> 2. Restart solr, to load the updated schemas >> 3. Start the full import >> >> Initially, the solr index and the relational database were located on the >> same harddisk. After the crash, we moved >> the index to a separate harddisk, but nevertheless this harddisk crashed >> too. >> >> I'd really appreciate any hints on what we might do wrong when importing >> data, as we can't release this >> on our production servers when there's the risk of harddisk failures. >> >> >> thanks. >> >> >> -robert >> >> >> >> >> >>
