Re: Dataimport destroys our harddisks
That's the same series we use... we hade problems when running other disk-heavy operations like rsync and backup on them too.. But in our case we mostly had hangs or load > 180 :P... Can you simulate very heavy random disk i/o? if so then you could check if you still have the same problems... That's all I can be of help with, good luck :) /Sven 2010/12/2 Robert Gründler : > On Dec 2, 2010, at 15:43 , Sven Almgren wrote: > >> What Raid controller do you use, and what kernel version? (Assuming >> Linux). We hade problems during high load with a 3Ware raid controller >> and the current kernel for Ubuntu 10.04, we hade to downgrade the >> kernel... >> >> The problem was a bug in the driver that only showed up with very high >> disk load (as is the case when doing imports) >> > > We're running freebsd: > > RaidController 3ware 9500S-8 > Corrupt unit: Raid-10 3725.27GB 256K Stripe Size without BBU > Freebsd 7.2, UFS Filesystem. > > > >> /Sven >> >> 2010/12/2 Robert Gründler : The very first thing I'd ask is "how much free space is on your disk when this occurs?" Is it possible that you're simply filling up your disk? >>> >>> no, i've checked that already. all disks have plenty of space (they have >>> a capacity of 2TB, and are currently filled up to 20%. >>> do note that an optimize may require up to 2X the size of your index if/when it occurs. Are you sure you aren't optimizing as you add items to your index? >>> >>> index size is not a problem in our case. Our index currently has about 3GB. >>> >>> What do you mean with "optimizing as you add items to your index"? >>> But I've never heard of Solr causing hard disk crashes, >>> >>> neither did we, and google is the same opinion. >>> >>> One thing that i've found is the mergeFactor value: >>> >>> http://wiki.apache.org/solr/SolrPerformanceFactors#mergeFactor >>> >>> Our sysadmin speculates that maybe the chunk size of our raid/harddisks >>> and the segment size of the lucene index does not play well together. >>> >>> Does the lucene segment size affect how the data is written to the disk? >>> >>> >>> thanks for your help. >>> >>> >>> -robert >>> >>> >>> >>> >>> >>> >>> Best Erick 2010/12/2 Robert Gründler > Hi, > > we have a serious harddisk problem, and it's definitely related to a > full-import from a relational > database into a solr index. > > The first time it happened on our development server, where the > raidcontroller crashed during a full-import > of ~ 8 Million documents. This happened 2 weeks ago, and in this period 2 > of the harddisks where the solr > index files are located stopped working (we needed to replace them). > > After the crash of the raid controller, we decided to move the development > of solr/index related stuff to our > local development machines. > > Yesterday i was running another full-import of ~10 Million documents on my > local development machine, > and during the import, a harddisk failure occurred. Since this failure, my > harddisk activity seems to > be around 100% all the time, even if no solr server is running at all. > > I've been googling the last 2 days to find some info about solr related > harddisk problems, but i didn't find anything > useful. > > Are there any steps we need to take care of in respect to harddisk > failures > when doing a full-import? Right now, > our steps look like this: > > 1. Delete the current index > 2. Restart solr, to load the updated schemas > 3. Start the full import > > Initially, the solr index and the relational database were located on the > same harddisk. After the crash, we moved > the index to a separate harddisk, but nevertheless this harddisk crashed > too. > > I'd really appreciate any hints on what we might do wrong when importing > data, as we can't release this > on our production servers when there's the risk of harddisk failures. > > > thanks. > > > -robert > > > > > > >>> >>> > >
Re: Dataimport destroys our harddisks
On Dec 2, 2010, at 15:43 , Sven Almgren wrote: > What Raid controller do you use, and what kernel version? (Assuming > Linux). We hade problems during high load with a 3Ware raid controller > and the current kernel for Ubuntu 10.04, we hade to downgrade the > kernel... > > The problem was a bug in the driver that only showed up with very high > disk load (as is the case when doing imports) > We're running freebsd: RaidController 3ware 9500S-8 Corrupt unit: Raid-10 3725.27GB 256K Stripe Size without BBU Freebsd 7.2, UFS Filesystem. > /Sven > > 2010/12/2 Robert Gründler : >>> The very first thing I'd ask is "how much free space is on your disk >>> when this occurs?" Is it possible that you're simply filling up your >>> disk? >> >> no, i've checked that already. all disks have plenty of space (they have >> a capacity of 2TB, and are currently filled up to 20%. >> >>> >>> do note that an optimize may require up to 2X the size of your index >>> if/when it occurs. Are you sure you aren't optimizing as you add >>> items to your index? >>> >> >> index size is not a problem in our case. Our index currently has about 3GB. >> >> What do you mean with "optimizing as you add items to your index"? >> >>> But I've never heard of Solr causing hard disk crashes, >> >> neither did we, and google is the same opinion. >> >> One thing that i've found is the mergeFactor value: >> >> http://wiki.apache.org/solr/SolrPerformanceFactors#mergeFactor >> >> Our sysadmin speculates that maybe the chunk size of our raid/harddisks >> and the segment size of the lucene index does not play well together. >> >> Does the lucene segment size affect how the data is written to the disk? >> >> >> thanks for your help. >> >> >> -robert >> >> >> >> >> >> >> >>> >>> Best >>> Erick >>> >>> 2010/12/2 Robert Gründler >>> Hi, we have a serious harddisk problem, and it's definitely related to a full-import from a relational database into a solr index. The first time it happened on our development server, where the raidcontroller crashed during a full-import of ~ 8 Million documents. This happened 2 weeks ago, and in this period 2 of the harddisks where the solr index files are located stopped working (we needed to replace them). After the crash of the raid controller, we decided to move the development of solr/index related stuff to our local development machines. Yesterday i was running another full-import of ~10 Million documents on my local development machine, and during the import, a harddisk failure occurred. Since this failure, my harddisk activity seems to be around 100% all the time, even if no solr server is running at all. I've been googling the last 2 days to find some info about solr related harddisk problems, but i didn't find anything useful. Are there any steps we need to take care of in respect to harddisk failures when doing a full-import? Right now, our steps look like this: 1. Delete the current index 2. Restart solr, to load the updated schemas 3. Start the full import Initially, the solr index and the relational database were located on the same harddisk. After the crash, we moved the index to a separate harddisk, but nevertheless this harddisk crashed too. I'd really appreciate any hints on what we might do wrong when importing data, as we can't release this on our production servers when there's the risk of harddisk failures. thanks. -robert >> >>
Re: Dataimport destroys our harddisks
What Raid controller do you use, and what kernel version? (Assuming Linux). We hade problems during high load with a 3Ware raid controller and the current kernel for Ubuntu 10.04, we hade to downgrade the kernel... The problem was a bug in the driver that only showed up with very high disk load (as is the case when doing imports) /Sven 2010/12/2 Robert Gründler : >> The very first thing I'd ask is "how much free space is on your disk >> when this occurs?" Is it possible that you're simply filling up your >> disk? > > no, i've checked that already. all disks have plenty of space (they have > a capacity of 2TB, and are currently filled up to 20%. > >> >> do note that an optimize may require up to 2X the size of your index >> if/when it occurs. Are you sure you aren't optimizing as you add >> items to your index? >> > > index size is not a problem in our case. Our index currently has about 3GB. > > What do you mean with "optimizing as you add items to your index"? > >> But I've never heard of Solr causing hard disk crashes, > > neither did we, and google is the same opinion. > > One thing that i've found is the mergeFactor value: > > http://wiki.apache.org/solr/SolrPerformanceFactors#mergeFactor > > Our sysadmin speculates that maybe the chunk size of our raid/harddisks > and the segment size of the lucene index does not play well together. > > Does the lucene segment size affect how the data is written to the disk? > > > thanks for your help. > > > -robert > > > > > > > >> >> Best >> Erick >> >> 2010/12/2 Robert Gründler >> >>> Hi, >>> >>> we have a serious harddisk problem, and it's definitely related to a >>> full-import from a relational >>> database into a solr index. >>> >>> The first time it happened on our development server, where the >>> raidcontroller crashed during a full-import >>> of ~ 8 Million documents. This happened 2 weeks ago, and in this period 2 >>> of the harddisks where the solr >>> index files are located stopped working (we needed to replace them). >>> >>> After the crash of the raid controller, we decided to move the development >>> of solr/index related stuff to our >>> local development machines. >>> >>> Yesterday i was running another full-import of ~10 Million documents on my >>> local development machine, >>> and during the import, a harddisk failure occurred. Since this failure, my >>> harddisk activity seems to >>> be around 100% all the time, even if no solr server is running at all. >>> >>> I've been googling the last 2 days to find some info about solr related >>> harddisk problems, but i didn't find anything >>> useful. >>> >>> Are there any steps we need to take care of in respect to harddisk failures >>> when doing a full-import? Right now, >>> our steps look like this: >>> >>> 1. Delete the current index >>> 2. Restart solr, to load the updated schemas >>> 3. Start the full import >>> >>> Initially, the solr index and the relational database were located on the >>> same harddisk. After the crash, we moved >>> the index to a separate harddisk, but nevertheless this harddisk crashed >>> too. >>> >>> I'd really appreciate any hints on what we might do wrong when importing >>> data, as we can't release this >>> on our production servers when there's the risk of harddisk failures. >>> >>> >>> thanks. >>> >>> >>> -robert >>> >>> >>> >>> >>> >>> > >
Re: Dataimport destroys our harddisks
> The very first thing I'd ask is "how much free space is on your disk > when this occurs?" Is it possible that you're simply filling up your > disk? no, i've checked that already. all disks have plenty of space (they have a capacity of 2TB, and are currently filled up to 20%. > > do note that an optimize may require up to 2X the size of your index > if/when it occurs. Are you sure you aren't optimizing as you add > items to your index? > index size is not a problem in our case. Our index currently has about 3GB. What do you mean with "optimizing as you add items to your index"? > But I've never heard of Solr causing hard disk crashes, neither did we, and google is the same opinion. One thing that i've found is the mergeFactor value: http://wiki.apache.org/solr/SolrPerformanceFactors#mergeFactor Our sysadmin speculates that maybe the chunk size of our raid/harddisks and the segment size of the lucene index does not play well together. Does the lucene segment size affect how the data is written to the disk? thanks for your help. -robert > > Best > Erick > > 2010/12/2 Robert Gründler > >> Hi, >> >> we have a serious harddisk problem, and it's definitely related to a >> full-import from a relational >> database into a solr index. >> >> The first time it happened on our development server, where the >> raidcontroller crashed during a full-import >> of ~ 8 Million documents. This happened 2 weeks ago, and in this period 2 >> of the harddisks where the solr >> index files are located stopped working (we needed to replace them). >> >> After the crash of the raid controller, we decided to move the development >> of solr/index related stuff to our >> local development machines. >> >> Yesterday i was running another full-import of ~10 Million documents on my >> local development machine, >> and during the import, a harddisk failure occurred. Since this failure, my >> harddisk activity seems to >> be around 100% all the time, even if no solr server is running at all. >> >> I've been googling the last 2 days to find some info about solr related >> harddisk problems, but i didn't find anything >> useful. >> >> Are there any steps we need to take care of in respect to harddisk failures >> when doing a full-import? Right now, >> our steps look like this: >> >> 1. Delete the current index >> 2. Restart solr, to load the updated schemas >> 3. Start the full import >> >> Initially, the solr index and the relational database were located on the >> same harddisk. After the crash, we moved >> the index to a separate harddisk, but nevertheless this harddisk crashed >> too. >> >> I'd really appreciate any hints on what we might do wrong when importing >> data, as we can't release this >> on our production servers when there's the risk of harddisk failures. >> >> >> thanks. >> >> >> -robert >> >> >> >> >> >>
Re: Dataimport destroys our harddisks
The very first thing I'd ask is "how much free space is on your disk when this occurs?" Is it possible that you're simply filling up your disk? do note that an optimize may require up to 2X the size of your index if/when it occurs. Are you sure you aren't optimizing as you add items to your index? But I've never heard of Solr causing hard disk crashes, it doesn't do anything special but read/write... Best Erick 2010/12/2 Robert Gründler > Hi, > > we have a serious harddisk problem, and it's definitely related to a > full-import from a relational > database into a solr index. > > The first time it happened on our development server, where the > raidcontroller crashed during a full-import > of ~ 8 Million documents. This happened 2 weeks ago, and in this period 2 > of the harddisks where the solr > index files are located stopped working (we needed to replace them). > > After the crash of the raid controller, we decided to move the development > of solr/index related stuff to our > local development machines. > > Yesterday i was running another full-import of ~10 Million documents on my > local development machine, > and during the import, a harddisk failure occurred. Since this failure, my > harddisk activity seems to > be around 100% all the time, even if no solr server is running at all. > > I've been googling the last 2 days to find some info about solr related > harddisk problems, but i didn't find anything > useful. > > Are there any steps we need to take care of in respect to harddisk failures > when doing a full-import? Right now, > our steps look like this: > > 1. Delete the current index > 2. Restart solr, to load the updated schemas > 3. Start the full import > > Initially, the solr index and the relational database were located on the > same harddisk. After the crash, we moved > the index to a separate harddisk, but nevertheless this harddisk crashed > too. > > I'd really appreciate any hints on what we might do wrong when importing > data, as we can't release this > on our production servers when there's the risk of harddisk failures. > > > thanks. > > > -robert > > > > > >
Dataimport destroys our harddisks
Hi, we have a serious harddisk problem, and it's definitely related to a full-import from a relational database into a solr index. The first time it happened on our development server, where the raidcontroller crashed during a full-import of ~ 8 Million documents. This happened 2 weeks ago, and in this period 2 of the harddisks where the solr index files are located stopped working (we needed to replace them). After the crash of the raid controller, we decided to move the development of solr/index related stuff to our local development machines. Yesterday i was running another full-import of ~10 Million documents on my local development machine, and during the import, a harddisk failure occurred. Since this failure, my harddisk activity seems to be around 100% all the time, even if no solr server is running at all. I've been googling the last 2 days to find some info about solr related harddisk problems, but i didn't find anything useful. Are there any steps we need to take care of in respect to harddisk failures when doing a full-import? Right now, our steps look like this: 1. Delete the current index 2. Restart solr, to load the updated schemas 3. Start the full import Initially, the solr index and the relational database were located on the same harddisk. After the crash, we moved the index to a separate harddisk, but nevertheless this harddisk crashed too. I'd really appreciate any hints on what we might do wrong when importing data, as we can't release this on our production servers when there's the risk of harddisk failures. thanks. -robert