Re: [zfs-discuss] Still more questions WRT selecting a mobo for small ZFS RAID
On Sat, 15 Nov 2008 13:38:53 -0600 Al Hopper [EMAIL PROTECTED] wrote: So I guess I've been lucky also - but IMHO the failure rate for RAM these days is pretty small[1]. I've also been around hundreds of SPARC boxes and, again, very, few RAM failures (one is all that I can remember). Risk management is exactly that. You have to determine where the risk is and how important it is and how likely it is to bite. And then allocate costs from your budget to minimize that risk. So I guess, I do have to go for ECC ram when I build a new server. I also understood that like you wrote The P45 based boards are a no-brainer Intel MoBo's are a no-go when you want ECC ram -and- want it a little cheap. So, what -is- a really good MB that supports ECC ram (min.8MB) and what processor is recommended? -- Dick Hoogendijk -- PGP/GnuPG key: 01D2433D + http://nagual.nl/ | SunOS sxce snv101 ++ + All that's really worth doing is what we do for others (Lewis Carrol) ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Still more questions WRT selecting a mobo for small ZFS RAID
Ian Collins wrote: Al Hopper wrote: On Sat, Nov 15, 2008 at 9:26 AM, Richard Elling [EMAIL PROTECTED] wrote: dick hoogendijk wrote: On Sat, 15 Nov 2008 18:49:17 +1300 Ian Collins [EMAIL PROTECTED] wrote: [EMAIL PROTECTED] wrote: WD Caviar Black drive [...] Intel E7200 2.53GHz 3MB L2 The P45 based boards are a no-brainer 16G of DDR2-1066 with P45 or 8G of ECC DDR2-800 with 3210 based boards That is the question. I guess the answer is how valuable is your data? I disagree. The answer is: go for the 16G and make backups. The 16G system will work far more easy and I may be lucky but in the past years I did not have ZFS issues with my non-ECC ram ;-) You are lucky. I recommend ECC RAM for any data that you care about. Remember, if there is a main memory corruption, that may impact the data that ZFS writes which will negate any on-disk redundancy. And yes, this does occur -- check the archives for the tales of woe. I agree with your recommendation Richard. OTOH I've built/used a bunch of systems over several years that were mostly non ECC equipped and only lost one DIMM along the way. So I guess I've been lucky also - but IMHO the failure rate for RAM these days is pretty small[1]. I've also been around hundreds of SPARC boxes and, again, very, few RAM failures (one is all that I can remember). I think the situation will change with the current expansion in RAM sizes. Five years ago with mainly 32 bit x86 systems, 4G of ram was a lot (even on most Sparc boxes). Today 32 and 64GB are becoming common. Desktop systems have seen similar growth. Let's do some math. A generally accepted Soft Error Rate (SER) for DRAMs is 1,000 FITs or an Annualized Failure Rate (AFR) of 0.88%. If a non-ECC DIMM has 8 chips then your AFR is 7%, or 14% for 16 chip DIMMs. My desktop has 4 DIMMs at 16-chips each, so I should expect an AFR of 56%. Since these are soft errors, a RAM test program may not detect it. ECC will dramatically reduce the system-level effects of SERs. Extended ECC will further reduce this by about 2 orders of magnitude. ZFS also uses system RAM in a way it hasn't been used before. Memory that would have been unused or holding static pages is now churning rapidly, in a way similar memory testers like memtest86. Random patterns are cycling though RAM like never before, greatly increasing the chances for hitting a bad bit or addressing error. I've had RAM faults that have taken hours with memtest86 to hit the trigger bit pattern that would have gone unnoticed for years if I hadn't seen data corruption with ZFS. ZFS may turn out to be the ultimate RAM soak tester! :-) no, not really. SERs are more of a problem for idle DRAM because the probability of a SER affecting you is a function of the time the data has been sitting in RAM waiting to be affected by upsets. Note: there are some studies suggesting a correlation between SERs and hard faults. In practice, it doesn't really matter why or how the fault occurred, the solution is ECC, Extended ECC, or memory mirroring. -- richard ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Still more questions WRT selecting a mobo for small ZFS RAID
[EMAIL PROTECTED] wrote: RTL8211C IP checksum offload is broken. You can disable it, but you have to edit /etc/system. See CR 6686415 for details. http://bugs.opensolaris.org/bugdatabase/view_bug.do?bug_id=6686415 -- richard I think the proper way to state this is the driver doesn't properly support checksum offload. (In many of the newer realtek cards the way the offload is done is differently) Yes, this is a better way to say it. The bug is in, or can be solved by, the driver. -- richard ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Still more questions WRT selecting a mobo for small ZFS RAID
I looked at this a month back, i was leaning towards intel for performance and power consumption but went for AMD doe to lack of ECC support in most of the Intel chipsets. I went for a AM2+ GeForce 8200 motherboard which seemed more stable with Solaris than 8300. With the AM2+ socket I can wait for the new 45nm CPUs, I bought the cheapest dual-core I could find for now (which did not support PM). I am very happy with the system except for the fact that the onboard NIC doesn't work. Which NIC is that? Casper ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Still more questions WRT selecting a mobo for small ZFS RAID
On Sat, 15 Nov 2008 18:49:17 +1300 Ian Collins [EMAIL PROTECTED] wrote: [EMAIL PROTECTED] wrote: WD Caviar Black drive [...] Intel E7200 2.53GHz 3MB L2 The P45 based boards are a no-brainer 16G of DDR2-1066 with P45 or 8G of ECC DDR2-800 with 3210 based boards That is the question. I guess the answer is how valuable is your data? I disagree. The answer is: go for the 16G and make backups. The 16G system will work far more easy and I may be lucky but in the past years I did not have ZFS issues with my non-ECC ram ;-) -- Dick Hoogendijk -- PGP/GnuPG key: 01D2433D + http://nagual.nl/ | SunOS sxce snv101 ++ + All that's really worth doing is what we do for others (Lewis Carrol) ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Still more questions WRT selecting a mobo for small ZFS RAID
dick hoogendijk wrote: On Sat, 15 Nov 2008 18:49:17 +1300 Ian Collins [EMAIL PROTECTED] wrote: [EMAIL PROTECTED] wrote: WD Caviar Black drive [...] Intel E7200 2.53GHz 3MB L2 The P45 based boards are a no-brainer 16G of DDR2-1066 with P45 or 8G of ECC DDR2-800 with 3210 based boards That is the question. I guess the answer is how valuable is your data? I disagree. The answer is: go for the 16G and make backups. The 16G system will work far more easy and I may be lucky but in the past years I did not have ZFS issues with my non-ECC ram ;-) You are lucky. I recommend ECC RAM for any data that you care about. Remember, if there is a main memory corruption, that may impact the data that ZFS writes which will negate any on-disk redundancy. And yes, this does occur -- check the archives for the tales of woe. -- richard ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Still more questions WRT selecting a mobo for small ZFS RAID
On Sat, Nov 15, 2008 at 9:26 AM, Richard Elling [EMAIL PROTECTED] wrote: dick hoogendijk wrote: On Sat, 15 Nov 2008 18:49:17 +1300 Ian Collins [EMAIL PROTECTED] wrote: [EMAIL PROTECTED] wrote: WD Caviar Black drive [...] Intel E7200 2.53GHz 3MB L2 The P45 based boards are a no-brainer 16G of DDR2-1066 with P45 or 8G of ECC DDR2-800 with 3210 based boards That is the question. I guess the answer is how valuable is your data? I disagree. The answer is: go for the 16G and make backups. The 16G system will work far more easy and I may be lucky but in the past years I did not have ZFS issues with my non-ECC ram ;-) You are lucky. I recommend ECC RAM for any data that you care about. Remember, if there is a main memory corruption, that may impact the data that ZFS writes which will negate any on-disk redundancy. And yes, this does occur -- check the archives for the tales of woe. I agree with your recommendation Richard. OTOH I've built/used a bunch of systems over several years that were mostly non ECC equipped and only lost one DIMM along the way. So I guess I've been lucky also - but IMHO the failure rate for RAM these days is pretty small[1]. I've also been around hundreds of SPARC boxes and, again, very, few RAM failures (one is all that I can remember). Risk management is exactly that. You have to determine where the risk is and how important it is and how likely it is to bite. And then allocate costs from your budget to minimize that risk. Remember that you won't totally eliminate all risk - but you can minimize it. At the time when there was a big cost delta between ECC and non ECC RAM parts, I always went with the most (non ECC) RAM that the budget would support. That was my personal risk assessment and priority. I think it was a good decision and it did'nt cause me any grief. [1] I do recommend that you test the heck out of new RAM parts and ensure that they get some airflow - especially if they are getting a supply of hot air from any nearby CPU coolers. Even the simple finger test will tell you if you need a fan for your RAM DIMMs. -- Al Hopper Logical Approach Inc,Plano,TX [EMAIL PROTECTED] Voice: 972.379.2133 Timezone: US CDT OpenSolaris Governing Board (OGB) Member - Apr 2005 to Mar 2007 http://www.opensolaris.org/os/community/ogb/ogb_2005-2007/ ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Still more questions WRT selecting a mobo for small ZFS RAID
On Fri, Nov 14, 2008 at 4:43 PM, gnomad [EMAIL PROTECTED] wrote: Like many others, I am looking to put together a SOHO NAS based on ZFS/CIFS. The plan is 6 x 1TB drives in RAIDZ2 configuration, driven via mobo with 6 SATA ports. I've read most, if not all, of the threads here, as well as sbredon's excellent article on building a home NAS, yet I still have a number of unanswered questions. I was leaning heavily towards the M2N-E for a while, but it has a number of strikes against it: - Discontinued, not likely to be available to build additional rigs - No longer available via Newegg - MCP55 copy/fs lockup problem still unresolved Although the Nvidia 750a chipset seems to be supported by the sata_nv driver, I have read several open bug reports which indicate it may not be a good choice. Likewise, I have read several negatives on the entire line of AMD southbridges under Solaris. As it stands, the only compelling reason to go with an AMD CPU seems to be support for ECC in inexpensive motherboards. On the Intel side of things, there seem to be plenty of good reports of the ICH 7/8/9 chipsets that support AHCI and no negatives. The only negative on the Intel side seems to be the lack of ECC memory outside of expensive server mobos. As things stand, I am leaning towards a P35/ICH9 or P45/ICH10 setup. In particular, I am considering either the P5E-VM HDMI for a small integrated solution or the GA-EP45-UD3P for something with greater expandability. So, my questions: - Has the MCP55 copy/fs lockup bug been fixed yet? - Have the Nvidia 750a driver issues been resolved? - Are the AMD southbridges still in the not-recomemded category? - Are there any downsides to going with with an ICH controller with the AHCI driver? - Are there any performance issues to be aware of with the ICH vs the MCP55? - Are there any inexpensive Intel mobos that do support ECC memory and the latest Intel 45 nm CPUs? - Any comments on the P5E-VM or GA-EP45 boards? Many thanks in advance! -- I'm going to be somewhat rude and bypass your list of detailed questions - but give you my thoughts on a motherboard recommendation (and other hardware). a) related to the 1Tb disks, I'd highly recommend the WD Caviar Black drive. Its fast and the firmware does a great job on different workloads that vary between large file sequencial read (workloads) to (workloads that demand) lots of small random reads/writes. Their dual processor controller architecture really works. b) If I were building a system today, I'd go Intel - even thought I'm an AMD fanboy - but I can't recommend AMD today ... unfortunately. c) RAM is the most important attribute of a ZFS based server. Think lots of RAM. Unfortunately, Intel has turned the market into a two-tier market, with the lower (price) tier limited to 4 DIMM slots. So, pick a board that has been tested with 4 * 2Gb or 4 * 4Gb DIMM configs and plan on building a system with at least 4*2Gb DIMMs today. c1) If you have a choice, based on your budgetary constraints, between (for example) 4*1G of performance RAM and 4*2Gb of value (main stream performance) RAM - go with value RAM. Whatever you do, PLEASE maximize system memory capacity. d) The P45 based boards are a no-brainer. Great performance, good pricing, reasonable power consumption and highly mature. e) If the board is going to be *only* used as a NAS, the current CPU sweet spot is, IMHO, the IntelIntel Core 2 Duo E7200 (45nm, 2.53 GHz, 3MB L2 Cache). Plenty of horsepower, low-power consumption, nice cache capacity and priced to go! f) If you intend to use the box for other demanding tasks (for example, running other OS under VirtualBox) and need more CPU power, I'd pick the E8400 (dual core). But remember, the priority is RAM capacity first, upgraded CPU second. I really think that the E7200 will work well in your application. f) Look at the following reviews and pick a motherboard from any of them: http://www.tomshardware.com/reviews/energy-efficient-motherboard,2067.html http://techreport.com/articles.x/15737 http://www.tomshardware.com/reviews/intel-p45-motherboard,2001.html I really don't think you can go wrong with any Intel based system that has had a halfway decent review report card. There has *never* been a better time to buy/build a system. HTH. Email me offlist if I can help. Regards, -- Al Hopper Logical Approach Inc,Plano,TX [EMAIL PROTECTED] Voice: 972.379.2133 Timezone: US CDT OpenSolaris Governing Board (OGB) Member - Apr 2005 to Mar 2007 http://www.opensolaris.org/os/community/ogb/ogb_2005-2007/ ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Still more questions WRT selecting a mobo for small ZFS RAID
WD Caviar Black drive [...] Intel E7200 2.53GHz 3MB L2 The P45 based boards are a no-brainer 16G of DDR2-1066 with P45 or 8G of ECC DDR2-800 with 3210 based boards That is the question. Rob ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Still more questions WRT selecting a mobo for small ZFS RAID
[EMAIL PROTECTED] wrote: WD Caviar Black drive [...] Intel E7200 2.53GHz 3MB L2 The P45 based boards are a no-brainer 16G of DDR2-1066 with P45 or 8G of ECC DDR2-800 with 3210 based boards That is the question. I guess the answer is how valuable is your data? -- Ian. ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
