Re: formatting SD flash memory (rant) - was RE: Damaged images
On 3/14/2011 10:25 PM, Krisjanis Linkevics wrote: Now whether cards become bad with time or not is mostly dependant on the hardware/software controller on the card itself - depending on how good it is at choosing places to write files and how good it is at marking the bad spots on the card - the card could either die very fast or live practically forever. So what card you buy really matters. Kris, I am somewhat confused now. What you say makes perfect sense except one detail. I thought that CF cards were those that had controller on board. The SD cards as I understand don't have controller on board. Therefore it makes certain sense (may be not too much sense, but still) to write to/format the card in the same controller (the camera). I am not sure if reading from the card can actually damage its contents... Another question I'd like to ask - how many read/write cycles there has to be made before a certain location on the card becomes flaky? I mean what is card's MTBF? You see, I still have that 1GB SD card (SanDisk) that I bought back in 2006 that still works. My empirical understanding is that several tens of thousands of read/writes don't have significant influence on the card performance. Boris -- PDML Pentax-Discuss Mail List PDML@pdml.net http://pdml.net/mailman/listinfo/pdml_pdml.net to UNSUBSCRIBE from the PDML, please visit the link directly above and follow the directions.
Re: formatting SD flash memory (rant) - was RE: Damaged images
On Mon, Mar 14, 2011 at 09:16:11PM -0400, Bruce Walker wrote: On 11-03-14 9:07 PM, steve harley wrote: of course reformatting frequently will increase the number of writes of those directory blocks, and thus wear those blocks out even faster Yeah, exactly, which is one reason why I don't habitually format; I just erase the images. And just how, exactly, does this result in less directory rewriting? -- PDML Pentax-Discuss Mail List PDML@pdml.net http://pdml.net/mailman/listinfo/pdml_pdml.net to UNSUBSCRIBE from the PDML, please visit the link directly above and follow the directions.
RE: formatting SD flash memory (rant) - was RE: Damaged images
Kris, I am somewhat confused now. What you say makes perfect sense except one detail. I thought that CF cards were those that had controller on board. The SD cards as I understand don't have controller on board. Therefore it makes certain sense (may be not too much sense, but still) to write to/format the card in the same controller (the camera). I am not sure if reading from the card can actually damage its contents... SD cards have controllers inside as well. This is hard data from an Apacer datasheet of 2005: The SD Memory Card includes an intelligent controller that manages interfaced protocols and data storage and retrieval as well as Error Correction Code (ECC) algorithms, defect handling and diagnostics, power management and Content Protection for Recordable Media related functions. So using SD cards you are essentially shielded from hardware failures of the storage by that controller. What it should be able to do is relocate data even for bad writes and even relocate file system records - the most write-intensive parts of the memory. Whether a card fails or not is directly dependent on how this controller operates and what failures it is programmed to circumvent. As all those safeguards potentially take processing time on the card, expect high-speed cheap cards to have suckier controllers that don't perform 100% on-the-fly checks. Now as this is a market economy we are talking about, there are bound to be cheaper less complicated (or just older) controllers out there that the cheaper cards use, with luck that should never be a problem for an end user but if that controller is slower and there are errors to be corrected and the camera cuts power to the card too quickly when powered off - anything could happen. I am using SSDs in all my computers and there the problem is way more pronounced because of the frequent random writes. It happens that a drive is put on market with defective firmware and because of the frequent writes the users see the problem already in a couple months. SSDs usually have user-upgradeable firmware that can at least partly solve the problems. SDs don't have user-upgradeable firmware so if you put a substandard card on the market users will probably start experiencing issues in a couple years - when nobody can do a thing about it. Another question I'd like to ask - how many read/write cycles there has to be made before a certain location on the card becomes flaky? I mean what is card's MTBF? You see, I still have that 1GB SD card (SanDisk) that I bought back in 2006 that still works. My empirical understanding is that several tens of thousands of read/writes don't have significant influence on the card performance. Controllers in modern cards are designed so as to balance writes across the card. That coupled with good error detection and correction routines should make the card last forever under normal load. That is assuming normal error rate. Could happen that the memory on the cards is produced from crappy materials or shipped with some obvious faults (like the first batch of K-5 sensors) - that makes this discussion a purely theoretical one, we have no knowledge of what quality materials are used for which cards Boris kris -- PDML Pentax-Discuss Mail List PDML@pdml.net http://pdml.net/mailman/listinfo/pdml_pdml.net to UNSUBSCRIBE from the PDML, please visit the link directly above and follow the directions.
Re: formatting SD flash memory (rant) - was RE: Damaged images
On 3/15/2011 11:29 AM, Krisjanis Linkevics wrote: SD cards have controllers inside as well. This is hard data from an Apacer datasheet of 2005: The SD Memory Card includes an intelligent controller that manages interfaced protocols and data storage and retrieval as well as Error Correction Code (ECC) algorithms, defect handling and diagnostics, power management and Content Protection for Recordable Media related functions. I stand corrected. Thanks for the clarification. Controllers in modern cards are designed so as to balance writes across the card. That coupled with good error detection and correction routines should make the card last forever under normal load. That is assuming normal error rate. Could happen that the memory on the cards is produced from crappy materials or shipped with some obvious faults (like the first batch of K-5 sensors) - that makes this discussion a purely theoretical one, we have no knowledge of what quality materials are used for which cards So, it effectively implies that using brand names such as SanDisk or Lexar is a good idea even if their cards cost somewhat more than those produced by second tier manufacturers... That is, it is equivalent to hope or belief that SanDisk has ability and inclination to invest in proper RD, QA and QC so as to roll out quality products to market... Boris -- PDML Pentax-Discuss Mail List PDML@pdml.net http://pdml.net/mailman/listinfo/pdml_pdml.net to UNSUBSCRIBE from the PDML, please visit the link directly above and follow the directions.
RE: formatting SD flash memory (rant) - was RE: Damaged images
Controllers in modern cards are designed so as to balance writes across the card. That coupled with good error detection and correction routines should make the card last forever under normal load. That is assuming normal error rate. Could happen that the memory on the cards is produced from crappy materials or shipped with some obvious faults (like the first batch of K-5 sensors) - that makes this discussion a purely theoretical one, we have no knowledge of what quality materials are used for which cards So, it effectively implies that using brand names such as SanDisk or Lexar is a good idea even if their cards cost somewhat more than those produced by second tier manufacturers... That is, it is equivalent to hope or belief that SanDisk has ability and inclination to invest in proper RD, QA and QC so as to roll out quality products to market... Boris We can hope :) The lifetime warranty some card manufacturers offer should be an indication that they really do try to provide a superior product but that doesn't necessarily mean that the same product is not available from other manufacturers cheaper - could be they are using the same components off the same production line anyway. kris -- PDML Pentax-Discuss Mail List PDML@pdml.net http://pdml.net/mailman/listinfo/pdml_pdml.net to UNSUBSCRIBE from the PDML, please visit the link directly above and follow the directions.
Re: formatting SD flash memory (rant) - was RE: Damaged images
On 11-03-14 4:25 PM, Krisjanis Linkevics wrote: But formatting may (although it's no guarantee) repair a glitch in an iffy card and let you continue using it. -bmw Some facts and common sense: SD memory is NAND flash which means the following things: 1) manufacturers ship it with errors already on the chip; 2) it deteriorates over time (with write/delete cycles); 3) it contains software and hardware to detect new errors on the fly and record information about the bad blocks; 4) said software isolates the card from the outside world - no program writes directly to the free space on the card. What formatting these does should be exactly nothing (or equivalent to deleting files) but formatting is done with a piece of software on the host computer (or camera) and therefore can introduce more writes/deletes than necessary. What people formatting cards usually think is that the whole card will be writeen full of zeros - which - although thankfully not the case - is actually what you don't want to do - ever (because of the wear and tear). Now whether cards become bad with time or not is mostly dependant on the hardware/software controller on the card itself - depending on how good it is at choosing places to write files and how good it is at marking the bad spots on the card - the card could either die very fast or live practically forever. So what card you buy really matters. Usage - whether you write large volumes of data or routinely format the card with a formatting tool that comes with your operating system and was never intended for formatting flash memory - comes second. With luck the controller on the card can deal with your formatting tool and fool it into thinking that the card has been formatted - although if we look at an empty card that you are formatting to the same file system it already has and with the same options, there should be no writes at all. So what this all comes down to is: if it aint broken, don't fix it. If it breaks, pray to god that you can actually do anything about it - chances are you can't - at least formatting can only stumble upon errors in a very limited space at the beginning of the card - and is very much like a blind man trying to walk into the only tree that's in the middle of a very large field. kris Kris, I wouldn't call that a rant, nor do I disagree with any of your facts and sense. One point though: when most modern OSes format a FAT block device, they simply create a new empty directory structure without touching anything else; the so-called Fast Format. Only a few writes are done, so not too hazardous to the overall device write allowance. Not too much different from del *.* in fact. Now if a bad block is discovered on your Flash device while trying to read the directory structure, I assume that the bad block will immediately be remapped, but the damage is already done. Not physically damaged of course, but files could appear to be unreadable, contain holes, etc. The fix is to format the card and that will replace the directory structure. Life goes on. -bmw -- PDML Pentax-Discuss Mail List PDML@pdml.net http://pdml.net/mailman/listinfo/pdml_pdml.net to UNSUBSCRIBE from the PDML, please visit the link directly above and follow the directions.
Re: formatting SD flash memory (rant) - was RE: Damaged images
On 2011-03-14 14:25 , Krisjanis Linkevics wrote: What formatting these does should be exactly nothing (or equivalent to deleting files) but formatting is done with a piece of software on the host computer (or camera) and therefore can introduce more writes/deletes than necessary. What people formatting cards usually think is that the whole card will be writeen full of zeros - which - although thankfully not the case - is actually what you don't want to do - ever (because of the wear and tear). i'm with you on this Kris; these days on hard disks as well formatting usually just clears the catalog unless you do a security erase; it's not like the low-level formatting of yore, and it is pretty unnecessary; i format only a few times a year, in the camera, which at a best case would clear any bad pointers in the catalog; i don't expect it to improve a card in any way -- PDML Pentax-Discuss Mail List PDML@pdml.net http://pdml.net/mailman/listinfo/pdml_pdml.net to UNSUBSCRIBE from the PDML, please visit the link directly above and follow the directions.
Re: formatting SD flash memory (rant) - was RE: Damaged images
On 2011-03-14 14:54 , Bruce Walker wrote: Now if a bad block is discovered on your Flash device while trying to read the directory structure, I assume that the bad block will immediately be remapped, but the damage is already done. Not physically damaged of course, but files could appear to be unreadable, contain holes, etc. The fix is to format the card and that will replace the directory structure. Life goes on. formatting won't fix a bad block, the image file that is thus damaged should just be deleted -- PDML Pentax-Discuss Mail List PDML@pdml.net http://pdml.net/mailman/listinfo/pdml_pdml.net to UNSUBSCRIBE from the PDML, please visit the link directly above and follow the directions.
RE: formatting SD flash memory (rant) - was RE: Damaged images
From: pdml-boun...@pdml.net [mailto:pdml-boun...@pdml.net] On Behalf Of Krisjanis Linkevics But formatting may (although it's no guarantee) repair a glitch in an iffy card and let you continue using it. -bmw Some facts and common sense: I'm sorry, but you're going to have to unsubscribe now. [...] -- PDML Pentax-Discuss Mail List PDML@pdml.net http://pdml.net/mailman/listinfo/pdml_pdml.net to UNSUBSCRIBE from the PDML, please visit the link directly above and follow the directions.
Re: formatting SD flash memory (rant) - was RE: Damaged images
On Mon, Mar 14, 2011 at 5:52 PM, Bob W p...@web-options.com wrote: From: pdml-boun...@pdml.net [mailto:pdml-boun...@pdml.net] On Behalf Of Krisjanis Linkevics But formatting may (although it's no guarantee) repair a glitch in an iffy card and let you continue using it. -bmw Some facts and common sense: I'm sorry, but you're going to have to unsubscribe now. If only I format my CF and SD cards after every download. I am still using CF cards i bought back in 2001-2002 when i bought my D1. No issues so far doing it , this way. Dave [...] -- PDML Pentax-Discuss Mail List PDML@pdml.net http://pdml.net/mailman/listinfo/pdml_pdml.net to UNSUBSCRIBE from the PDML, please visit the link directly above and follow the directions. -- Documenting Life in Rural Ontario. www.caughtinmotion.com http://brooksinthecountry.blogspot.com/ York Region, Ontario, Canada -- PDML Pentax-Discuss Mail List PDML@pdml.net http://pdml.net/mailman/listinfo/pdml_pdml.net to UNSUBSCRIBE from the PDML, please visit the link directly above and follow the directions.
Re: formatting SD flash memory (rant) - was RE: Damaged images
On 11-03-14 5:21 PM, steve harley wrote: On 2011-03-14 14:54 , Bruce Walker wrote: Now if a bad block is discovered on your Flash device while trying to read the directory structure, I assume that the bad block will immediately be remapped, but the damage is already done. Not physically damaged of course, but files could appear to be unreadable, contain holes, etc. The fix is to format the card and that will replace the directory structure. Life goes on. formatting won't fix a bad block, the image file that is thus damaged should just be deleted Steve, I didn't say that. I said that formatting would fix a screwed-up directory structure (and/or files) caused by their having been a bad block in the directory structure itself. The odds of that happening are relatively high because the directory structure is updated for very many file operations, like opening and closing them. Once you have a corrupted directory structure, formatting is about your only recourse. (Well there's chkdsk /f or the Mac's Diskutil, but we won't go there.) -bmw -- PDML Pentax-Discuss Mail List PDML@pdml.net http://pdml.net/mailman/listinfo/pdml_pdml.net to UNSUBSCRIBE from the PDML, please visit the link directly above and follow the directions.
Re: formatting SD flash memory (rant) - was RE: Damaged images
On 2011-03-14 18:37 , Bruce Walker wrote: On 11-03-14 5:21 PM, steve harley wrote: On 2011-03-14 14:54 , Bruce Walker wrote: Now if a bad block is discovered on your Flash device while trying to read the directory structure, I assume that the bad block will immediately be remapped, but the damage is already done. Not physically damaged of course, but files could appear to be unreadable, contain holes, etc. The fix is to format the card and that will replace the directory structure. Life goes on. formatting won't fix a bad block, the image file that is thus damaged should just be deleted Steve, I didn't say that. I said that formatting would fix a screwed-up directory structure (and/or files) caused by their having been a bad block in the directory structure itself. The odds of that happening are relatively high because the directory structure is updated for very many file operations, like opening and closing them. makes more sense when you say it that way -- it sounded like a different claim when i read contain holes (i.e. the remapped block is in a file) followed by the fix is to format -- yes, some blocks used for the directory are written fairly often (though not necessarily when opening a file) of course reformatting frequently will increase the number of writes of those directory blocks, and thus wear those blocks out even faster -- PDML Pentax-Discuss Mail List PDML@pdml.net http://pdml.net/mailman/listinfo/pdml_pdml.net to UNSUBSCRIBE from the PDML, please visit the link directly above and follow the directions.
Re: formatting SD flash memory (rant) - was RE: Damaged images
On 11-03-14 9:07 PM, steve harley wrote: On 2011-03-14 18:37 , Bruce Walker wrote: On 11-03-14 5:21 PM, steve harley wrote: On 2011-03-14 14:54 , Bruce Walker wrote: Now if a bad block is discovered on your Flash device while trying to read the directory structure, I assume that the bad block will immediately be remapped, but the damage is already done. Not physically damaged of course, but files could appear to be unreadable, contain holes, etc. The fix is to format the card and that will replace the directory structure. Life goes on. formatting won't fix a bad block, the image file that is thus damaged should just be deleted Steve, I didn't say that. I said that formatting would fix a screwed-up directory structure (and/or files) caused by their having been a bad block in the directory structure itself. The odds of that happening are relatively high because the directory structure is updated for very many file operations, like opening and closing them. makes more sense when you say it that way -- it sounded like a different claim when i read contain holes (i.e. the remapped block is in a file) followed by the fix is to format -- yes, some blocks used for the directory are written fairly often (though not necessarily when opening a file) of course reformatting frequently will increase the number of writes of those directory blocks, and thus wear those blocks out even faster Yeah, exactly, which is one reason why I don't habitually format; I just erase the images. But these cards are quite smart and vastly improved over earlier Flash tech. As blocks wear out they are remapped to other unused blocks. Over time the card just appears to have reduced capacity. So overall you'll get much better lifetime out of them than earlier generations would have provided. I don't think the 10K write cycles thing is much of a real issue for average users anymore. -bmw -- PDML Pentax-Discuss Mail List PDML@pdml.net http://pdml.net/mailman/listinfo/pdml_pdml.net to UNSUBSCRIBE from the PDML, please visit the link directly above and follow the directions.
