Re: [Tinyos-help] More TEP 103 questions
On Tuesday 29 August 2006 10:05 am, David Gay wrote: On 8/28/06, R. Steve McKown [EMAIL PROTECTED] wrote: I'd love to see the spec guarantee that read returns valid data. Are you envisioning a read returns a len of 0 on bad data or end of log? This seems to fit nicely with the implementation of at45db, where bad data on a page and end of log conditions are generally synonymous. I'm simply envisioning that it's guaranteed (well, within the bounds provided by simple error detection codes) not to return invalid data. It's actual behaviour on invalid data would be left unspecified (skip, think it's end of log, whatever - we might want to guarantee that it does return at least, though). Yeah, I asked an impl ? in a spec context. Sorry about that. What you are suggesting should would work well for our expected uses of log storage. A thought: You earlier introduced the idea of having appendDone deliver indication of data lost due to a circular log wrap. A similar change to readDone could deliver indication of data lost to corruption. Thanks, Steve ___ Tinyos-help mailing list Tinyos-help@Millennium.Berkeley.EDU https://mail.millennium.berkeley.edu/cgi-bin/mailman/listinfo/tinyos-help
Re: [Tinyos-help] More TEP 103 questions
On 8/28/06, R. Steve McKown [EMAIL PROTECTED] wrote: So shouldn't then TEP 103 specify data integrity requirements? I can't imagine a use of log storage where it'd be ok to read back different data than I'd originally appended. And if that's true, wouldn't data integrity be more efficient close to the physical storage mechanisms where is particular peculiarities and benefits be accommodated than out in the application? Currently, TEP103 specifies that crashes and reboots don't cause data loss, and that writes are atomic. Offhand, I'd say ensuring this requires some kind of CRC, so adding some data-corruption detection guarantee (well, within the limits that CRCs provide) probably wouldn't be too bad. However, providing guaranteed data corruption detection is trickier (see below). How about something along these lines: - seek() returns a new error code if the app attempts to seek into an expected valid area of the log but cannot do so. Perhaps some implementations might have a metadata structure that could cause this condition. If this condition happens, seek should automatically forward to the first whole record in the log that it can identify. The error return from seek() allows the application to know that data are unavailable at the desired seek location. The app could issue another seek() if it's doing its own blocking, or go ahead and read if the records defined by append operations are suitable. - read(), on SUCCESS, provides a reasonable guarantee that the data originally appended are read back without corruption. Read should return a new error code if the data to be returned resides in an area of the flash that is known corrupt. The integrity check presumably happens during a read operation, optimized according to implementation (i.e. only once per flash page, etc). The read pointer should automatically move to the next whole record just past the known corrupt area. The application can read again or do a seek. The at45db implementation will currently notice when data is invalid. However, it tends to believe that this means it's at the end of a log (because an unintialised page, or a failed page write will give you something with an invalid CRC). Detecting the different between failure and data corruption would require extra complexity, I think (especially given a circular log). A reasonable compromise might be that read is guaranteed to return valid data. But recovering data from a corrupted log (i.e., not from crash/reboot style failure) is something which is probably best left to off-mote tools, I think. David Gay ___ Tinyos-help mailing list Tinyos-help@Millennium.Berkeley.EDU https://mail.millennium.berkeley.edu/cgi-bin/mailman/listinfo/tinyos-help
Re: [Tinyos-help] More TEP 103 questions
On Monday 28 August 2006 16:53, David Gay wrote: On 8/28/06, R. Steve McKown [EMAIL PROTECTED] wrote: So shouldn't then TEP 103 specify data integrity requirements? [snip] [snip] A reasonable compromise might be that read is guaranteed to return valid data. I'd love to see the spec guarantee that read returns valid data. Are you envisioning a read returns a len of 0 on bad data or end of log? This seems to fit nicely with the implementation of at45db, where bad data on a page and end of log conditions are generally synonymous. The specification addition read is guaranteed to return valid data, as you consider above, would afford plenty of implementation flexibility. The stm25p impl, which uses virtual addresses and can identify end of log separately from a bad block, could silently skip bad data and continue at the next block in the log, if one exists. I believe the existing record atomicity requirements would apply to skipping bad data (bad data is record-atomic); if so, the loss of records due to bad data should impact an application no more than loss of records due to appending new data to a full circular log. But recovering data from a corrupted log (i.e., not from crash/reboot style failure) is something which is probably best left to off-mote tools, I think. For the types of applications we'll develop, if the mote can't read some data from a log, the value of that data would never be worth enough to take the time and effort to extract it using other methods. I presume, using the mechanisms we're discussing, that an app could monitor error conditions from log IO operations to decide if it should try to erase the log and continue, stop and whine, etc. All the best, Steve ___ Tinyos-help mailing list Tinyos-help@Millennium.Berkeley.EDU https://mail.millennium.berkeley.edu/cgi-bin/mailman/listinfo/tinyos-help
Re: [Tinyos-help] More TEP 103 questions
On 8/25/06, R. Steve McKown [EMAIL PROTECTED] wrote: Hi, It seems the only method supported by TEP 103 to determine an empty LogStorage log is to perform a read and receive 0 bytes in return. Is this a correct assessment? Hmm, yes. I can't find where TEP 103 specifies the initial read position of a log, such as after reboot but before the first call to LogRead.read(). The closest I can find is from the first para of 3.3: Such logging should be reliable (a mote crash should not lose data). Is this a strong enough pronouncement to indicate that implementations must explicitly set an initial read position to the start of the oldest record in the log? The oldest record was my intent at least. It sounds like we forgot to say that... (will fix) What's the best practice for initially configuring the various Block/Log/Config storage components' volumes, especially in a moderate volume production setting? I'm currently thinking a separate program needs to do intitial setup before the main application is installed. I think it's too new for their to be a best practice yet... We assumed you would consider your application's needs, flash chip characteristics (erase size, etc), and pick accordingly (each kind of flash chip has a separate configuration file for volumes). David Gay ___ Tinyos-help mailing list Tinyos-help@Millennium.Berkeley.EDU https://mail.millennium.berkeley.edu/cgi-bin/mailman/listinfo/tinyos-help
Re: [Tinyos-help] More TEP 103 questions
On Friday 25 August 2006 09:59 am, David Gay wrote: On 8/25/06, R. Steve McKown [EMAIL PROTECTED] wrote: What's the best practice for initially configuring the various Block/Log/Config storage components' volumes, especially in a moderate volume production setting? I'm currently thinking a separate program needs to do intitial setup before the main application is installed. I think it's too new for their to be a best practice yet... We assumed you would consider your application's needs, flash chip characteristics (erase size, etc), and pick accordingly (each kind of flash chip has a separate configuration file for volumes). I'm thinking more about initializing flash volumes for the various storage abstractions. Initialization would be an issue for new motes with factory fresh flash, or motes which have been reprogrammed after a change to the volume organization (volumes-impl.xml file). For sake of argument, I'm going to call these conditions invalid volume, relative to the storage abstraction which is (to be) using the volume. If I read TEP 103 correctly, BlockStorageC and ConfigStorageC can detect an invalid volume, using BlockRead.verify and ConfigStorage.valid (after mount()), respectively. Is this correct? I can't find a similar mechanism for LogStorageC; is there one? TEP 103 doesn't say, but I'm assuming if LogStorageC has an invalid volume, LogWrite.erase() must first be called? Thanks for your patience. Steve ___ Tinyos-help mailing list Tinyos-help@Millennium.Berkeley.EDU https://mail.millennium.berkeley.edu/cgi-bin/mailman/listinfo/tinyos-help
Re: [Tinyos-help] More TEP 103 questions
On 8/25/06, R. Steve McKown [EMAIL PROTECTED] wrote: I'm thinking more about initializing flash volumes for the various storage abstractions. Initialization would be an issue for new motes with factory fresh flash, or motes which have been reprogrammed after a change to the volume organization (volumes-impl.xml file). For sake of argument, I'm going to call these conditions invalid volume, relative to the storage abstraction which is (to be) using the volume. If I read TEP 103 correctly, BlockStorageC and ConfigStorageC can detect an invalid volume, using BlockRead.verify and ConfigStorage.valid (after mount()), respectively. Is this correct? Yes. I can't find a similar mechanism for LogStorageC; is there one? TEP 103 doesn't say, but I'm assuming if LogStorageC has an invalid volume, LogWrite.erase() must first be called? Yes, erase must be called first. We spent some time wondering about explicit or implicit mount operations - explicit mount gives a good place to indicate invalid volumes, but I believe we ended up w/o it for the reason outlined below. The issue with specifying invalid volume detection is that it's probably impractical (impossible?) to do w/o checking the whole volume (*): imagine the following scenario: - the mote had a log in blocks 0 - 100 - some later use puts some random data in block 55 - the most recent use again uses blocks 0 - 100 for a log There's no obvious way to notice that random data short of checking all the blocks. FWIW, the at45db log storage implementation returns FAIL when it has problems locating the end of the log on the first operation. But the lack of a FAIL return doesn't guarantee anything if the log's blocks have been used for non-log purposes (it's doing a binary search for the end of the log, so may easily miss weird blocks...). David Gay *: note that this is what BlockStorageC.verify and ConfigStorageC.valid end up doing. I guess that attempting to read the whole log would be one way of detecting problems - if you get any failures, then the log is invalid and must be erased... ___ Tinyos-help mailing list Tinyos-help@Millennium.Berkeley.EDU https://mail.millennium.berkeley.edu/cgi-bin/mailman/listinfo/tinyos-help
