The O'Reilly Palm Programming book has a nice chapter on what needs to happen during a sync. After reading David's book below, it looks pretty much like what O'Reilly describes should happen.
I included some rambling comments below. David A. Desrosiers wrote: >>Does the Palm store details of which fields have changed over and above >>the simple setting of the "dirty" flag on a modified record ? >> > > As you probably know, there are two ways of sync'ing, Fast and Slow. > (and Judd can probably help here also): > > 1. Fast syncing is when the same Palm syncronizes to the same > desktop or laptop or networkd system all the time. Most people > believe this is the "norm". In this situation, you can reasonably > assume that all of the proper modify/archive/delete bits on the > Palm are accurate. > > What's required: > > a. Iterate through the desktop data with a local dektop-resident > conduit which looks at a locally-stored instance of the Palm > data. > > b. Iterate through the Palm data. For each changed record on the > Palm, the conduit should be treated as follows: > > i. Archived records should be added to a queue or local database > on the desktop and mark it as a pending delete on the desktop, > then delete the record from the Palm > > ii. Deleted records are marked as a pending delete in the desktop > database and the record is removed from the Palm. It's > important to note that records aren't actually "deleted" on > the Palm until a sync occurs, even though you may not see them > through the Palm UI any longer, they're still there until a > sync removes them. > > iii. Modified records on the Palm are modified locally in the > desktop database/storage structure (in our case, XML for this > example) > > iv. New records create on the Palm are simply created on the > desktop side as well at sync time > > c. Compare the local data with the remote (Palm) data > > i. Archived records are removed from the Palm and puts the entry > in an "Archived" database/structure on the desktop and marks > it as a pending delete on the desktop > > ii. Deleted records are removed from the Palm and marked as a > pending delete in the desktop database/structure > > iii. Modified records will replicate the modifications to the Palm > and then clear the modification flag from the record in the > desktop database/structure > > iv. New records will have the contents copied to the Palm and then > should clear the 'Added' flag from the same record in the > desktop database/structure > > At this point, any records marked for pending deletion or deletion > are purged from the desktop data and both Palm and desktop should match, > record-for-record. Once the modifications are confirmed, all deletes acted > upon, and new records replicated in the right direction, the temporary > storage is then copied over to "permanent" storage. The existing local data > prior to this commit is stored as well, and is used for a Slow Sync. > > 2. Slow syncing is what happens when one Palm is syncronized with > more than one system or desktop. One example is syncronizing your > Palm with your work computer, then coming home and syncronizing > it with your home computer. > > In this situation, the modify/archive/delete flags are not going > to be useful to the second system you sync to, because there is > no reference there to validate them. In order to find out what > was changed, despite what the Palm data says, we need to have a > delta between current data and "last snapshot" of that data. The > tertiary database we copied back after doing s "Fast Sync" above > can be used for this. This should give us an accurate picture of > the last time the Palm and the desktop matched exactly. In the > time since this last sync with this desktop, the data on either > desktop or Palm may have changed. All of the bits in the desktop > data will still remain relevant, but the bits in the Palm data > are no longer useful, since they would have been > changed/updated/etc. when that Palm was sync'd to desktop_2. > > To do this properly, the desktop conduit must read each record on > the Palm into memory or a structure for comparison against that > tertiary "backup" database that we saved during our Fast Sync. > Palm, Inc. calls this the "remote database". Let's use 'rdb' for > short here (where 'ldb' would stand for 'local database'). > > What's required: > > a. Records in rdb and Palm match, no changes are made to either > unit. > > b. The rdb has no record, but the Palm does. This record is > considered "new" and marked as modified in the rdb. > > c. The backup record is missing from the rdb, the record in the > rdb was deleted or archived, and the record is marked as > deleted. > > d. The backup record and the record in the rdb are not the same, > with the record in the rdb having been modified. The record is > marked as "changed" on the Palm. > > There are a few situations where this gets ugly. > > a. A record is deleted on the Palm and deleted on the desktop (or > vice versa). What we should do is remove the deleted record and > the changed record is copied to the opposite device. > > b. A record is archived in one database and changed in the other. > The archived version of the record is put into the archive db > (adb) and the changed copy is copied to the other device. > > c. A record is archived in one device, and deleted on the other. In > this case, we should just put the record into the adb. > > d. Records are changed in both places (desktop and Palm), but > changed differently in exactly the same location. In this case, > you'd end up with two records, each full copies of the other with > the addition of the "discrepant" information. > > e. A record is changed identically in both locations. Ideally this > should result in only one record existing in both locations, an > exact duplicate of the other. > > The "Slow Sync" is much slower because every record has to be copied > to the desktop with the conduit and compared to the data found there when it > arrives. > > >>Whilst I can see how a DESKTOP app might allow for the setting of a >>dirty flag as an attribute at the "field" level, in the case you >>describe above, how would you know (flag) which field had changed on the >>Palm ? >> > > The comparison has to happen at the desktop when doing a slow sync > as above. Fast sync in this case wouldn't require atomic syncs to happen. > > >>Unless you keep a clean copy of the last-sync'd (clean) record, as well >>as the Desktop changes, you have no way to tell what has changed >>(rather, compute the sum of the changes). >> > > You have to keep a clean copy, or else sync becomes useless. > > >>Consider the case where you change the same piece of info (field) on >>both the Desktop app and the Palm but the changes which are made are not >>the same. Dont you need the equivalent of transaction logs on both sides >>of the conduit to work out the complete set of changes ? >> > > You only need Palm data, desktop data, and "archive" data for this > comparison to derive a delta between the "Last Known Good" sync and the one > currently being enqueued. > > >>So, I guess my underlying question is: is there any other info that can >>be used to identify records and/or changes to them in the way you >>suggest: maybe I missed something ? >> > > I'm still thinking through this. Neal and some other people seem to > think we may be able to get a bit more async with these devices, and the > results I've seen with similar devices using similar ports and protocols > alludes to this. It would speed up things slightly. However, this can even > be made faster at the desktop level by a simple "pre-parse" at sync time. > When you initiate a sync, the local copies of the data are "diffed" for > differences, and the delta between *THEM* is what the Palm data is compared > against. Let me chew on this a for a bit, I'm sure I'm missing some glaring > hole in the design here. > I'm not sure if I know what you are saying, but the palm only provides record level atomicity. I don't think there is a way to just change part of a record. Not during the sync anyway. I think a pre-parse at sync time will slow things down. There will be a hang time when the desktop is creating the delta file and no communication is going on between the devices. Where as doing it inline you have serial/USB buffers filling up at the same time as you are reading the local data and calculating the next move. > Can anyone else find some holes in this approach so far? > There are no ways to sync prc files correctly. There are no ways to sync categories correctly. The app info block can't be synced. The sort block can't be synced. Corect me if I'm wrong on any of this. I would recomend keeping the last synced database and then keeping a delta file instead of a current desktop database. That way the delta file is always there and ready to go even before the sync button is pressed. Then if you need to keep a current local DB to be compatible with something then do that also, it won't hurt. You could do some funky things with data transaction logging. Suppose Joe changes his phone extention from x123 to x144 and you change it on the palm and the desktop. 1 record would result after a sync if you were to sync after that. No sync takes place though. Then he changes it again to x145 and you only change it on the desktop. Data changes are logged in the delta file. A usual sync would create 2 records on the palm and on the desktop because the correct extension is not known. Here is how a logged data change sync would work. Data before sync: Palm: Joe x144 Desktop: Joe x123 with changes in desktop delta DB (x144 and x145). Data after palm record changes are made to the desktop. Palm: Joe x144 Desktop: Joe x144 Desktop delta change x144 is applied to desktop db and since records match the palm, nothing more is required. Palm: Joe x144 Desktop: Joe x144 Desktop delta change x145 is applied to desktop db and the palm DB. Palm: Joe x145 Desktop: Joe x145 We're left with one record instead of two and it is correct. There are all kinds of scenarios and I don't think they are worth writing the logic for personally. I say in collisions, just log it and let the human sort it out. Judd _______________________________________________ Pilot-unix mailing list [EMAIL PROTECTED] http://hcirisc.cs.binghamton.edu/mailman/listinfo/pilot-unix
