Thanks Richard, I'll dig into this more next week. On Fri, Sep 2, 2016 at 6:03 PM, Richard Newman <rnew...@mozilla.com> wrote:
> Is there documentation on best practices for making a new data type that >>> will eventually be synced? I'm trying to figure out a schema and storage >>> backend for Form Autofill profiles (on desktop to start) and though we're >>> not implementing the sync engine initially, I would like the data to be >>> ready for sync without significant re-implemention of the storage. >>> >> >> I'm not aware of a link to such documentation. >> > > Me neither, sadly. There's a lot of accumulated knowledge in mailing > lists, wikis, and bugs… but digging it up takes more time than just writing > it here! > > > You're correct. A GUID as the primary key is preferred, since you'll need >> to track a (remote) GUID anyway. >> > > To expand on this: > > Sync records have an identifier. This is preferably a short (12-char) > Places-style GUID, but some older engines use UUIDs instead. The identifier > is its name on the server, so it's cleartext — for this reason we don't > just use URLs or straightforward hashes of URLs. > > In some cases you might wish to store a salt somewhere and hash URLs — no > need to store a GUID. > > In some circumstances you'll need to replace your local GUID with one from > the server. > > In addition to Nick's notes about best practices/thoughts, here are some > (not necessarily aimed at autofill): > > > - You'll need a way to decide at the moment of syncing which local > records have changed. The desktop engines typically watch observer > notifications, but this has drawbacks — it's hard to spot exactly what > changed (move? rename?), and we tend to do crazy things like not send all > the notifications if too many items change. On Android we use timestamps, > which is wrong, but so it goes. On iOS (and soon on desktop for bookmarks!) > we directly keep track of Sync-related changes as storage operations hit > the disk, which is ideal. I've discussed this at some length if you're > interested > <https://160.twinql.com/syncing-and-storage-on-three-platforms/>. > - You'll need to figure out how your feature works with Refresh > Firefox, and what Sync is supposed to do in that case. > - You'll have to decide whether your feature: > - Gets a checkbox of its own in Preferences and in the FxA setup > flow. Form History, for example, reuses History's checkbox. > - Is enabled by default. (See existing bugs > <https://bugzilla.mozilla.org/show_bug.cgi?id=608231>.) > - Syncs on desktop, iOS, and/or Android. > - Document your object format > <http://docs.services.mozilla.com/sync/objectformats.html>. I can help > with that. > - You should give some thought to how the object format might change > over time, and whether it's acceptable to lock out clients who don't > support the new format. This is a topic of its own. > - Consider writing TPS tests. The rest of the Sync team can help with > that. > - Involve the server and ops folks in your design process. Records > that change very frequently, are very large, or both, are not friendly > to sync <https://bugzilla.mozilla.org/show_bug.cgi?id=1222594>. > - Consider whether Firefox Sync itself is a good candidate for syncing > your data type. Sync is good for: > - Strong crypto > - Relatively small and infrequent changes > - Non-realtime > - Independent records > - Less than 25MB total > - No sharing between users. > > so, for example, we use Kinto on desktop and Android for a bunch of stuff > that doesn't require that collection of properties. > > Thanks for more details on best practices for the engine implementation but, to be clear, there are no short term plans to implement the sync engine, the focus is on the storage for the feature itself but I'm hoping to make it friendly to writing the sync engine later. Regardless, these notes will be useful for that future time when we're implementing the engine. As for best practices, the number one thing is to remember that Sync is an >> object store with no good way to maintain references between objects. >> Therefore, if you can keep records independent of each other, you will have >> a *much* better time. As an example, history records are independent, but >> bookmark records are not: bookmark folders reference bookmark items (and >> vice-versa -- parents!) and no end of pain ensues trying to keep references >> coherent. >> > > I can't agree with this enough. Reading the design notes for iOS bookmark > sync <https://gist.github.com/rnewman/f95720050df3d5515351>, or talking > to markh and Kit, should be enough to scare you away from interdependent > records. > > > Storage implementations don't have to be complicated on Android; neither >> do Sync engine implementations. (Once you've decided how to reconcile >> conflicts, of course!) >> > > Indeed: it's simpler to write a safe Java JSON-backed store and make it > accessible to Sync and to Gecko than it is to access a Gecko-owned sqlite > database. > > > The storage layer exists, but it's not currently used. The motivation is >> multi-fold. First, we see many PW storage failures on Android. That is, >> the DB is corrupt, crypto failed, whatever. Second, we see concurrency >> problems, performance loss, and memory waste due to the required additional >> process for Gecko's SQLite compatibility. And finally, we'd get closer to >> severing the Fennec front-end from Gecko if the store was independent of >> Gecko. (In general, we've learned never to use Gecko if we want a good >> Android experience.) >> > > It's also the only sane way to support Sync + a master password equivalent > on Android. > > > Happy to elaborate in more detail; it would help for you to post your >> suggested schema and Desktop storage proposal. >> > > Likewise. Thanks for involving sync-dev early in the process! >
_______________________________________________ Sync-dev mailing list Sync-dev@mozilla.org https://mail.mozilla.org/listinfo/sync-dev
