Hi All, Many of you have been involved in discussing this in meetings over the past week, but I wanted to send a summary email to ensure we have a clear shared understanding of what's happening, of why, and of what the next steps are from here.
Key points: - The Application Services team have decided to stop using Mentat for the user data components we are building in 2018. - We now plan to ship the initial versions of these components using SQLite for local data storage, while continuing to build them in Rust for cross-platform re-usability. - This decision was driven by a need to remove uncertainty around the shipping timeline for Fenix, and is not a reflection on the longer-term suitability of Mentat. This is a significant change in short-term strategy, but we're confident it's the right one - it increases our chances of delivering baseline functionality for Fenix by the end of the year, while still making incremental progress towards improving the Sync and Storage story across Mozilla's product ecosystem as a whole. For consumers of our upcoming Sync & Storage components, this change should not have any visible impact on you, except perhaps for the components getting into your hands more quickly. But please reach out if you have questions or concerns. I've included more details below in my favourite "hypothetical Q & A" format; if I appear to have gotten any of the details wrong, please correct me! I also think it would be useful to do a more explicit and deliberate retrospective on our first attempt to use Mentat for these components. I've started a doc here to gather some initial thoughts, please feel free to dive in with your own contributions: https://docs.google.com/document/d/1o17fCo7O1k4ZmCU3YTepW2vY5CdLolJ2CRgwumpn5mA/ An open offer for the Mentat team: would you like us to set up an explicit Retrospective meeting, to review our experiences over the last few months and to help inform next steps for Mentat development? Finally...Nick, Emily and Grisha, thank you for your hard work and support in pushing this Mentat experiment forward! It's disappointing to have to cut it short for a looming deadline, but I hope we can pick it up again in the future once the Application Services component story is more fully fleshed out. Cheers, Ryan -------------------------------- Why did we decide to stop using Mentat? We have a Q4-2018 deadline to deliver re-usable rust components for storing logins, history and bookmarks. There has been growing uncertainty about our ability to hit that deadline. A significant amount of that uncertainty has to do with how "ready" Mentat will be, and how ready we need it to be for the data in question: - On-disk storage size for history is much larger in Mentat than in the existing Places SQLite schema. How effectively can we control that size using Mentat's excision functionality? At what performance cost? How well will that interact with syncing? - History data "top sites" queries need to sort by aggregate fields, which is not yet implemented in Mentat. - History data queries will need Materialized Views for performance, which are not yet implemented in Mentat. - Bookmarks queries will need to transitively load tree-structured data, which it's not clear how to implement in Mentat's query language. - Bookmarks data needs to store items as ordered lists, which it's unclear how to do effectively in Mentat. These are all tractable problems, and are on the Mentat development roadmap. But combined with the fact that our team is still learning the basics of how to "think in Mentat", they represent too many unknowns too close to our delivery date. Why did we decide to switch to SQLite? In many ways SQLite is the de-facto choice for local data storage in Mozilla products. More importantly for our purposes, SQLite represents a lot of knowns: we have deep experience with using it, we understand its query capabilities and performance characteristics, we have existing implementations of history and bookmarks data from other Firefox products. This will let us more confidently plan the delivery timeline of our components as we approach the end-of-year deadline. Why now? Based on previous experience, we estimate that it will take several months to build preview-release-quality implementations of logins, history and bookmarks on top of SQLite. We're now several months away from our delivery deadline, making the decision basically now or never. What are the risks of changing to this new strategy? There are several, but they're better understood. We risk having left this decision too late, with insufficient time to execute it. As noted above, we feel like there's enough time to execute based on previous experience. Mentat was specifically designed to address known shortcomings in the SQLite-based storage approach taken by existing Firefox applications, such as difficult schema migrations and unreliable Sync support. We risk further entrenching those shortcomings by copying them into new applications. We'll mitigate this risk by consolidating applications on a single cross-platform implementation that's easier to change and maintain, and by applying some lessons from Mentat to the design of our component APIs. We risk slowly re-inventing large parts of Mentat in an ad-hoc way as we try to avoid the shortcomings of previous SQLite implementations. I'm not sure there's any mitigation for this apart from continuing to push forward on Mentat as a research project, with an eye to migrating over to it when it's ready. We are continuing with the plan to build sync and storage components in Rust, which carries its own set of risks and unknowns. We're feeling broadly confident that this plan will succeed in the time we have available, and switching storage layers does not alter that assessment. What happens to the Logins storage component, which is already using Mentat? Our current plan is to replace Mentat with SQLite for this component as well. Mentat appears to be working out fine for this specific data type, but there is little short-term value in having one component use a different storage layer than all the others, and removing Mentat will reduce the overall size and complexity of our suite of storage components. What about the Mentat Storage Server work we were planning for Q4? Let's put it on hold. Most of the ongoing work on Mentat's sync capabilities is client-side, and has been able to progress well atop the existing prototype Mentat storage server, which we will continue to run and maintain. This also means dropping our Q3 OKR around planning out a Mentat Storage Server, and focusing instead on the work to improve the existing Sync1.5 server infrastructure. What does this mean for Mentat longer-term? A great question! But I'm not the right person to answer it. Personally, I hope Mozilla will continue to invest in Mentat as a research project, without the aggressive timeline of building components for Fenix. Our short-term focus on building re-usable storage components and getting them in to new applications now, will potentially make it easier to migrate our product line to use Mentat in the future. Once our initial suite of Sync and Storage components are ready for release, I think it would be a valuable exercise to build a Mentat version of each of them and use it for a more direct evaluation of this new technology compared to raw SQLite.
_______________________________________________ Sync-dev mailing list [email protected] https://mail.mozilla.org/listinfo/sync-dev
