Re: Client Implementation - Milestone 1

2013-08-15 Thread Andreas Gal
Mostly just a +1. The plan is concise and incremental. It's exactly what we need. Nice work. I don't see any red flags to add FFOS along the way. We might volunteer as first post sync-1.1 storage backend consumer once that part shakes out after milestone 1. We will try to share the desktop HTML UI,

Re: (decision) Re: What does "Client Implementation" mean?

2013-08-13 Thread Andreas Gal
your logic breaks down in case of Persona as well. So how about we move on now for real instead of piling on any more claims that don't hold up. Andreas Lloyd Hilaiel wrote: On Aug 13, 2013, at 11:53 AM, Andreas Gal wrote: You seem to be implying that I am uncomfortable with proje

Re: (decision) Re: What does "Client Implementation" mean?

2013-08-13 Thread Andreas Gal
You seem to be implying that I am uncomfortable with projects using different tools. That is as unfair as it is inaccurate. I heavily use github, github issues and a whole range of tools that are not bugzilla and hg. Every project we do in research is on github. Every project I start that is

Re: Implementation approaches for Create Account / Sign In

2013-08-12 Thread Andreas Gal
When in doubt, lets do what we know how to do. We know how to write native UIs. We have used that a 1000 times before. We have one example of using a Web-hosted UI, but its not really similar at all. Engineering is finding simple solutions for complex problems, not fancy solutions for simple

Re: (decision) Re: Implementation approaches for Create Account / Sign In

2013-08-12 Thread Andreas Gal
Chris, we have a fairly established software engineering process for our Firefox products, that includes quality and security update mechanisms. We are trying to deliver a product on a compressed timeline here. Re-inventing the Mozilla software engineering process seems not like the fastest p

Re: (decision) Re: What does "Client Implementation" mean?

2013-08-12 Thread Andreas Gal
Lloyd Hilaiel wrote: On Aug 12, 2013, at 4:27 PM, Johnathan Nightingale mailto:john...@mozilla.com>> wrote: On Aug 9, 2013, at 9:11 PM, Mark Finkle wrote: My only strong opinions are: 1. Using bugzilla as the one source of truth for bugs. Even b2g had to do it. 2. ELM is the place where t

Re: The Sync 1.1 elephant^H^H^H^H^H^H option

2013-08-12 Thread Andreas Gal
Andreas: "and those cases we can handle with client-driven migration." Lloyd: it seems like client managed upgrade is the way to go here, " Sounds like we are in agreement here, no? Andreas ___ Sync-dev mailing list Sync-dev@mozilla.org https://mail

Re: The Sync 1.1 elephant^H^H^H^H^H^H option

2013-08-11 Thread Andreas Gal
Ryan Kelly wrote: On 12/08/2013 11:38 AM, Andreas Gal wrote: Ryan Kelly wrote: On 11/08/2013 4:36 PM, Andreas Gal wrote: once we went through one flag day and have the data stored in cleartext we can do arbitrary storage format and wire protocol format changes. Worst case we have to

Re: The Sync 1.1 elephant^H^H^H^H^H^H option

2013-08-11 Thread Andreas Gal
Ryan Kelly wrote: On 11/08/2013 4:36 PM, Andreas Gal wrote: once we went through one flag day and have the data stored in cleartext we can do arbitrary storage format and wire protocol format changes. Worst case we have to operate two services against the same data store (reving the wire

Re: The Sync 1.1 elephant^H^H^H^H^H^H option

2013-08-10 Thread Andreas Gal
ne of code reuse wherever possible. That way we can rev protocols and storage formats as needed, without unilaterally locking out clients or wiping data. (And our proposed storage protocols -- treesync and queuesync -- both address much of the durability-defeating aspects of both Sync 1.1 and 2.0.)

Re: The Sync 1.1 elephant^H^H^H^H^H^H option

2013-08-10 Thread Andreas Gal
I am having a hard time parsing your double negatives here to be honest. We can go the 1.1 route without two flag days. If you disagree, please speak up. If not, lets move on. Andreas Sent from Mobile. On Aug 10, 2013, at 17:04, Nick Alexander wrote: > On 13-08-09 9:55 PM, Andreas Gal wr

Re: The Sync 1.1 elephant^H^H^H^H^H^H option

2013-08-09 Thread Andreas Gal
I didn't say 0. I said that the 2 flag day argument is bogus. We will have 1 flag day. Do we agree now? Andreas Sent from Mobile. On Aug 9, 2013, at 21:45, Nick Alexander wrote: > On 13-08-09 9:33 PM, Andreas Gal wrote: >> Ok. So neither require a flag day. Changing data formats

Re: Implementation approaches for Create Account / Sign In

2013-08-09 Thread Andreas Gal
Please never ever use localStorage. It's evil sync legacy crap. Andreas Sent from Mobile. On Aug 9, 2013, at 21:33, Jedediah Parsons wrote: > > On this single point: > >> Technical question. IIUC, the Persona shim stores private user details >> in localStorage (and in persona.org cookies). >

Re: The Sync 1.1 elephant^H^H^H^H^H^H option

2013-08-09 Thread Andreas Gal
: > > On Aug 9, 2013, at 8:57 PM, Andreas Gal wrote: > >> Help me understand why we need a second flag day when changing wire >> formats or storage formats. > > Auth changeover wouldn't (necessarily) require a flag day. We could set the > old servers to accept

Re: The Sync 1.1 elephant^H^H^H^H^H^H option

2013-08-09 Thread Andreas Gal
Help me understand why we need a second flag day when changing wire formats or storage formats. Andreas Sent from Mobile. On Aug 9, 2013, at 18:02, Brian Warner wrote: > On 8/9/13 4:37 PM, Lloyd Hilaiel wrote: > >> So I'd like to open a venting thread here. Let's capture once and for >> all th

Re: Defining the goals for New Sync v1

2013-08-06 Thread Andreas Gal
I would suggest the wording "persistent and scalable" instead of "more reliable". Reliable is a broad term and a deep rabbit hole :) Andreas Sent from Mobile. On Aug 6, 2013, at 6:56, Deb Richardson wrote: > Fair enough. Thanks, Lloyd :) > > ~ d > > - Original Message - >> On Aug 1, 2

Re: representing bookmarks (was Re: queue-sync-over-CouchDB)

2013-07-30 Thread Andreas Gal
good shortcut. I don't see any advantage of Tumblers, except for being ... well pretty clever. Andreas Ryan Kelly wrote: On 31/07/2013 5:30 AM, Andreas Gal wrote: I think using /_bulk_docs and /_changes is pretty efficient if the data model puts each item (bookmark, password, etc) in a sep

Re: queue-sync-over-CouchDB

2013-07-30 Thread Andreas Gal
Mark Finkle wrote: I think we should be very careful to solve the problem we are trying to solve here, not a more general problem we don't care about. As a user, I won't be editing my bookmarks concurrently on two devices with any significant frequency. Even in those cases

Re: queue-sync-over-CouchDB

2013-07-30 Thread Andreas Gal
I think using /_bulk_docs and /_changes is pretty efficient if the data model puts each item (bookmark, password, etc) in a separate CouchDB document. There's clearly a spectrum here: * one-document DB, with one giant JSON blob containing all e.g. bookmarks. (I think https://wiki.mozilla.org/U

Re: queue-sync-over-CouchDB

2013-07-30 Thread Andreas Gal
Not having to carry pouchdb on the client side is definitely tempting. Also, as we discussed earlier, CouchDB's replication algorithm is not a perfect fit for our star-shaped nodes. Its meant for a more interconnected graph. A single outgoing changes queue avoids carrying more history on the

Re: straw man proposal for a data model for CouchDB-based sync

2013-07-29 Thread Andreas Gal
Mark Finkle wrote: I like the level of detail in the doc. You didn't include Tabs as a data type, but I think it's pretty easy to infer a structure based on the definition of the other types. I have a few questions / comments: The MVP product specs didn't talk about other data types. But yes,

straw man proposal for a data model for CouchDB-based sync

2013-07-28 Thread Andreas Gal
https://wiki.mozilla.org/User:Gal/SyncDataModel The title is a bit misleading. The data model is closely tied to the protocol and a conflict avoidance mechanism (push notifications), which are discussed as well. Please poke holes in this. For small changes/corrections that you think are non

Re: whats the key anyway?

2013-07-27 Thread Andreas Gal
Sent from Mobile. On Jul 27, 2013, at 7:17, Mark Finkle wrote: -- There is so much wrong with your analysis that I don't know where to start. Fortunately, its also so far out of scope that I don't have to. Your perspective is appreciated, but we have made certain

Re: whats the key anyway?

2013-07-27 Thread Andreas Gal
Sent from Mobile. On Jul 27, 2013, at 1:02, Gavin Sharp wrote: > On Sat, Jul 27, 2013 at 12:45 AM, Andreas Gal wrote: >>> As a user, I expect that setting up Sync on my phone doesn't delete the data >>> on my desktop or phone, instead merging everything together in

Re: whats the key anyway?

2013-07-27 Thread Andreas Gal
Richard Newman wrote: Concurrent modifications happen (both from user input and due to user activity), and a system not specifically designed to address this (and races, and offlining, and other kinds of timing/conflict issues) will *at sufficient usage* suffer from a background hum of user

Re: whats the key anyway?

2013-07-26 Thread Andreas Gal
oach. Iterate! Until then all eyes should be on the ball: make us no longer be DFL. I agree that _needless_ complexity is our enemy. But we should be aware of the consequences of apparently easy simplifications. I am happy to entertain any concrete examples where we should add more complexity, but p

Re: whats the key anyway?

2013-07-26 Thread Andreas Gal
was good enough, when > it manifestly was not, at any level of abstraction. We should not be trading > the very basics of correctness for the possibility of a closer ship date. > > - Original Message - > From: "Andreas Gal" > To: "Richard Newman" >

Re: whats the key anyway?

2013-07-26 Thread Andreas Gal
There are many different ways to do this. And it simply doesn't matter which one we chose. Losing a usage count is irrelevant. As long we are reasonably counting, that's perfectly fine. Complexity is our biggest enemy here. Precision is not a design goal. Fast delivery of a solution that reasonably

Re: whats the key anyway?

2013-07-26 Thread Andreas Gal
Sent from Mobile. On Jul 26, 2013, at 17:16, Richard Newman wrote: >> stupid. Instead, if we define the key (_id) to be "httpRealm" and >> "formSubmitURL|usernameField|passwordField" for the two password types, > > Hashed, of course. > >> we get conflict resolution for free by couchdb. It will m

Re: Passive Sign-In to PiCL

2013-07-26 Thread Andreas Gal
I think the flow could look like this: Firefox version X with new sync starts for the first time. A doorhanger comes down and asks "Do you want to store your browser profile with Mozilla? Yes/No." If Yes, we create an account for you, and locally in your browser store an auto-generated user

Re: remote couchdb vs shadow database vs one database

2013-07-26 Thread Andreas Gal
Nobody is asking to move all core services. We are talking about history, bookmarks and passwords, for now. Andreas Mark Finkle wrote: I did some experiments with pouchdb inside Gecko, using passwords and bo

Re: Passive Sign-In to PiCL

2013-07-26 Thread Andreas Gal
Mark Finkle wrote: I worry about this approach in that Firefox does not know my Facebook password unless I ask Firefox to save it. We have your facebook cookie, which is equivalent to identify you. We are not storing anything extra here. As a UA, we simply have this info. Andreas Even then,

Re: remote couchdb vs shadow database vs one database

2013-07-26 Thread Andreas Gal
Can you share? Lloyd Hilaiel wrote: On Jul 26, 2013, at 1:58 PM, Andreas Gal <mailto:andreas@gmail.com>> wrote: If I remember correctly, the average user has 500 passwords. Thats not a big deal compressed. Bookmarks is 500-ish as well. History 5000-ish. We can easily limit hi

Re: remote couchdb vs shadow database vs one database

2013-07-26 Thread Andreas Gal
ators. Should we give this a spin? Andreas Gavin Sharp wrote: On Fri, Jul 26, 2013 at 12:39 PM, Andreas Gal wrote: You answered your own question. If I store passwords in a pouchdb replacing the current storage (which is very desirable and would greatly reduce complexity), and I am i

Re: remote couchdb vs shadow database vs one database

2013-07-26 Thread Andreas Gal
async backends. I can spin the event loop or I can block on pouchdb on a different thread. Both approaches suck. Sent from Mobile. On Jul 26, 2013, at 12:05, Gavin Sharp wrote: > On Fri, Jul 26, 2013 at 11:53 AM, Andreas Gal wrote: >> The main problem we are running into here >&

Re: whats the key anyway?

2013-07-26 Thread Andreas Gal
obile. On Jul 26, 2013, at 12:06, Lloyd Hilaiel wrote: On Jul 26, 2013, at 1:03 PM, Andreas Gal wrote: In short, what I heard yesterday ("lets copy data in case of conflict") is a noble theory, but I am afraid wrong in practice, and I would like to hear comments on the observation above

whats the key anyway?

2013-07-26 Thread Andreas Gal
Couchdb uses uuids as keys for documents, but we can also chose our own. Choosing the right key is critical to leverage couchdb's replication protocol well for conflict resolution. Lets look at passwords. If we use a uuid as password key (_id), then if you add the same password on two devices

history

2013-07-26 Thread Andreas Gal
The CouchDB protocol has some properties where I am not convinced that its a good fit for us. Amongst others, each document has an unlimited history that is tracked locally. This is actually needed for multi-party replication. I would like to understand how we will deal with this. Lets say we

remote couchdb vs shadow database vs one database

2013-07-26 Thread Andreas Gal
I did some experiments with pouchdb inside Gecko, using passwords and bookmarks as my two main use cases. As Dale pointed out, if we use a shadow database and then replicate the shadow database, we also have to replicate from the current storage backends (password, bookmark places, history pl