Re: [Standards] Proposed XMPP Extension: Roster Versioning
Dave Cridland <[EMAIL PROTECTED]> writes: >> But that's neither here nor there. The question is whether: >> >> (1) acking an occasional roster push from the server to the client >> (where BTW the server *does* know your full JID) is a serious >> problem >> that we need to solve because it wastes large amounts of bandwidth >> >> > It's not the bandwidth, it's the additional transmission I'm more > concerned with. For every stanza, there's an addition TCP data > packet, and therefore there'll be a further ACK - both of which will > cost power. > > The alternative is that we quietly explain to people that they needn't > bother replying to the push, which I really don't like. I'm > pretty sure that the mobile developers won't want to reply to every > roster push, anyway. I think this is a problem that has already been solved in practice - if the server doesn't include an id attribute in the roster push (despite what RFC 3920 says), the client won't send a reply, being liberal about what it accepts. At least, that's what jabber.el does after I encountered that problem. (And I promptly reported a bug against that server--I think it was jabberd2--and it was fixed ☺) -- Magnus JID: [EMAIL PROTECTED]
Re: [Standards] Proposed XMPP Extension: Roster Versioning
Dave Cridland wrote: > On Mon Mar 10 23:45:28 2008, Peter Saint-Andre wrote: >> One potential problem: this is not a nice, small, incremental change. >> Now servers and clients must support: >> >> 1. The 'sequence' attribute. >> 2. Roster pushes via message, not IQ. >> 3. Multiple items in a roster push. >> 4. Multiple items in a roster set. >> >> The more we change, the less likely it is that clients and servers will >> add this feature. Then we're back where we started. >> >> "If it ain't broke, don't fix it." >> >> So what's broken? >> >> 1. Huge roster gets every time you log in. The 'sequence' stuff fixes >> this. >> >> What's not broken? >> >> 2. Roster pushes via IQ. >> 3. One item per roster push. >> 4. One item per roster set. >> >> Why are we fixing 2, 3, and 4? Just because we can? Because it is more >> elegant? Or because it really solves a big problem on the network? >> >> Unless there is a compelling need, I'd vote for changing as little as >> possible. > > The problem is that if you go for Joe's concept - which is certainly > elegant - of having the roster "diff" simply sent as a bunch of pushes, > then these - being sent as - need to be acknowledged. Now, I > appreciate that clients don't always do this, but the fact is that they > should be doing so. I'm unhappy with suggesting that clients quietly > ignore RFC 3920 because nobody cares. > > So we can fix that simply by using instead of - it's a > pretty trivial change, and it eliminates the type='result' reply for > clients on all pushes, so it's inarguably a performance improvement. But why are we fixing this? In the real world, you will receive 2 or 3 roster pushes when you log in. Why are we optimizing for this? Perhaps all this talk of optimizing is wrong -- it makes a lot more sense IMHO to work toward satisficing. Receiving and acking 2 or 3 roster pushes is not that big a deal. I say let's ignore that and move on to bigger problems. > Next up, Joe introduces a problem in as much as the client can't tell > when it's finished receiving updates, and is now receiving bona-fide > pushes. Joe says this isn't a problem, but I'm unconvinced - client > authors have said this is a niggle, at least, for the initial presence > push, after all. I don't think this is a problem, but if it is then I think it's easily solve -- include the latest sequence number in the roster result and when you get that push you know you're up to date. > We can easily avoid this by having multiple items in a push - again, > this cuts down on overhead, and so is a performance improvement. A minor one. Not worth working on IMHO. > These two are simply taking Joe's concept - the update-as-push - and > cleaning it up, getting the protocol better. Needless optimization, IMHO. Sequences + pushes is good enough. > Now... On to point 4. This isn't really needed, and it's not solving a > big problem. But it's not a huge addition, and it does gain us a bit, > and - perhaps most importantly - it fits neatly into this spec, and is > too small to be independent. See above. Peter -- Peter Saint-Andre https://stpeter.im/ smime.p7s Description: S/MIME Cryptographic Signature
Re: [Standards] Proposed XMPP Extension: Roster Versioning
On Tue Mar 18 02:51:53 2008, Peter Saint-Andre wrote: Dave Cridland wrote: > It's not the bandwidth, it's the additional transmission I'm more > concerned with. For every stanza, there's an addition TCP data > packet, and therefore there'll be a further ACK - both of which will > cost power. > > The alternative is that we quietly explain to people that they needn't > bother replying to the push, which I really don't like. I'm pretty > sure that the mobile developers won't want to reply to every roster > push, anyway. You make a point. I note you don't say if it's a good one or not. :-) I got my start in the Jabber community as the documentation guy. As a result, I've always seen it as my job to document the protocols as they are used in the community. In that sense I am a conservative, so I don't immediately think "gosh how much can we change things?" but "how little can we change to get our desired functionality?" In my opinion that's one reason our community has stuck together rather than pulled apart (with different projects and companies defining their own extensions). Changing core functionality in a significant way scares me because the feedback I typically receive is that it scares our developer community. Maybe I'm being unduly conservative here, but it seems to me that we legitimately might want to have roster sequencing without changing the core behavior of roster pushes. The converse problem is that for every extension which changes existing behaviour, rather than adding - and that's clearly the case here - you add a code-path. So each extension *of this type* needs to do as much as possible, to minimize having, at some later point in the future, a scenario where there are 4 or 5 extensions covering various optimizations surrounding rosters, and therefore 24 or 120 different combinations. Each of those combinations needs testing, and while it's possible for us to reduce them by making them mandate each other, there's a risk of ending up with theoretical server capabilities that we simply cannot test. So yes, I agree, there's no point in making such dramatic changes as to put developers off, but at the same time, we don't want to build 120 different roster behaviours. 2 is quite enough. To summarize the discussion so far, roster sequencing enables the client to know if the roster has not changed (and this is what saves us the greatest amount of bandwidth). Beyond that I see several possible bundles, from smallest change from current behavior to greatest change from current behavior... 1. If the client provides a sequence number on login and the roster has changed since that sequence number, the client receives one old-fashioned roster push (IQ-set) for each item that has been added, modified, or removed. Subsequent roster changes are sent via old-fashioned roster pushes (IQ-set). 2. If the client provides a sequence number on login and the roster has changed since that sequence number, the client receives a "diff" -- that is, a full set of the items that have been added, modified, or removed -- in one IQ set from the server to the client (and this diff may include multiple items). Subsequent roster changes are sent via old-fashioned roster pushes (IQ-set), with one item per push. Both the above can be rolled into one extension easily - that's what the roster sequence validity/identifier thing I vaguely suggested was for. As Joe suggested, though, the diff works fine as a set of old-fashioned roster-pushes. 3. If the client provides a sequence number on login and the roster has changed since that sequence number, the client receives a "diff" -- that is, a full set of the items that have been added, modified, or removed -- in one from the server to the client (and this diff may include multiple items). Subsequent roster changes are sent via stanzas, which may include multiple items. Well, this is two changes: 1) Sending multiple roster-pushes in one stanza. 2) Sending roster-push stanzas as instead of . Is that an accurate summary? Are there other options on the table? Let's get clear on the options so that we can figure out which one is the best way forward. I think this is basically accurate, but there is another way of looking at it - there are four extensions on the table; we can pick 0..4 of them: 1) Maintain an indentifier for a particular roster state, such that a client needn't download the roster afresh unless it's changed. 2) Maintain an identifier for a particular roster state, such that a client may only download the outstanding pushes. 3) Allow multiple pushes. 4) Put pushes in stanzas. Now, I think that 1 & 2 are fairly easily merged into one, but 3 & 4 are pretty much independent. I think they're also pretty low-hanging fruit for servers and clients alike - the tougher part is getting 1 and/or 2 right. Dave. -- Dave Cridland - mai
Re: [Standards] Proposed XMPP Extension: Roster Versioning
Dave Cridland wrote: > On Mon Mar 17 16:31:06 2008, Peter Saint-Andre wrote: >> >> But that's neither here nor there. The question is whether: >> >> (1) acking an occasional roster push from the server to the client >> (where BTW the server *does* know your full JID) is a serious problem >> that we need to solve because it wastes large amounts of bandwidth >> >> > It's not the bandwidth, it's the additional transmission I'm more > concerned with. For every stanza, there's an addition TCP data > packet, and therefore there'll be a further ACK - both of which will > cost power. > > The alternative is that we quietly explain to people that they needn't > bother replying to the push, which I really don't like. I'm pretty > sure that the mobile developers won't want to reply to every roster > push, anyway. You make a point. >> or >> >> (2) sending roster pushes via is a pretty optimization that >> is more elegant than what we developed in 1999, but it fundamentally >> unnecessary >> >> I think (2) obtains. Therefore I think it's just fine to keep IQs for >> roster pushes. If it ain't broke, don't fix it. > > Nothing here is necessary; the question is whether we can, and should, > do it. > > Remember, rosters are not broken right now - they work just fine. So, if > we're going to change things, we may as well consider how much we can > change things, rather than how little. I got my start in the Jabber community as the documentation guy. As a result, I've always seen it as my job to document the protocols as they are used in the community. In that sense I am a conservative, so I don't immediately think "gosh how much can we change things?" but "how little can we change to get our desired functionality?" In my opinion that's one reason our community has stuck together rather than pulled apart (with different projects and companies defining their own extensions). Changing core functionality in a significant way scares me because the feedback I typically receive is that it scares our developer community. Maybe I'm being unduly conservative here, but it seems to me that we legitimately might want to have roster sequencing without changing the core behavior of roster pushes. To summarize the discussion so far, roster sequencing enables the client to know if the roster has not changed (and this is what saves us the greatest amount of bandwidth). Beyond that I see several possible bundles, from smallest change from current behavior to greatest change from current behavior... 1. If the client provides a sequence number on login and the roster has changed since that sequence number, the client receives one old-fashioned roster push (IQ-set) for each item that has been added, modified, or removed. Subsequent roster changes are sent via old-fashioned roster pushes (IQ-set). 2. If the client provides a sequence number on login and the roster has changed since that sequence number, the client receives a "diff" -- that is, a full set of the items that have been added, modified, or removed -- in one IQ set from the server to the client (and this diff may include multiple items). Subsequent roster changes are sent via old-fashioned roster pushes (IQ-set), with one item per push. 3. If the client provides a sequence number on login and the roster has changed since that sequence number, the client receives a "diff" -- that is, a full set of the items that have been added, modified, or removed -- in one from the server to the client (and this diff may include multiple items). Subsequent roster changes are sent via stanzas, which may include multiple items. Is that an accurate summary? Are there other options on the table? Let's get clear on the options so that we can figure out which one is the best way forward. Peter -- Peter Saint-Andre https://stpeter.im/ smime.p7s Description: S/MIME Cryptographic Signature
Re: [Standards] Proposed XMPP Extension: Roster Versioning
Pedro Melo wrote: > Hi, > > On Mar 17, 2008, at 10:11 PM, Peter Saint-Andre wrote: >> Pedro Melo wrote: >>> On Mar 17, 2008, at 3:22 PM, Peter Saint-Andre wrote: Pedro Melo wrote: > On Mar 11, 2008, at 12:06 AM, Dave Cridland wrote: >> On Mon Mar 10 23:45:28 2008, Peter Saint-Andre wrote: >>> One potential problem: this is not a nice, small, incremental >>> change. >>> Now servers and clients must support: >>> 1. The 'sequence' attribute. >>> 2. Roster pushes via message, not IQ. >>> 3. Multiple items in a roster push. >>> 4. Multiple items in a roster set. >>> The more we change, the less likely it is that clients and servers >>> will >>> add this feature. Then we're back where we started. >>> "If it ain't broke, don't fix it." >>> So what's broken? >>> 1. Huge roster gets every time you log in. The 'sequence' stuff >>> fixes >>> this. >>> What's not broken? >>> 2. Roster pushes via IQ. >>> 3. One item per roster push. >>> 4. One item per roster set. >>> Why are we fixing 2, 3, and 4? Just because we can? Because it is >>> more >>> elegant? Or because it really solves a big problem on the network? >>> Unless there is a compelling need, I'd vote for changing as >>> little as >>> possible. >> >> The problem is that if you go for Joe's concept - which is certainly >> elegant - of having the roster "diff" simply sent as a bunch of >> pushes, then these - being sent as - need to be acknowledged. >> Now, I appreciate that clients don't always do this, but the fact is >> that they should be doing so. I'm unhappy with suggesting that >> clients >> quietly ignore RFC 3920 because nobody cares. >> >> So we can fix that simply by using instead of - it's >> a pretty trivial change, and it eliminates the type='result' reply >> for >> clients on all pushes, so it's inarguably a performance improvement. > > Ok, I must have been missing something because if you want to fix > something, why not just do multiple items per roster push? Because it's never been done that way. > The argument that diff is complex doesn't play well with me. Each item > is a replacement item, so no diff required. The usual code for a > client > "foreach item update my model" will work with both. But clients have never expected multiple items in a roster push. However, that might be something clients could move to if they support sequencing. My question is: will clients do that? >>> >>> Yes. Basically if I'm a client and I'm telling the server that yes, I >>> support sequencing and I want to use it, the server is OK to change >>> previous rules if they are specified in the new XEP. >> >> Yes, I know that client developers *could* add this support, but my >> question is: *will* they really do so or will they see this spec and say >> "well this roster sequencing stuff looks nice but &@#$&?^!@ I need to >> modify my roster handling code and I haven't touched that in 3 years, >> what the heck is the XSF thinking?!?!" > > Yes, I understand that. > > The whole point about this was saving bandwidth. True. I contend that not receiving the main roster on login is the major savings. If a few roster pushes dribble in after you login, it's no big deal in the grand scheme of things (after all, how many people log in with multiple clients and experience multiple roster changes while they are offline with a given client?). > The current method of a > single item per push would waste a bit in the IQ replies. A bit, yes. But we don't have to save every last little byte. Let's differentiate between what's nice and what's necessary. > The > alternatives suggested where re-using messages to deliver roster pushes, > or ignore the IQ replies with a new IQ type "fire-and-forget" (not the > final type of course :) ) or bundle up multiple roster items in a single > push. I might have missed another approach... > > Of all this, I think the multiple items per push is the least intrusive. > But if even that is a problem, we could always make it an option: when > we tell the server that we want to use roster sync, tell him that we > support multiple items per push also. Don't get me wrong: I'm not deeply opposed to multiple items in a roster push. If we bundle that feature with roster synchronization then we'll probably be OK. I just wanted to make sure we performed a sanity check with our wonderful developer community before we moved ahead. :) Peter -- Peter Saint-Andre https://stpeter.im/ smime.p7s Description: S/MIME Cryptographic Signature
Re: [Standards] Proposed XMPP Extension: Roster Versioning
On Mon Mar 17 16:31:06 2008, Peter Saint-Andre wrote: Dave Cridland wrote: > On Mon Mar 17 15:22:28 2008, Peter Saint-Andre wrote: >> Pedro Melo wrote: >> > Lets not abuse the meaning of just because we like their >> > network properties, like we abused in the past because it >> > broadcasts well. >> >> +1. >> >> > Absolutely. So I'm looking forward to seeing the update to XEP-0060 > which changes PubSub pushes from being to , in (for > example) section 7.1.2.1. > > Or, alternately, could someone explain to me why the use of > stanzas for the 7.1.2.1 case is not "abuse", whereas would be. Looks to > me like they're the same use-case. Very funny. :P We use messages there in part because using IQs would require knowing the full JID (and stock pubsub services do not know that). Gosh - you mean you chose because of its network properties? ;-) But that's neither here nor there. The question is whether: (1) acking an occasional roster push from the server to the client (where BTW the server *does* know your full JID) is a serious problem that we need to solve because it wastes large amounts of bandwidth It's not the bandwidth, it's the additional transmission I'm more concerned with. For every stanza, there's an addition TCP data packet, and therefore there'll be a further ACK - both of which will cost power. The alternative is that we quietly explain to people that they needn't bother replying to the push, which I really don't like. I'm pretty sure that the mobile developers won't want to reply to every roster push, anyway. or (2) sending roster pushes via is a pretty optimization that is more elegant than what we developed in 1999, but it fundamentally unnecessary I think (2) obtains. Therefore I think it's just fine to keep IQs for roster pushes. If it ain't broke, don't fix it. Nothing here is necessary; the question is whether we can, and should, do it. Remember, rosters are not broken right now - they work just fine. So, if we're going to change things, we may as well consider how much we can change things, rather than how little. Dave. -- Dave Cridland - mailto:[EMAIL PROTECTED] - xmpp:[EMAIL PROTECTED] - acap://acap.dave.cridland.net/byowner/user/dwd/bookmarks/ - http://dave.cridland.net/ Infotrope Polymer - ACAP, IMAP, ESMTP, and Lemonade
Re: [Standards] Proposed XMPP Extension: Roster Versioning
Hi, On Mar 17, 2008, at 10:11 PM, Peter Saint-Andre wrote: Pedro Melo wrote: On Mar 17, 2008, at 3:22 PM, Peter Saint-Andre wrote: Pedro Melo wrote: On Mar 11, 2008, at 12:06 AM, Dave Cridland wrote: On Mon Mar 10 23:45:28 2008, Peter Saint-Andre wrote: One potential problem: this is not a nice, small, incremental change. Now servers and clients must support: 1. The 'sequence' attribute. 2. Roster pushes via message, not IQ. 3. Multiple items in a roster push. 4. Multiple items in a roster set. The more we change, the less likely it is that clients and servers will add this feature. Then we're back where we started. "If it ain't broke, don't fix it." So what's broken? 1. Huge roster gets every time you log in. The 'sequence' stuff fixes this. What's not broken? 2. Roster pushes via IQ. 3. One item per roster push. 4. One item per roster set. Why are we fixing 2, 3, and 4? Just because we can? Because it is more elegant? Or because it really solves a big problem on the network? Unless there is a compelling need, I'd vote for changing as little as possible. The problem is that if you go for Joe's concept - which is certainly elegant - of having the roster "diff" simply sent as a bunch of pushes, then these - being sent as - need to be acknowledged. Now, I appreciate that clients don't always do this, but the fact is that they should be doing so. I'm unhappy with suggesting that clients quietly ignore RFC 3920 because nobody cares. So we can fix that simply by using instead of - it's a pretty trivial change, and it eliminates the type='result' reply for clients on all pushes, so it's inarguably a performance improvement. Ok, I must have been missing something because if you want to fix something, why not just do multiple items per roster push? Because it's never been done that way. The argument that diff is complex doesn't play well with me. Each item is a replacement item, so no diff required. The usual code for a client "foreach item update my model" will work with both. But clients have never expected multiple items in a roster push. However, that might be something clients could move to if they support sequencing. My question is: will clients do that? Yes. Basically if I'm a client and I'm telling the server that yes, I support sequencing and I want to use it, the server is OK to change previous rules if they are specified in the new XEP. Yes, I know that client developers *could* add this support, but my question is: *will* they really do so or will they see this spec and say "well this roster sequencing stuff looks nice but &@#$&?^!@ I need to modify my roster handling code and I haven't touched that in 3 years, what the heck is the XSF thinking?!?!" Yes, I understand that. The whole point about this was saving bandwidth. The current method of a single item per push would waste a bit in the IQ replies. The alternatives suggested where re-using messages to deliver roster pushes, or ignore the IQ replies with a new IQ type "fire-and- forget" (not the final type of course :) ) or bundle up multiple roster items in a single push. I might have missed another approach... Of all this, I think the multiple items per push is the least intrusive. But if even that is a problem, we could always make it an option: when we tell the server that we want to use roster sync, tell him that we support multiple items per push also. Best regards, -- Pedro Melo Blog: http://www.simplicidade.org/notes/ XMPP ID: [EMAIL PROTECTED] Use XMPP!
Re: [Standards] Proposed XMPP Extension: Roster Versioning
Pedro Melo wrote: > Hi, > > On Mar 17, 2008, at 3:22 PM, Peter Saint-Andre wrote: >> Pedro Melo wrote: >>> On Mar 11, 2008, at 12:06 AM, Dave Cridland wrote: On Mon Mar 10 23:45:28 2008, Peter Saint-Andre wrote: > One potential problem: this is not a nice, small, incremental change. > Now servers and clients must support: > 1. The 'sequence' attribute. > 2. Roster pushes via message, not IQ. > 3. Multiple items in a roster push. > 4. Multiple items in a roster set. > The more we change, the less likely it is that clients and servers > will > add this feature. Then we're back where we started. > "If it ain't broke, don't fix it." > So what's broken? > 1. Huge roster gets every time you log in. The 'sequence' stuff fixes > this. > What's not broken? > 2. Roster pushes via IQ. > 3. One item per roster push. > 4. One item per roster set. > Why are we fixing 2, 3, and 4? Just because we can? Because it is more > elegant? Or because it really solves a big problem on the network? > Unless there is a compelling need, I'd vote for changing as little as > possible. The problem is that if you go for Joe's concept - which is certainly elegant - of having the roster "diff" simply sent as a bunch of pushes, then these - being sent as - need to be acknowledged. Now, I appreciate that clients don't always do this, but the fact is that they should be doing so. I'm unhappy with suggesting that clients quietly ignore RFC 3920 because nobody cares. So we can fix that simply by using instead of - it's a pretty trivial change, and it eliminates the type='result' reply for clients on all pushes, so it's inarguably a performance improvement. >>> >>> Ok, I must have been missing something because if you want to fix >>> something, why not just do multiple items per roster push? >> >> Because it's never been done that way. >> >>> The argument that diff is complex doesn't play well with me. Each item >>> is a replacement item, so no diff required. The usual code for a client >>> "foreach item update my model" will work with both. >> >> But clients have never expected multiple items in a roster push. >> However, that might be something clients could move to if they support >> sequencing. My question is: will clients do that? > > Yes. Basically if I'm a client and I'm telling the server that yes, I > support sequencing and I want to use it, the server is OK to change > previous rules if they are specified in the new XEP. Yes, I know that client developers *could* add this support, but my question is: *will* they really do so or will they see this spec and say "well this roster sequencing stuff looks nice but &@#$&?^!@ I need to modify my roster handling code and I haven't touched that in 3 years, what the heck is the XSF thinking?!?!" smime.p7s Description: S/MIME Cryptographic Signature
Re: [Standards] Proposed XMPP Extension: Roster Versioning
Hi, On Mar 17, 2008, at 3:22 PM, Peter Saint-Andre wrote: Pedro Melo wrote: On Mar 11, 2008, at 12:06 AM, Dave Cridland wrote: On Mon Mar 10 23:45:28 2008, Peter Saint-Andre wrote: One potential problem: this is not a nice, small, incremental change. Now servers and clients must support: 1. The 'sequence' attribute. 2. Roster pushes via message, not IQ. 3. Multiple items in a roster push. 4. Multiple items in a roster set. The more we change, the less likely it is that clients and servers will add this feature. Then we're back where we started. "If it ain't broke, don't fix it." So what's broken? 1. Huge roster gets every time you log in. The 'sequence' stuff fixes this. What's not broken? 2. Roster pushes via IQ. 3. One item per roster push. 4. One item per roster set. Why are we fixing 2, 3, and 4? Just because we can? Because it is more elegant? Or because it really solves a big problem on the network? Unless there is a compelling need, I'd vote for changing as little as possible. The problem is that if you go for Joe's concept - which is certainly elegant - of having the roster "diff" simply sent as a bunch of pushes, then these - being sent as - need to be acknowledged. Now, I appreciate that clients don't always do this, but the fact is that they should be doing so. I'm unhappy with suggesting that clients quietly ignore RFC 3920 because nobody cares. So we can fix that simply by using instead of - it's a pretty trivial change, and it eliminates the type='result' reply for clients on all pushes, so it's inarguably a performance improvement. Ok, I must have been missing something because if you want to fix something, why not just do multiple items per roster push? Because it's never been done that way. The argument that diff is complex doesn't play well with me. Each item is a replacement item, so no diff required. The usual code for a client "foreach item update my model" will work with both. But clients have never expected multiple items in a roster push. However, that might be something clients could move to if they support sequencing. My question is: will clients do that? Yes. Basically if I'm a client and I'm telling the server that yes, I support sequencing and I want to use it, the server is OK to change previous rules if they are specified in the new XEP. Best regards, -- Pedro Melo Blog: http://www.simplicidade.org/notes/ XMPP ID: [EMAIL PROTECTED] Use XMPP!
Re: [Standards] Proposed XMPP Extension: Roster Versioning
Richard Dobson wrote: > >> Very funny. :P >> >> We use messages there in part because using IQs would require knowing >> the full JID (and stock pubsub services do not know that). >> >> But that's neither here nor there. The question is whether: >> >> (1) acking an occasional roster push from the server to the client >> (where BTW the server *does* know your full JID) is a serious problem >> that we need to solve because it wastes large amounts of bandwidth >> >> or >> >> (2) sending roster pushes via is a pretty optimization that >> is more elegant than what we developed in 1999, but it fundamentally >> unnecessary >> >> I think (2) obtains. Therefore I think it's just fine to keep IQs for >> roster pushes. If it ain't broke, don't fix it. >> >> Peter >> > > Also surely if you are doing roster sequencing I would have thought it > would be important for the server to know if you have received the push > or not in order to keep things in sync correctly? So the ack is actually > quite useful. I'm not sure if the server absolutely needs to know if you have received the push because in general I think the server will just send out all the pushes without blocking on receipt of an ack. However, I can imagine cases where the server might want to receive the acks for some kind of tracking purpose. Peter -- Peter Saint-Andre https://stpeter.im/ smime.p7s Description: S/MIME Cryptographic Signature
Re: [Standards] Proposed XMPP Extension: Roster Versioning
Very funny. :P We use messages there in part because using IQs would require knowing the full JID (and stock pubsub services do not know that). But that's neither here nor there. The question is whether: (1) acking an occasional roster push from the server to the client (where BTW the server *does* know your full JID) is a serious problem that we need to solve because it wastes large amounts of bandwidth or (2) sending roster pushes via is a pretty optimization that is more elegant than what we developed in 1999, but it fundamentally unnecessary I think (2) obtains. Therefore I think it's just fine to keep IQs for roster pushes. If it ain't broke, don't fix it. Peter Also surely if you are doing roster sequencing I would have thought it would be important for the server to know if you have received the push or not in order to keep things in sync correctly? So the ack is actually quite useful. Richard
Re: [Standards] Proposed XMPP Extension: Roster Versioning
Dave Cridland wrote: > On Mon Mar 17 15:22:28 2008, Peter Saint-Andre wrote: >> Pedro Melo wrote: >> > Lets not abuse the meaning of just because we like their >> > network properties, like we abused in the past because it >> > broadcasts well. >> >> +1. >> >> > Absolutely. So I'm looking forward to seeing the update to XEP-0060 > which changes PubSub pushes from being to , in (for > example) section 7.1.2.1. > > Or, alternately, could someone explain to me why the use of > stanzas for the 7.1.2.1 case is not "abuse", whereas would be. Looks to > me like they're the same use-case. Very funny. :P We use messages there in part because using IQs would require knowing the full JID (and stock pubsub services do not know that). But that's neither here nor there. The question is whether: (1) acking an occasional roster push from the server to the client (where BTW the server *does* know your full JID) is a serious problem that we need to solve because it wastes large amounts of bandwidth or (2) sending roster pushes via is a pretty optimization that is more elegant than what we developed in 1999, but it fundamentally unnecessary I think (2) obtains. Therefore I think it's just fine to keep IQs for roster pushes. If it ain't broke, don't fix it. Peter smime.p7s Description: S/MIME Cryptographic Signature
Re: [Standards] Proposed XMPP Extension: Roster Versioning
On Mon Mar 17 15:22:28 2008, Peter Saint-Andre wrote: Pedro Melo wrote: > Lets not abuse the meaning of just because we like their > network properties, like we abused in the past because it > broadcasts well. +1. Absolutely. So I'm looking forward to seeing the update to XEP-0060 which changes PubSub pushes from being to , in (for example) section 7.1.2.1. Or, alternately, could someone explain to me why the use of stanzas for the 7.1.2.1 case is not "abuse", whereas would be. Looks to me like they're the same use-case. Dave. -- Dave Cridland - mailto:[EMAIL PROTECTED] - xmpp:[EMAIL PROTECTED] - acap://acap.dave.cridland.net/byowner/user/dwd/bookmarks/ - http://dave.cridland.net/ Infotrope Polymer - ACAP, IMAP, ESMTP, and Lemonade
Re: [Standards] Proposed XMPP Extension: Roster Versioning
Pedro Melo wrote: > Hi, > > On Mar 11, 2008, at 12:06 AM, Dave Cridland wrote: >> On Mon Mar 10 23:45:28 2008, Peter Saint-Andre wrote: >>> One potential problem: this is not a nice, small, incremental change. >>> Now servers and clients must support: >>> 1. The 'sequence' attribute. >>> 2. Roster pushes via message, not IQ. >>> 3. Multiple items in a roster push. >>> 4. Multiple items in a roster set. >>> The more we change, the less likely it is that clients and servers will >>> add this feature. Then we're back where we started. >>> "If it ain't broke, don't fix it." >>> So what's broken? >>> 1. Huge roster gets every time you log in. The 'sequence' stuff fixes >>> this. >>> What's not broken? >>> 2. Roster pushes via IQ. >>> 3. One item per roster push. >>> 4. One item per roster set. >>> Why are we fixing 2, 3, and 4? Just because we can? Because it is more >>> elegant? Or because it really solves a big problem on the network? >>> Unless there is a compelling need, I'd vote for changing as little as >>> possible. >> >> The problem is that if you go for Joe's concept - which is certainly >> elegant - of having the roster "diff" simply sent as a bunch of >> pushes, then these - being sent as - need to be acknowledged. >> Now, I appreciate that clients don't always do this, but the fact is >> that they should be doing so. I'm unhappy with suggesting that clients >> quietly ignore RFC 3920 because nobody cares. >> >> So we can fix that simply by using instead of - it's >> a pretty trivial change, and it eliminates the type='result' reply for >> clients on all pushes, so it's inarguably a performance improvement. > > Ok, I must have been missing something because if you want to fix > something, why not just do multiple items per roster push? Because it's never been done that way. > The argument that diff is complex doesn't play well with me. Each item > is a replacement item, so no diff required. The usual code for a client > "foreach item update my model" will work with both. But clients have never expected multiple items in a roster push. However, that might be something clients could move to if they support sequencing. My question is: will clients do that? > So "abusing" message just so that you don't need to reply seems wrong. > If we really need result-less IQs then make them so: allow for type="notify"> or something and be done with it. And make it > broadcast-able to all connected resources. Yes, it needs to be in > bis-work, but then again, if you don't advertise that you want > incremental rosters, you won't get them. All is fair if you pre-declare... > > Lets not abuse the meaning of just because we like their > network properties, like we abused in the past because it > broadcasts well. +1. >> Next up, Joe introduces a problem in as much as the client can't tell >> when it's finished receiving updates, and is now receiving bona-fide >> pushes. Joe says this isn't a problem, but I'm unconvinced - client >> authors have said this is a niggle, at least, for the initial presence >> push, after all. > > We lived with not knowing when the presence stream finishes, and that > was acceptable because presence is really a distributed concept so it is > impossible to know when other servers will respond. > > Rosters on the other hand are purely local to the domain concepts, so it > should be easy to say when they are over. > > On the other hand, I agree with Joe: our clients already have to deal > with roster pushes at any point in time, so I don't see the a problem here. Agreed. Peter -- Peter Saint-Andre https://stpeter.im/ smime.p7s Description: S/MIME Cryptographic Signature
Re: [Standards] Proposed XMPP Extension: Roster Versioning
Hi, On Mar 11, 2008, at 12:06 AM, Dave Cridland wrote: On Mon Mar 10 23:45:28 2008, Peter Saint-Andre wrote: One potential problem: this is not a nice, small, incremental change. Now servers and clients must support: 1. The 'sequence' attribute. 2. Roster pushes via message, not IQ. 3. Multiple items in a roster push. 4. Multiple items in a roster set. The more we change, the less likely it is that clients and servers will add this feature. Then we're back where we started. "If it ain't broke, don't fix it." So what's broken? 1. Huge roster gets every time you log in. The 'sequence' stuff fixes this. What's not broken? 2. Roster pushes via IQ. 3. One item per roster push. 4. One item per roster set. Why are we fixing 2, 3, and 4? Just because we can? Because it is more elegant? Or because it really solves a big problem on the network? Unless there is a compelling need, I'd vote for changing as little as possible. The problem is that if you go for Joe's concept - which is certainly elegant - of having the roster "diff" simply sent as a bunch of pushes, then these - being sent as - need to be acknowledged. Now, I appreciate that clients don't always do this, but the fact is that they should be doing so. I'm unhappy with suggesting that clients quietly ignore RFC 3920 because nobody cares. So we can fix that simply by using instead of - it's a pretty trivial change, and it eliminates the type='result' reply for clients on all pushes, so it's inarguably a performance improvement. Ok, I must have been missing something because if you want to fix something, why not just do multiple items per roster push? The argument that diff is complex doesn't play well with me. Each item is a replacement item, so no diff required. The usual code for a client "foreach item update my model" will work with both. ... So "abusing" message just so that you don't need to reply seems wrong. If we really need result-less IQs then make them so: allow for or something and be done with it. And make it broadcast-able to all connected resources. Yes, it needs to be in bis- work, but then again, if you don't advertise that you want incremental rosters, you won't get them. All is fair if you pre- declare... Lets not abuse the meaning of just because we like their network properties, like we abused in the past because it broadcasts well. Next up, Joe introduces a problem in as much as the client can't tell when it's finished receiving updates, and is now receiving bona-fide pushes. Joe says this isn't a problem, but I'm unconvinced - client authors have said this is a niggle, at least, for the initial presence push, after all. We lived with not knowing when the presence stream finishes, and that was acceptable because presence is really a distributed concept so it is impossible to know when other servers will respond. Rosters on the other hand are purely local to the domain concepts, so it should be easy to say when they are over. On the other hand, I agree with Joe: our clients already have to deal with roster pushes at any point in time, so I don't see the a problem here. Best regards, -- Pedro Melo Blog: http://www.simplicidade.org/notes/ XMPP ID: [EMAIL PROTECTED] Use XMPP!
Re: [Standards] Proposed XMPP Extension: Roster Versioning
Dave Cridland wrote: > On Mon Mar 10 23:45:28 2008, Peter Saint-Andre wrote: >> One potential problem: this is not a nice, small, incremental change. >> Now servers and clients must support: >> >> 1. The 'sequence' attribute. >> 2. Roster pushes via message, not IQ. >> 3. Multiple items in a roster push. >> 4. Multiple items in a roster set. >> >> The more we change, the less likely it is that clients and servers will >> add this feature. Then we're back where we started. >> >> "If it ain't broke, don't fix it." >> >> So what's broken? >> >> 1. Huge roster gets every time you log in. The 'sequence' stuff fixes >> this. >> >> What's not broken? >> >> 2. Roster pushes via IQ. >> 3. One item per roster push. >> 4. One item per roster set. >> >> Why are we fixing 2, 3, and 4? Just because we can? Because it is more >> elegant? Or because it really solves a big problem on the network? >> >> Unless there is a compelling need, I'd vote for changing as little as >> possible. > > The problem is that if you go for Joe's concept - which is certainly > elegant - of having the roster "diff" simply sent as a bunch of pushes, > then these - being sent as - need to be acknowledged. Now, I > appreciate that clients don't always do this, but the fact is that they > should be doing so. I'm unhappy with suggesting that clients quietly > ignore RFC 3920 because nobody cares. > > So we can fix that simply by using instead of - it's a > pretty trivial change, and it eliminates the type='result' reply for > clients on all pushes, so it's inarguably a performance improvement. > > Next up, Joe introduces a problem in as much as the client can't tell > when it's finished receiving updates, and is now receiving bona-fide > pushes. Joe says this isn't a problem, but I'm unconvinced - client > authors have said this is a niggle, at least, for the initial presence > push, after all. > > We can easily avoid this by having multiple items in a push - again, > this cuts down on overhead, and so is a performance improvement. > > These two are simply taking Joe's concept - the update-as-push - and > cleaning it up, getting the protocol better. > > Now... On to point 4. This isn't really needed, and it's not solving a > big problem. But it's not a huge addition, and it does gain us a bit, > and - perhaps most importantly - it fits neatly into this spec, and is > too small to be independent. The way you present things, it all sounds very reasonable. But in the end we have strayed somewhat far from RFC 3921. Will client and server developers come along for the ride? Are the costs worth the benefits? I think we need some input from someone other than you, me, and Joe in order to know. I volunteer Rachel Blackman for the job. ;-) Peter -- Peter Saint-Andre https://stpeter.im/ smime.p7s Description: S/MIME Cryptographic Signature
Re: [Standards] Proposed XMPP Extension: Roster Versioning
On Mon Mar 10 23:45:28 2008, Peter Saint-Andre wrote: One potential problem: this is not a nice, small, incremental change. Now servers and clients must support: 1. The 'sequence' attribute. 2. Roster pushes via message, not IQ. 3. Multiple items in a roster push. 4. Multiple items in a roster set. The more we change, the less likely it is that clients and servers will add this feature. Then we're back where we started. "If it ain't broke, don't fix it." So what's broken? 1. Huge roster gets every time you log in. The 'sequence' stuff fixes this. What's not broken? 2. Roster pushes via IQ. 3. One item per roster push. 4. One item per roster set. Why are we fixing 2, 3, and 4? Just because we can? Because it is more elegant? Or because it really solves a big problem on the network? Unless there is a compelling need, I'd vote for changing as little as possible. The problem is that if you go for Joe's concept - which is certainly elegant - of having the roster "diff" simply sent as a bunch of pushes, then these - being sent as - need to be acknowledged. Now, I appreciate that clients don't always do this, but the fact is that they should be doing so. I'm unhappy with suggesting that clients quietly ignore RFC 3920 because nobody cares. So we can fix that simply by using instead of - it's a pretty trivial change, and it eliminates the type='result' reply for clients on all pushes, so it's inarguably a performance improvement. Next up, Joe introduces a problem in as much as the client can't tell when it's finished receiving updates, and is now receiving bona-fide pushes. Joe says this isn't a problem, but I'm unconvinced - client authors have said this is a niggle, at least, for the initial presence push, after all. We can easily avoid this by having multiple items in a push - again, this cuts down on overhead, and so is a performance improvement. These two are simply taking Joe's concept - the update-as-push - and cleaning it up, getting the protocol better. Now... On to point 4. This isn't really needed, and it's not solving a big problem. But it's not a huge addition, and it does gain us a bit, and - perhaps most importantly - it fits neatly into this spec, and is too small to be independent. Dave. -- Dave Cridland - mailto:[EMAIL PROTECTED] - xmpp:[EMAIL PROTECTED] - acap://acap.dave.cridland.net/byowner/user/dwd/bookmarks/ - http://dave.cridland.net/ Infotrope Polymer - ACAP, IMAP, ESMTP, and Lemonade
Re: [Standards] Proposed XMPP Extension: Roster Versioning
Further thoughts... Dave Cridland wrote: > The IQ responses can be gotten rid of by sending these - and all - > roster pushes as message stanzas instead. So, if your client uses > XEP-0237, you're implicitly telling the server to send you roster pushes > as stanzas. Now clients can legally not respond to roster > push iq's, because they're not iq's anymore. Hoorah! > > We can get rid of per-push stanza wrapping overhead by allowing multiple > items per push, too. > > Now what we have is that when you ask for your roster, you get a single > based push containing all items, or else you get your iq > result with the replacement roster. Hardly any overhead at all, now. > > Given that we're now handling multiple pushes, it seems fair to go a > step further, and allow clients to have multiple items in a roster set. One potential problem: this is not a nice, small, incremental change. Now servers and clients must support: 1. The 'sequence' attribute. 2. Roster pushes via message, not IQ. 3. Multiple items in a roster push. 4. Multiple items in a roster set. The more we change, the less likely it is that clients and servers will add this feature. Then we're back where we started. "If it ain't broke, don't fix it." So what's broken? 1. Huge roster gets every time you log in. The 'sequence' stuff fixes this. What's not broken? 2. Roster pushes via IQ. 3. One item per roster push. 4. One item per roster set. Why are we fixing 2, 3, and 4? Just because we can? Because it is more elegant? Or because it really solves a big problem on the network? Unless there is a compelling need, I'd vote for changing as little as possible. Peter -- Peter Saint-Andre https://stpeter.im/ smime.p7s Description: S/MIME Cryptographic Signature
Re: [Standards] Proposed XMPP Extension: Roster Versioning
Dave Cridland wrote: > The IQ responses can be gotten rid of by sending these - and all - > roster pushes as message stanzas instead. So, if your client uses > XEP-0237, you're implicitly telling the server to send you roster pushes > as stanzas. Now clients can legally not respond to roster > push iq's, because they're not iq's anymore. Hoorah! > > We can get rid of per-push stanza wrapping overhead by allowing multiple > items per push, too. > > Now what we have is that when you ask for your roster, you get a single > based push containing all items, or else you get your iq > result with the replacement roster. Hardly any overhead at all, now. > > Given that we're now handling multiple pushes, it seems fair to go a > step further, and allow clients to have multiple items in a roster set. Aha, it's one of *those*. Yes, a fancy-dancy "I know what I'm doing Mr. Server so let's be modernistic and forget about all that old fuddy-duddy XMPP 1.0 garbage, shall we?" feature! Joe and I talked about that the other day for some other features (starttls and sasl without stream restarts, thank you very much!). It's a fascinating concept and I plan to write it up for starttls and sasl real soon now... > I still think it'd be nice to go the final step and allow clients to > specify a sequence value here to allow for lockless atomic sets, too. > > I'll write some text. :-) Text is always appreciated, thanks! ;-) Peter -- Peter Saint-Andre https://stpeter.im/ smime.p7s Description: S/MIME Cryptographic Signature
Re: [Standards] Proposed XMPP Extension: Roster Versioning
On Mon Mar 10 21:55:33 2008, Peter Saint-Andre wrote: Dave Cridland wrote: > On Mon Mar 10 20:17:53 2008, Joe Hildebrand wrote: >> - If the roster has been completely deleted, the response would be >> empty, but have version 0 > > I've a diff for specifying a sequence validity, which changes when the > roster is effectively a new one. (So it'll change if the account is > deleted and recreated, and it'll change if the server is restored from > backup, and it'll change if the server simply can't really be bothered > to use any kind of useful sequence, and instead changes it on every > change). Joe's suggestion works for me here, but I confess that I don't understand the concept of a sequence validity marker. If we did things Joe's way, then the server would have to know whether the roster had been destroyed and recreated since the client asked last. And that involves telepathy, I think. Instead, we push this to the client. Servers just plonk an arbitrary splodge (note the extremely technical language) on a roster when it's created, and they tell the client what it is at the beginning of a session. Now, if the client sees that the spodge the server says is different from the one they saw last, they know their cache is now useless, and can discard it, and start from scratch. This is useful if: a) The server has to be restored from backup, and so clients may be out of sync. The SysAdmin now just pushes the button that says "invalidate all sequences". Or maybe the server can tell. (By storing these on a persistent, but not backed up, medium). b) The server doesn't really have sequences at all. In this instance, it pretends to have a sequence for the duration of the session, but it presents some hash of the roster's contents as the roster validity splodge. No changes? No worries. Changes? Cast aside thy cache, little client, and I shall bestow upon thee a new roster. c) The account has been deleted and recreated, and so the new roster is no relation at all of the one the client has cached. d) The heat death of the universe has been and gone, and the 256-bit sequence finally rolls over on the roster of Saint Peter Saint-Andre (formally canonized by the temple of XMPP, now the state religion of the universe). I pondered that for a while before settling on the diff. My main concern with the roster-push-only model is that there could be an awful lot of roster pushes, each of which (1) has TCP and IQ overhead associated with it and (2) needs to be acked by the client per RFC 3920 (but in practice not all clients do this). But if there would be lots of roster pushes (where "lots" is defined by that smart server to which your dumb client is connected) then the server could send you the complete roster, right? Joe and I pondered some more. The TCP overhead is largely lost by pipelining - the server knows how many pushes it needs to send, so that's one TCP packet, one ACK. The IQ responses can be gotten rid of by sending these - and all - roster pushes as message stanzas instead. So, if your client uses XEP-0237, you're implicitly telling the server to send you roster pushes as stanzas. Now clients can legally not respond to roster push iq's, because they're not iq's anymore. Hoorah! We can get rid of per-push stanza wrapping overhead by allowing multiple items per push, too. Now what we have is that when you ask for your roster, you get a single based push containing all items, or else you get your iq result with the replacement roster. Hardly any overhead at all, now. Given that we're now handling multiple pushes, it seems fair to go a step further, and allow clients to have multiple items in a roster set. I still think it'd be nice to go the final step and allow clients to specify a sequence value here to allow for lockless atomic sets, too. I'll write some text. :-) Dave. -- Dave Cridland - mailto:[EMAIL PROTECTED] - xmpp:[EMAIL PROTECTED] - acap://acap.dave.cridland.net/byowner/user/dwd/bookmarks/ - http://dave.cridland.net/ Infotrope Polymer - ACAP, IMAP, ESMTP, and Lemonade
Re: [Standards] Proposed XMPP Extension: Roster Versioning
Peter Saint-Andre wrote: > Dave Cridland wrote: >> On Mon Mar 10 20:17:53 2008, Joe Hildebrand wrote: >>> In section 2.4, would it be easier to implement in existing clients if: >>> - All changes would be sent to the client as roster pushes after the >>> iq/response, ordered such that the last one had the current version >>> number > > I pondered that for a while before settling on the diff. My main concern > with the roster-push-only model is that there could be an awful lot of > roster pushes, each of which (1) has TCP and IQ overhead associated with > it and (2) needs to be acked by the client per RFC 3920 (but in practice > not all clients do this). But if there would be lots of roster pushes > (where "lots" is defined by that smart server to which your dumb client > is connected) then the server could send you the complete roster, right? > >> That's quite a neat design. It eliminates the diff attribute. > > Yes, I think it would. > >>> Then, all I have to do is store my current roster, and apply diffs to >>> it in exactly one way. >>> >>> >> Yes, I like this. > > It's definitely worth pondering some more. OK, I pondered it some more. I think the roster-push-only design is superior, because clients already know how to handle roster pushes, whereas they don't know how to handle roster diffs. If we can define roster sequencing in such a way that it uses only the primitives that clients already understand (full roster and roster push), then clients will be more likely to implement roster sequencing. And that's a Good Thing. Now we just need to describe a good rule of thumb for servers to follow in deciding whether to send the full roster or a series of roster pushes. Peter -- Peter Saint-Andre https://stpeter.im/ smime.p7s Description: S/MIME Cryptographic Signature
Re: [Standards] Proposed XMPP Extension: Roster Versioning
Dave Cridland wrote: > On Mon Mar 10 20:17:53 2008, Joe Hildebrand wrote: >> In section 2.4, would it be easier to implement in existing clients if: >> >> - The response to the iq/get with version 0 was a full roster, in the >> existing format, with the addition of the current version number > > I think it is, isn't it? The spec actually allows a diff from nothing, > but in this instance, it doesn't matter whether it's technically a > "diff" or not, since you'll get full items for everything that's > changed, and everything will have done. Agreed, I think that's what the server returns in this case. >> - The response to iq/get could always be a full roster, if the server >> got confused in any way, or know that it would be more efficient to >> just start from scratch > > This is (or should be) the case. Agreed. >> - If the roster has been completely deleted, the response would be >> empty, but have version 0 > > I've a diff for specifying a sequence validity, which changes when the > roster is effectively a new one. (So it'll change if the account is > deleted and recreated, and it'll change if the server is restored from > backup, and it'll change if the server simply can't really be bothered > to use any kind of useful sequence, and instead changes it on every > change). Joe's suggestion works for me here, but I confess that I don't understand the concept of a sequence validity marker. >> - The response to iq/get would otherwise be empty, except for the new >> version number >> - All changes would be sent to the client as roster pushes after the >> iq/response, ordered such that the last one had the current version >> number I pondered that for a while before settling on the diff. My main concern with the roster-push-only model is that there could be an awful lot of roster pushes, each of which (1) has TCP and IQ overhead associated with it and (2) needs to be acked by the client per RFC 3920 (but in practice not all clients do this). But if there would be lots of roster pushes (where "lots" is defined by that smart server to which your dumb client is connected) then the server could send you the complete roster, right? > That's quite a neat design. It eliminates the diff attribute. Yes, I think it would. >> Then, all I have to do is store my current roster, and apply diffs to >> it in exactly one way. >> >> > Yes, I like this. It's definitely worth pondering some more. Peter -- Peter Saint-Andre https://stpeter.im/ smime.p7s Description: S/MIME Cryptographic Signature
Re: [Standards] Proposed XMPP Extension: Roster Versioning
On Mon Mar 10 21:33:18 2008, Joe Hildebrand wrote: On Mar 10, 2008, at 2:52 PM, Dave Cridland wrote: 1) How does the client know that it's got all the updates? How does a client know it has all of its presence? It doesn't, and it doesn't matter. I'm not convinced it doesn't matter in that case, either. Same thing here. What's the difference between these roster pushes and any other roster push? These definitely happened in the past. 2) Doesn't this mean that every roster push has to be acknowledged? Doesn't this increase the transmissions required from a client? (Note that the client cannot pipeline them all into the same TCP packet, because of (1) - otherwise, it'd presumably compress well). No. Same as any roster push. You just have to store the version number, and your associated data model. Newton's second law of iq stanzas states that to every type='set|get'/>, there is an equal and opposite type='result|error'/>, though, so surely each push will generate a transmission? Using one of these trendy instant messaging things, Joe told me that nobody, apparently, sends type='result' anyway - is this true, folks, and does it matter? (I think it's a mistake to encourage this). Dave. -- Dave Cridland - mailto:[EMAIL PROTECTED] - xmpp:[EMAIL PROTECTED] - acap://acap.dave.cridland.net/byowner/user/dwd/bookmarks/ - http://dave.cridland.net/ Infotrope Polymer - ACAP, IMAP, ESMTP, and Lemonade
Re: [Standards] Proposed XMPP Extension: Roster Versioning
On Mar 10, 2008, at 2:52 PM, Dave Cridland wrote: 1) How does the client know that it's got all the updates? How does a client know it has all of its presence? It doesn't, and it doesn't matter. Same thing here. What's the difference between these roster pushes and any other roster push? 2) Doesn't this mean that every roster push has to be acknowledged? Doesn't this increase the transmissions required from a client? (Note that the client cannot pipeline them all into the same TCP packet, because of (1) - otherwise, it'd presumably compress well). No. Same as any roster push. You just have to store the version number, and your associated data model. -- Joe Hildebrand
Re: [Standards] Proposed XMPP Extension: Roster Versioning
On Mon Mar 10 20:32:17 2008, Dave Cridland wrote: On Mon Mar 10 20:17:53 2008, Joe Hildebrand wrote: - The response to iq/get would otherwise be empty, except for the new version number - All changes would be sent to the client as roster pushes after the iq/response, ordered such that the last one had the current version number That's quite a neat design. It eliminates the diff attribute. Then, all I have to do is store my current roster, and apply diffs to it in exactly one way. Yes, I like this. But then I thought a little harder whilst finishing cooking. 1) How does the client know that it's got all the updates? 2) Doesn't this mean that every roster push has to be acknowledged? Doesn't this increase the transmissions required from a client? (Note that the client cannot pipeline them all into the same TCP packet, because of (1) - otherwise, it'd presumably compress well). So I think yes - easiest to get rapid deployment, but no - it's much harder to get a quality implementation. You can take our existing "diff" format and push each item through your internal roster push handling, incidentally. Not much refectoring involved, although certainly more than your design, but you get a clean protocol at the end of it. Dave. -- Dave Cridland - mailto:[EMAIL PROTECTED] - xmpp:[EMAIL PROTECTED] - acap://acap.dave.cridland.net/byowner/user/dwd/bookmarks/ - http://dave.cridland.net/ Infotrope Polymer - ACAP, IMAP, ESMTP, and Lemonade
Re: [Standards] Proposed XMPP Extension: Roster Versioning
On Mar 10, 2008, at 2:32 PM, Dave Cridland wrote: On Mon Mar 10 20:17:53 2008, Joe Hildebrand wrote: In section 2.4, would it be easier to implement in existing clients if: - The response to the iq/get with version 0 was a full roster, in the existing format, with the addition of the current version number I think it is, isn't it? The spec actually allows a diff from nothing, but in this instance, it doesn't matter whether it's technically a "diff" or not, since you'll get full items for everything that's changed, and everything will have done. Oh. I hadn't fully understood the diff attribute. If we keep it, can it get it's own example? -- Joe Hildebrand
Re: [Standards] Proposed XMPP Extension: Roster Versioning
On Mon Mar 10 20:17:53 2008, Joe Hildebrand wrote: In section 2.4, would it be easier to implement in existing clients if: - The response to the iq/get with version 0 was a full roster, in the existing format, with the addition of the current version number I think it is, isn't it? The spec actually allows a diff from nothing, but in this instance, it doesn't matter whether it's technically a "diff" or not, since you'll get full items for everything that's changed, and everything will have done. - The response to iq/get could always be a full roster, if the server got confused in any way, or know that it would be more efficient to just start from scratch This is (or should be) the case. - If the roster has been completely deleted, the response would be empty, but have version 0 I've a diff for specifying a sequence validity, which changes when the roster is effectively a new one. (So it'll change if the account is deleted and recreated, and it'll change if the server is restored from backup, and it'll change if the server simply can't really be bothered to use any kind of useful sequence, and instead changes it on every change). - The response to iq/get would otherwise be empty, except for the new version number - All changes would be sent to the client as roster pushes after the iq/response, ordered such that the last one had the current version number That's quite a neat design. It eliminates the diff attribute. Then, all I have to do is store my current roster, and apply diffs to it in exactly one way. Yes, I like this. Dave. -- Dave Cridland - mailto:[EMAIL PROTECTED] - xmpp:[EMAIL PROTECTED] - acap://acap.dave.cridland.net/byowner/user/dwd/bookmarks/ - http://dave.cridland.net/ Infotrope Polymer - ACAP, IMAP, ESMTP, and Lemonade
Re: [Standards] Proposed XMPP Extension: Roster Versioning
In section 2.4, would it be easier to implement in existing clients if: - The response to the iq/get with version 0 was a full roster, in the existing format, with the addition of the current version number - The response to iq/get could always be a full roster, if the server got confused in any way, or know that it would be more efficient to just start from scratch - If the roster has been completely deleted, the response would be empty, but have version 0 - The response to iq/get would otherwise be empty, except for the new version number - All changes would be sent to the client as roster pushes after the iq/response, ordered such that the last one had the current version number Then, all I have to do is store my current roster, and apply diffs to it in exactly one way. On Mar 4, 2008, at 2:54 PM, XMPP Extensions Editor wrote: The XMPP Extensions Editor has received a proposal for a new XEP. Title: Roster Versioning Abstract: This specification proposes a modification to the XMPP roster management protocol to support versioning of rosters for more efficient downloading of the rost er information. URL: http://www.xmpp.org/extensions/inbox/roster-versioning.html The XMPP Council will decide at its next meeting whether to accept this proposal as an official XEP. -- Joe Hildebrand
Re: [Standards] Proposed XMPP Extension: Roster Versioning
Thanks for taking the time for the explanation, whilst I still think it would be advantageous for clients to not gage any meaning from the version ids (or whatever we are now going to call them) and just leave all the logic to the server to deal with, i'm not too bothered as to what format they are as as myself and others here have pointed out its pretty much a server implementation issue and as Pedro mentioned the "MAY be a unique identifer that is opaque to the client but understood by the server" statement would seem to keep things open and allow enough flexibility for us to do things in the best way for our implementations (and luckily if I do choose to implement the ids as the increasing integers it will be very easy and just a small change to the section that is currently generating the timestamps). Richard
Re: [Standards] Proposed XMPP Extension: Roster Versioning
Just an additional note, OSCAR (AIM/ICQ) does this too. Basically all you really do is see what version of your SSI is stored on the server, see if that's the same number as what you have recorded in your client, and if they differ, clearly something has changed and you need to pull the update from the server. It is sequential, but it doesn't matter that it's sequential. (or at least I've never seen an instance where it matters) I might be remembering this wrong too.. it might be that you send the version YOU know about, and OSCAR either sends you "you're good" or it sends you your roster. One of those. Either way, doesn't matter what the "key" is really. =) It's just conceptually easier to think about it in terms of version numbers. Daniel - "Dave Cridland" <[EMAIL PROTECTED]> wrote: > On Thu Mar 6 22:23:51 2008, Pedro Melo wrote: > > Others might counter-propose something meaningful but some are > > proposing something opaque, without meaning for the client. And > > that is the best way for clients not to mess up. > > Okay, I call. :-) > > How *can* clients mess this up by knowing they can compare two > values? > > It's never, to my knowledge, been a problem in ACAP or IMAP, both of > > which use something similar. It's been quite handy in IMAP CONDSTORE, > > in fact. > > The only particularly curious thing about integers is that it's > possible to note that, given two values, there's a maximum number of > > changes that have happened between them. (But a fixed minimum of > zero, since the server's allowed to skip values). But even if a > client decides that the sequence values always change by one, and > only for a visible change, I can't see any possibility for a protocol > > error. I'm not even sure what you'd do with that erroneous > information, but maybe I'm too used to dealing with these things. > > So, please enlighten me - what is it that I'm missing? > > > Agreed. Hence a recommendation on implementation notes to use a > > counter with those properties. > > If clients did "mess it up", probably by testing against servers > using an integer, relying on it somehow, then finding servers with an > > opaque string in the field, how much worse will the problem be? (See > > Curtis's email for a very accurate account of how many > implementations get developed). > > Either make it "really" opaque, if you have to, or else make it > clearly defined. So if you're going to remove the "integer" property, > > then you'd best make it such that clients *cannot* compare it at all. > > (Good luck with specifying that). > > FWIW, I've said before that it's perfectly possible to map strictly > increasing modified timestamps to a strictly increasing integer. I'm > > not dictating server internals - you can generate a strictly > increasing sequence value however you like, as long as you can map it > > to an integer. > > Dave. > -- > Dave Cridland - mailto:[EMAIL PROTECTED] - xmpp:[EMAIL PROTECTED] > - acap://acap.dave.cridland.net/byowner/user/dwd/bookmarks/ > - http://dave.cridland.net/ > Infotrope Polymer - ACAP, IMAP, ESMTP, and Lemonade
Re: [Standards] Proposed XMPP Extension: Roster Versioning
On Thu Mar 6 22:23:51 2008, Pedro Melo wrote: Others might counter-propose something meaningful but some are proposing something opaque, without meaning for the client. And that is the best way for clients not to mess up. Okay, I call. :-) How *can* clients mess this up by knowing they can compare two values? It's never, to my knowledge, been a problem in ACAP or IMAP, both of which use something similar. It's been quite handy in IMAP CONDSTORE, in fact. The only particularly curious thing about integers is that it's possible to note that, given two values, there's a maximum number of changes that have happened between them. (But a fixed minimum of zero, since the server's allowed to skip values). But even if a client decides that the sequence values always change by one, and only for a visible change, I can't see any possibility for a protocol error. I'm not even sure what you'd do with that erroneous information, but maybe I'm too used to dealing with these things. So, please enlighten me - what is it that I'm missing? Agreed. Hence a recommendation on implementation notes to use a counter with those properties. If clients did "mess it up", probably by testing against servers using an integer, relying on it somehow, then finding servers with an opaque string in the field, how much worse will the problem be? (See Curtis's email for a very accurate account of how many implementations get developed). Either make it "really" opaque, if you have to, or else make it clearly defined. So if you're going to remove the "integer" property, then you'd best make it such that clients *cannot* compare it at all. (Good luck with specifying that). FWIW, I've said before that it's perfectly possible to map strictly increasing modified timestamps to a strictly increasing integer. I'm not dictating server internals - you can generate a strictly increasing sequence value however you like, as long as you can map it to an integer. Dave. -- Dave Cridland - mailto:[EMAIL PROTECTED] - xmpp:[EMAIL PROTECTED] - acap://acap.dave.cridland.net/byowner/user/dwd/bookmarks/ - http://dave.cridland.net/ Infotrope Polymer - ACAP, IMAP, ESMTP, and Lemonade
Re: [Standards] Proposed XMPP Extension: Roster Versioning
Hi. Please ignore this message. I hadn't read the latest version of the XEP, and the "MAY be a unique identifer that is opaque to the client but understood by the server" covers most if not all of my argument below. :) Best regards, On Mar 6, 2008, at 10:23 PM, Pedro Melo wrote: Hi, On Mar 6, 2008, at 4:27 AM, Peter Saint-Andre wrote: I'm not convinced that that's the intent. It seems that it might be useful in some situations for the client to know where it stands. If we are going to have semantic meaning the in the version then the spec must reflect what the meaning is. I agree that servers will use a counter or sequence per roster for this, because its efficient and they can easily see what needs to be sent to the client. What I don't get is why we should force that in the spec, given that the client never uses the version semantically, he only sends it back to the server. I think the version should be specifiec as a opaque string, and in the implementation notes recommend strongly the use of a counter for the reasons Dave Cridland gave. I'll let Dave Cridland explain why he thinks this is not overspecification. I think his experience with IETF email and data storage protocols is relevant here. In particular, during the Council meeting [1] he said: "It's a lot more efficient with an int, and everything else is either worse performance, fails entirely, or else is equivalent difficulty to an int." he is right but thats a server issue, Clients don't care, they store it and send it back later. So at most this is relevant as implementation notes for servers. Again: I think most of us agree that servers will use an integer counter because its the most efficient way to implement the entire XEP. What I don't see is the need to tell that in the spec, therefore limiting us for the future. "One interesting point is that with an int, there's always something a client can use to get the entire roster, with versioning turned on." Sending a 0? what about sending an empty version attribute? Same thing. "I'm not quite sure what the client ought to send if everything's completely opaque - it'd need more syntax." Empty? "Put it this way, since the counter-proposal involved timestamps, which are known to be broken, I'm pretty sure people will get stuff wrong without it being a MUST." Others might counter-propose something meaningful but some are proposing something opaque, without meaning for the client. And that is the best way for clients not to mess up. Server authors have to get it right based on their architecture of their server, and that is something that I don't think specs should meddle with. And previously on the list he said [2]: "strictly increasing numeric sequences have proved useful in the IMAP world, because clients can spot when things go wrong much more easily." Agreed. Hence a recommendation on implementation notes to use a counter with those properties. Best regtards, -- Pedro Melo Blog: http://www.simplicidade.org/notes/ XMPP ID: [EMAIL PROTECTED] Use XMPP! -- Pedro Melo Blog: http://www.simplicidade.org/notes/ XMPP ID: [EMAIL PROTECTED] Use XMPP!
Re: [Standards] Proposed XMPP Extension: Roster Versioning
Hi, On Mar 6, 2008, at 4:27 AM, Peter Saint-Andre wrote: I'm not convinced that that's the intent. It seems that it might be useful in some situations for the client to know where it stands. If we are going to have semantic meaning the in the version then the spec must reflect what the meaning is. I agree that servers will use a counter or sequence per roster for this, because its efficient and they can easily see what needs to be sent to the client. What I don't get is why we should force that in the spec, given that the client never uses the version semantically, he only sends it back to the server. I think the version should be specifiec as a opaque string, and in the implementation notes recommend strongly the use of a counter for the reasons Dave Cridland gave. I'll let Dave Cridland explain why he thinks this is not overspecification. I think his experience with IETF email and data storage protocols is relevant here. In particular, during the Council meeting [1] he said: "It's a lot more efficient with an int, and everything else is either worse performance, fails entirely, or else is equivalent difficulty to an int." he is right but thats a server issue, Clients don't care, they store it and send it back later. So at most this is relevant as implementation notes for servers. Again: I think most of us agree that servers will use an integer counter because its the most efficient way to implement the entire XEP. What I don't see is the need to tell that in the spec, therefore limiting us for the future. "One interesting point is that with an int, there's always something a client can use to get the entire roster, with versioning turned on." Sending a 0? what about sending an empty version attribute? Same thing. "I'm not quite sure what the client ought to send if everything's completely opaque - it'd need more syntax." Empty? "Put it this way, since the counter-proposal involved timestamps, which are known to be broken, I'm pretty sure people will get stuff wrong without it being a MUST." Others might counter-propose something meaningful but some are proposing something opaque, without meaning for the client. And that is the best way for clients not to mess up. Server authors have to get it right based on their architecture of their server, and that is something that I don't think specs should meddle with. And previously on the list he said [2]: "strictly increasing numeric sequences have proved useful in the IMAP world, because clients can spot when things go wrong much more easily." Agreed. Hence a recommendation on implementation notes to use a counter with those properties. Best regtards, -- Pedro Melo Blog: http://www.simplicidade.org/notes/ XMPP ID: [EMAIL PROTECTED] Use XMPP!
Re: [Standards] Proposed XMPP Extension: Roster Versioning
Curtis King wrote: > > On 6-Mar-08, at 12:50 AM, Justin Karneges wrote: > >> What counts as what is a matter of what the usual design practices and >> trends >> are for related specifications. > > An other factor to consider is how most people actually implement a > protocol. Guess what, it's not by reading the RFCs and XEPs in great > detail ;-) It's by protocol examples either from tcpdumps or the > specification itself and then testing against a common server or client > until it "works". So, it's very common for someone to miss the fact some > data should be treated as opaque. The best specifications define how the > data should be interpreted. Heh, true. Why do you think our specs have so many examples? "We put the example in example.com!" /psa smime.p7s Description: S/MIME Cryptographic Signature
Re: [Standards] Proposed XMPP Extension: Roster Versioning
On 6-Mar-08, at 12:50 AM, Justin Karneges wrote: What counts as what is a matter of what the usual design practices and trends are for related specifications. An other factor to consider is how most people actually implement a protocol. Guess what, it's not by reading the RFCs and XEPs in great detail ;-) It's by protocol examples either from tcpdumps or the specification itself and then testing against a common server or client until it "works". So, it's very common for someone to miss the fact some data should be treated as opaque. The best specifications define how the data should be interpreted. ck
Re: [Standards] Proposed XMPP Extension: Roster Versioning
Dave Cridland wrote: > On Thu Mar 6 17:25:18 2008, Peter Saint-Andre wrote: >> Dave Cridland wrote: >> >> > Then it doesn't get versioning. (BTW, I hate that name - there are no >> > versions kept. It's not like a client can ask for a previous version, >> > and nor does a server need to keep any previous versions). >> >> I'm not wedded to that name. Feel free to suggest another. Roster diffs? > > No, you don't (really) get diffs, either. It's easy to think that a diff > is generated between two versions, which isn't really happening. > (Thankfully - it'd be much harder to implement if it were the case). > > Roster change sequencing, which uses a roster change sequence, and > associates each change with a roster change sequence value? > > Or roster sequencing, etc, for a shorter term. s/version/roster-seq/ > throughout? Done in version 0.2, other clarifications added as well: http://www.xmpp.org/extensions/xep-0237.html Peter -- Peter Saint-Andre https://stpeter.im/ smime.p7s Description: S/MIME Cryptographic Signature
Re: [Standards] Proposed XMPP Extension: Roster Versioning
On Thu Mar 6 17:25:18 2008, Peter Saint-Andre wrote: Dave Cridland wrote: > Then it doesn't get versioning. (BTW, I hate that name - there are no > versions kept. It's not like a client can ask for a previous version, > and nor does a server need to keep any previous versions). I'm not wedded to that name. Feel free to suggest another. Roster diffs? No, you don't (really) get diffs, either. It's easy to think that a diff is generated between two versions, which isn't really happening. (Thankfully - it'd be much harder to implement if it were the case). Roster change sequencing, which uses a roster change sequence, and associates each change with a roster change sequence value? Or roster sequencing, etc, for a shorter term. s/version/roster-seq/ throughout? Dave. -- Dave Cridland - mailto:[EMAIL PROTECTED] - xmpp:[EMAIL PROTECTED] - acap://acap.dave.cridland.net/byowner/user/dwd/bookmarks/ - http://dave.cridland.net/ Infotrope Polymer - ACAP, IMAP, ESMTP, and Lemonade
Re: [Standards] Proposed XMPP Extension: Roster Versioning
Dave Cridland wrote: > Then it doesn't get versioning. (BTW, I hate that name - there are no > versions kept. It's not like a client can ask for a previous version, > and nor does a server need to keep any previous versions). I'm not wedded to that name. Feel free to suggest another. Roster diffs? smime.p7s Description: S/MIME Cryptographic Signature
Re: [Standards] Proposed XMPP Extension: Roster Versioning
On Thu Mar 6 09:45:17 2008, Richard Dobson wrote: A few hours of testing does not a reliable protocol make. Under what conditions? Is this a public server that people can test against? No you are correct, but even so the tone of some of the messages on this subject as far as I read them said that it wouldn't work, and under my limited testing it does seem to. I can make it publically accessible if you like and actually want to and will have a go. You can't rely on testing, sadly; you need to prove that it works. Timestamps probably do work in most cases, as long as the updates to the roster are atomically and sequentially performed on a single point, and your implementation is sufficiently slow that timestamps are obtained at a maximum frequency lower than their resolution. This is simply because then you have a strictly increasing sequence. However, timestamps don't always work like that even on a single machine. It just happens that they usually do. I'm not terribly keen on making the specification define a new RFC 2119 entry candidate for MUST USUALLY. ;-) "It's a lot more efficient with an int, and everything else is either worse performance, fails entirely, or else is equivalent difficulty to an int." What sort of things are these? Im not trying to say im right and he's wrong but id like a more full explanation on the points above and below so I can understand why Dave thinks it will break, because as far as I can see all the problems pointed out so far in this thread can be easily overcome. If you modify a timestamp value, or for that matter generate it in such a way that it's strictly increasing, then everything *will* work. But since you can model that trivially as a non-negative integer, it's not a problem. But since you can, it's a lot easier to simply use an arbitrary strictly increasing integer sequence anyway. "One interesting point is that with an int, there's always something a client can use to get the entire roster, with versioning turned on." Yea it can just omit the version attribute. Then it doesn't get versioning. (BTW, I hate that name - there are no versions kept. It's not like a client can ask for a previous version, and nor does a server need to keep any previous versions). "Put it this way, since the counter-proposal involved timestamps, which are known to be broken, I'm pretty sure people will get stuff wrong without it being a MUST." Why are they broken? See previous messages. They're not strictly increasing, nor are they strictly non-decreasing, nor are they even unique. And [3]: "You can't use timestamps - they're not strictly increasing, for various reasons. Why does it have to be strictly increasing, even if it was a timestamp? Because you want them to have the property that for any known state, the server can produce a delta to the current state. Now, as it happens, we allow in this specification for a server to give up and provide the complete roster, and given this escape hatch, "unique" is sufficient if you wanted to produce a really poor implementation. However, you do need to ensure it really is unique, and given multi-core multi-threading clustered servers, about the simplest way is to just use a strictly increasing integer sequence. After all, you need some central point to perform atomic updates of the roster, right? Now, if what you have is a sequence which usually goes up, but sometimes goes down, and is not unique, then updates can simply be lost to a client. Firstly, two roster changes could happen at precisely the same moment. To be fair, by introducing cluster node identifiers, and having a strict strong ordering of them, you could avoid this. Why is it a problem if two updates share the same version identifier? Couldn't they not just become part of a single atomic change? That's "strictly non-decreasing", and neither "unique" nor "strictly increasing". The problem here is that a roster push only contains one item, so two roster pushes would involve the same sequence value. (See rfc3921bis, Section 2.1.4) Given this, you need to ensure that either: a) There is some way to indicate to the client that it now has all roster pushes up to and including the sequence value. (ACAP style, if you're wondering, since ACAP also has atomic updates spanning multiple notifications). This involves an additional stanza, and if it's lost, a client will observe all the previous pushes with the same sequence value repeated - as such, it's inherently less efficient on dodgy connectivity. b) You cheat and lower the sequence value included in the roster pushes, such that the sequence value the client next produces to resynchronize will include all those pushes - effectively using a subsequent push as an indicator to the client that all pushes relating to previous sequence values have now been received. (
Re: [Standards] Proposed XMPP Extension: Roster Versioning
On Thu Mar 6 07:50:32 2008, Justin Karneges wrote: What counts as what is a matter of what the usual design practices and trends are for related specifications. My assessment, from what I read in the mailing list arguments, is that opaque was in the 'neutral' category. Thus it should not be Richard that needs to argue for opaque, but rather it should be Dave that needs to argue for integer. IMO, I guess. *sigh* I wouldn't put my name to it, but I suspect that merely "unique" is sufficient to avoid breakage in the client, although it'll produce a fairly inefficient protocol, equivalent to a simple "get-if-none-match". It's possible - but again, I wouldn't put my name to it - that "strictly non-decreasing" is okay, too. I'm a lot less sure about this, I've not thought too hard about the failure cases. I think this would yield inefficiency, too, but I think it might not break anything. Perhaps. For efficiency, then, you need a strictly increasing sequence, which has the useful property that it's known to work in several use-cases now. Now, potentially, this need not be a non-negative integer sequence, but non-negative integer sequence is, by far, the simplest one to implement, and has the convenient property that we know where it starts. They're a lot easier to implement than a modified strictly increasing timestamp. (I know, I've done both). There's the other advantage that strictly increasing integer sequences are very well understood, having been used in various places for a couple of decades for synchronization in network protocols. (The IMAP UID[NEXT] is one such, as is the more recent [HIGHEST]MODSEQ, used in real deployments. ACAP used a strictly increasing timestamp-derived value, and this is not only more complicated to increment, it's also slower, mandating a system call in addition to the contention point, and is nightmarishly difficult to explain clearly in a specification, and finally, it's not really quite an integral value, and instead needs comparing as a string, not an int) Now - should a client be able to compare two sequence values and see which one is higher? If so, we want a clearly defined sequence, if not, we don't care. Well, this has been the case for IMAP UIDs for years, and is remarkably handy in this particular instance, but that's unfair - UIDs are, by intent, designed for this. IMAP CONDSTORE was long thought to be able to be implemented in the client as pure strings. My own client implementation does actually compare values in a few cases for efficiency - and I'm only doing the synchronization side of CONDSTORE, which is equivalent to what we're doing here. I'd hate to discover late in the day that we actually need to do this. The assertion that by allowing client-side comparison of sequence values it's possible to introduce bugs is an interesting one, but as far as I can tell, baseless - it's yet to cause a problem in IMAP CONDSTORE or ACAP modtime, and indeed has actually been a bit of a benefit. Still, I'd be happy if there were a directive that clients SHOULD NOT compare locally. One case where client-side comparison of sequence values is very useful indeed is for external testing - it makes it much easier when you can perform simple verification. Recently, we've seen an upsurge in wire-based external testing/validation tools in IMAP, and they've proven very useful for server developers. So I'd lean toward "does no harm", and "can be useful in some circumstances", and therefore we need a well-defined strictly increasing sequence. There are many such sequences, however, but the one that strikes me as most useful is an integral one. Life because even easier when we know of a particular value which is lower than or equal to all others in the sequence, too. So, in conclusion, I'll stick to my guns and say we want a strictly increasing integer sequence. > "One interesting point is that with an int, there's always something a > client can use to get the entire roster, with versioning turned on." > > "I'm not quite sure what the client ought to send if everything's > completely opaque - it'd need more syntax." Maybe these could be explained here on the list. The implication in these statements is that the revision value has meaning to the client, and that the client can even create these values. Yes, it can, in one case - where it wishes to specify a sequence value that occurs prior to all others. Clients use this in the spec as written to request all updates from a sequence value lower than, or equal to, all possible sequence values. (This does, to be fair, break pre-populated rosters in a naïve implementation - if your server does prepopulated rosters, like shared rosters, you'll need to update the sequence when you [pre-]populate externally.) We can do this bootstrapping in two ways. Either a special attribute
Re: [Standards] Proposed XMPP Extension: Roster Versioning
A few hours of testing does not a reliable protocol make. Under what conditions? Is this a public server that people can test against? No you are correct, but even so the tone of some of the messages on this subject as far as I read them said that it wouldn't work, and under my limited testing it does seem to. I can make it publically accessible if you like and actually want to and will have a go. I'm not convinced that that's the intent. It seems that it might be useful in some situations for the client to know where it stands. Then id suggest that gets specified in the spec. We do need technologies that won't break. Sure but it hasn't been fully explained why it would break. "It's a lot more efficient with an int, and everything else is either worse performance, fails entirely, or else is equivalent difficulty to an int." What sort of things are these? Im not trying to say im right and he's wrong but id like a more full explanation on the points above and below so I can understand why Dave thinks it will break, because as far as I can see all the problems pointed out so far in this thread can be easily overcome. "One interesting point is that with an int, there's always something a client can use to get the entire roster, with versioning turned on." Yea it can just omit the version attribute. "I'm not quite sure what the client ought to send if everything's completely opaque - it'd need more syntax." What kind of syntax were you thinking of? "Put it this way, since the counter-proposal involved timestamps, which are known to be broken, I'm pretty sure people will get stuff wrong without it being a MUST." Why are they broken? And previously on the list he said [2]: "strictly increasing numeric sequences have proved useful in the IMAP world, because clients can spot when things go wrong much more easily." Ok thats certainly fair enough, although it would be good to specify that in the spec. And [3]: "You can't use timestamps - they're not strictly increasing, for various reasons. Why does it have to be strictly increasing, even if it was a timestamp? Firstly, two roster changes could happen at precisely the same moment. To be fair, by introducing cluster node identifiers, and having a strict strong ordering of them, you could avoid this. Why is it a problem if two updates share the same version identifier? Couldn't they not just become part of a single atomic change? Secondly, the clock on a computer can, and surprisingly often does, go backwards. That's a much harder problem to solve. As previously requested could you describe this further or point to some more information on this so I can understand how much of a problem this actually is. Thirdly, in a clustering situation, you'd have to ensure that the time on each cluster node was perfectly synchronized. No they don't as previously pointed out (the database layer could generate all the ids). So the closest you can do would be a modified timestamp that had additional logic during generation to ensure it never went backwards, in which case you don't need the cluster identifier anymore, and that's effectively the same as having a strictly increasing integer sequence anyway, so it's easier to just do that. But even if you did want to use timestamps, just representing them as an integer is pretty trivial. Look at the definition of "modtime" in ACAP (RFC 2244), which defines a strictly increasing modified timestamp represented using digits." Yup exactly so the issue about clocks going backwards can be easily overcome then. I don't see that these concerns were addressed on the list. And I don't see this as an appeal to authority or nannyism, just heeding the voice of experience. I did respond similar to above, but didn't see any response to my questions, I would appreciate some answers so I can at least understand where Dave is coming from. Is there any chance we can use BASE64 or something to compress the version identifiers in the roster pushes and whatnot?, just like is done with the hashes in caps? Or do we think its probably not really going to be a problem in the future? Granted as an auto incrementing integer it probably wont be much of an issue unless you have an incredibly busy roster, just trying to plan ahead. Richard
Re: [Standards] Proposed XMPP Extension: Roster Versioning
Hi!
Comments below marked as jph:
Cheer's
>-Original Message-
>From: [EMAIL PROTECTED]
>[mailto:[EMAIL PROTECTED] On Behalf Of ext Peter Saint-Andre
>Sent: 06 March, 2008 06:36
>To: XMPP Extension Discussion List
>Subject: Re: [Standards] Proposed XMPP Extension: Roster Versioning
>
>[EMAIL PROTECTED] wrote:
>> Hi!
>>
>> I readed the latest version 0.0.3 and I'm little bit confused of
>> statement (or I have missed something):
>>
>> --- clip ---
>> The "roster diff" can be understood as follows:
>>
>>1. Imagine that the client had an active presence session for the
>> entire time between its cached roster version (in this case,
>305) and
>> the new roster version (317).
>>2. During that time, the client would have received roster pushes
>> related to roster versions 306, 307, 308, 309, 310, 311, 312, 313,
>> 314, 315, 316, and 317.
>>3. However, some of those roster pushes might have contained
>> intermediate updates to the same roster item (e.g., changes in the
>> subscription state for [EMAIL PROTECTED] from "none" to "to" and
>> from "to" to "both").
>>4. The roster result would not include all of the intermediate
>> steps, only the final result of all changes applied while the client
>> was in fact offline.
>>
>> --- clip ---
>>
>> Q: How many roster version have to server remember? In the xep 0.0.3
>> there is clear spec for client site implementation, but
>server site is
>> missing "server site requirements". If I keep changing the
>roster add,
>> remove few times ("on/off relationship") --> how many roster
>versions
>> have to be server must remember backwads 1...n?
>
>I suppose that's up to the implementation. But if the server
>doesn't remember back to the version cached by the client then
>we need to specify the server's behavior (probably it would
>return the latest version of the entire roster, just as it does today).
>
jph: Yes, the "server's behavior" must be define.
I agree "probably it would >return the latest version of
the entire roster, just as it does today" is excellent and
simplest approach
and it will work.
>
>Peter
>
>--
>Peter Saint-Andre
>https://stpeter.im/
>
>
Re: [Standards] Proposed XMPP Extension: Roster Versioning
On Wednesday 05 March 2008 8:27 pm, Peter Saint-Andre wrote: > Justin Karneges wrote: > > I, too, don't understand why this value needs to be specified as being a > > strictly-increasing integer, if the intent is that clients should treat > > it as an opaque cookie thing anyway. > > I'm not convinced that that's the intent. It seems that it might be > useful in some situations for the client to know where it stands. That changes things then. I saw some mention of that in the council logs, but I must have missed that on the mailing list so I didn't consider it. > > The argument is that allowing the value to be some free form string is > > unnecessarily flexible. Sure. However, I'd argue that requiring it to > > be a strictly-increasing integer amounts to overspecification, especially > > when we have an established trend of using free form strings all over > > XMPP. I'd feel the same way if Dialback mandated a particular key > > format. We don't need "nanny specs". > > We do need technologies that won't break. Indeed. To clarify the point I was trying to make, imagine there are three categories for specifications: overly flexible, neutral, and restrictive. Opaque string: neutral. Strictly-increasing integer: restrictive. Compare this to the joke about stanzas: -only: neutral. or : overly flexible. What counts as what is a matter of what the usual design practices and trends are for related specifications. My assessment, from what I read in the mailing list arguments, is that opaque was in the 'neutral' category. Thus it should not be Richard that needs to argue for opaque, but rather it should be Dave that needs to argue for integer. IMO, I guess. Okay, enough of that nonsense. :) > > There's a big difference between avoiding overspecification (this) > > I'll let Dave Cridland explain why he thinks this is not > overspecification. I think his experience with IETF email and data > storage protocols is relevant here. In particular, during the Council > meeting [1] he said: > [snip] To be honest, all of this still reads like implementation notes to me, except: > "One interesting point is that with an int, there's always something a > client can use to get the entire roster, with versioning turned on." > > "I'm not quite sure what the client ought to send if everything's > completely opaque - it'd need more syntax." Maybe these could be explained here on the list. The implication in these statements is that the revision value has meaning to the client, and that the client can even create these values. -Justin
Re: [Standards] Proposed XMPP Extension: Roster Versioning
[EMAIL PROTECTED] wrote:
> Hi!
>
> I readed the latest version 0.0.3 and I'm little bit confused of
> statement (or I have
> missed something):
>
> --- clip ---
> The "roster diff" can be understood as follows:
>
>1. Imagine that the client had an active presence session for the
> entire time between its cached roster version (in this case, 305) and
> the new roster version (317).
>2. During that time, the client would have received roster pushes
> related to roster versions 306, 307, 308, 309, 310, 311, 312, 313, 314,
> 315, 316, and 317.
>3. However, some of those roster pushes might have contained
> intermediate updates to the same roster item (e.g., changes in the
> subscription state for [EMAIL PROTECTED] from "none" to "to" and from
> "to" to "both").
>4. The roster result would not include all of the intermediate steps,
> only the final result of all changes applied while the client was in
> fact offline.
>
> --- clip ---
>
> Q: How many roster version have to server remember? In the xep 0.0.3
> there is clear spec for client site implementation,
> but server site is missing "server site requirements". If I keep
> changing the roster add, remove few times
> ("on/off relationship") --> how many roster versions have to be server
> must remember backwads 1...n?
I suppose that's up to the implementation. But if the server doesn't
remember back to the version cached by the client then we need to
specify the server's behavior (probably it would return the latest
version of the entire roster, just as it does today).
Peter
--
Peter Saint-Andre
https://stpeter.im/
smime.p7s
Description: S/MIME Cryptographic Signature
Re: [Standards] Proposed XMPP Extension: Roster Versioning
Justin Karneges wrote: > On Wednesday 05 March 2008 5:33 pm, Richard Dobson wrote: >> Peter Saint-Andre wrote: >>> Alexander Gnauck wrote: Peter Saint-Andre schrieb: > Here is revised text: > >The value of the 'version' attribute SHOULD be a non-negative >integer representing a strictly increasing sequence number that >is increased with any change to the roster (whether or not the >client supports this extension) but MAY be a unique identifer that >is opaque to the client but understood by the server. In any case, >the 'version' attribute contained in roster pushes MUST be unique. great, +1 >>> The Council didn't like the change from MUST to SHOULD as discussed here: >>> >>> http://logs.jabber.org/[EMAIL PROTECTED]/2008-03-05.html#13:4 >>> 0:54 >>> >>> Discussion can continue, naturally. :) >> Oh well thats a shame, just have leave it as my own extension. >> Implemented timestamps this evening to try it out and its working fine >> thus far. A few hours of testing does not a reliable protocol make. Under what conditions? Is this a public server that people can test against? > I, too, don't understand why this value needs to be specified as being a > strictly-increasing integer, if the intent is that clients should treat it as > an opaque cookie thing anyway. I'm not convinced that that's the intent. It seems that it might be useful in some situations for the client to know where it stands. > The argument is that allowing the value to be some free form string is > unnecessarily flexible. Sure. However, I'd argue that requiring it to be a > strictly-increasing integer amounts to overspecification, especially when we > have an established trend of using free form strings all over XMPP. I'd feel > the same way if Dialback mandated a particular key format. We don't > need "nanny specs". We do need technologies that won't break. > There's a big difference between avoiding overspecification (this) I'll let Dave Cridland explain why he thinks this is not overspecification. I think his experience with IETF email and data storage protocols is relevant here. In particular, during the Council meeting [1] he said: "It's a lot more efficient with an int, and everything else is either worse performance, fails entirely, or else is equivalent difficulty to an int." "One interesting point is that with an int, there's always something a client can use to get the entire roster, with versioning turned on." "I'm not quite sure what the client ought to send if everything's completely opaque - it'd need more syntax." "Put it this way, since the counter-proposal involved timestamps, which are known to be broken, I'm pretty sure people will get stuff wrong without it being a MUST." And previously on the list he said [2]: "strictly increasing numeric sequences have proved useful in the IMAP world, because clients can spot when things go wrong much more easily." And [3]: "You can't use timestamps - they're not strictly increasing, for various reasons. Firstly, two roster changes could happen at precisely the same moment. To be fair, by introducing cluster node identifiers, and having a strict strong ordering of them, you could avoid this. Secondly, the clock on a computer can, and surprisingly often does, go backwards. That's a much harder problem to solve. Thirdly, in a clustering situation, you'd have to ensure that the time on each cluster node was perfectly synchronized. So the closest you can do would be a modified timestamp that had additional logic during generation to ensure it never went backwards, in which case you don't need the cluster identifier anymore, and that's effectively the same as having a strictly increasing integer sequence anyway, so it's easier to just do that. But even if you did want to use timestamps, just representing them as an integer is pretty trivial. Look at the definition of "modtime" in ACAP (RFC 2244), which defines a strictly increasing modified timestamp represented using digits." I don't see that these concerns were addressed on the list. And I don't see this as an appeal to authority or nannyism, just heeding the voice of experience. > and being > stupidly flexible (the joke in the council logs: "a stanza of type > SHOULD be used, but another arbitrary string MAY be used, such as > "). That was a reference to the backchannel discussions at the devcon, I think. But I wouldn't know since I wasn't privy to the backchannel. :P Peter [1] http://logs.jabber.org/[EMAIL PROTECTED]/2008-03-05.html [2] http://mail.jabber.org/pipermail/standards/2008-March/018116.html [3] http://mail.jabber.org/pipermail/standards/2008-March/018119.html smime.p7s Description: S/MIME Cryptographic Signature
Re: [Standards] Proposed XMPP Extension: Roster Versioning
On Wednesday 05 March 2008 5:33 pm, Richard Dobson wrote: > Peter Saint-Andre wrote: > > Alexander Gnauck wrote: > >> Peter Saint-Andre schrieb: > >>> Here is revised text: > >>> > >>>The value of the 'version' attribute SHOULD be a non-negative > >>>integer representing a strictly increasing sequence number that > >>>is increased with any change to the roster (whether or not the > >>>client supports this extension) but MAY be a unique identifer that > >>>is opaque to the client but understood by the server. In any case, > >>>the 'version' attribute contained in roster pushes MUST be unique. > >> > >> great, +1 > > > > The Council didn't like the change from MUST to SHOULD as discussed here: > > > > http://logs.jabber.org/[EMAIL PROTECTED]/2008-03-05.html#13:4 > >0:54 > > > > Discussion can continue, naturally. :) > > Oh well thats a shame, just have leave it as my own extension. > Implemented timestamps this evening to try it out and its working fine > thus far. I, too, don't understand why this value needs to be specified as being a strictly-increasing integer, if the intent is that clients should treat it as an opaque cookie thing anyway. The argument is that allowing the value to be some free form string is unnecessarily flexible. Sure. However, I'd argue that requiring it to be a strictly-increasing integer amounts to overspecification, especially when we have an established trend of using free form strings all over XMPP. I'd feel the same way if Dialback mandated a particular key format. We don't need "nanny specs". There's a big difference between avoiding overspecification (this) and being stupidly flexible (the joke in the council logs: "a stanza of type SHOULD be used, but another arbitrary string MAY be used, such as "). -Justin
Re: [Standards] Proposed XMPP Extension: Roster Versioning
Peter Saint-Andre wrote: Alexander Gnauck wrote: Peter Saint-Andre schrieb: Here is revised text: The value of the 'version' attribute SHOULD be a non-negative integer representing a strictly increasing sequence number that is increased with any change to the roster (whether or not the client supports this extension) but MAY be a unique identifer that is opaque to the client but understood by the server. In any case, the 'version' attribute contained in roster pushes MUST be unique. great, +1 The Council didn't like the change from MUST to SHOULD as discussed here: http://logs.jabber.org/[EMAIL PROTECTED]/2008-03-05.html#13:40:54 Discussion can continue, naturally. :) Oh well thats a shame, just have leave it as my own extension. Implemented timestamps this evening to try it out and its working fine thus far. Richard
Re: [Standards] Proposed XMPP Extension: Roster Versioning
Alexander Gnauck wrote: > Peter Saint-Andre schrieb: >> Here is revised text: >> >>The value of the 'version' attribute SHOULD be a non-negative >>integer representing a strictly increasing sequence number that >>is increased with any change to the roster (whether or not the >>client supports this extension) but MAY be a unique identifer that >>is opaque to the client but understood by the server. In any case, >>the 'version' attribute contained in roster pushes MUST be unique. > > great, +1 The Council didn't like the change from MUST to SHOULD as discussed here: http://logs.jabber.org/[EMAIL PROTECTED]/2008-03-05.html#13:40:54 Discussion can continue, naturally. :) Peter -- Peter Saint-Andre https://stpeter.im/ smime.p7s Description: S/MIME Cryptographic Signature
Re: [Standards] Proposed XMPP Extension: Roster Versioning
Hi!
Sorry missed to say that user might use several client, which caches the
roster.
Cheer's
>-Original Message-
>From: Hartikainen Juha.P (Nokia-S&S/Oulu)
>Sent: 05 March, 2008 23:37
>To: 'XMPP Extension Discussion List'
>Subject: RE: [Standards] Proposed XMPP Extension: Roster Versioning
>
>Hi!
>
>I readed the latest version 0.0.3 and I'm little bit confused
>of statement (or I have missed something):
>
>--- clip ---
>The "roster diff" can be understood as follows:
>
> 1. Imagine that the client had an active presence session
>for the entire time between its cached roster version (in this
>case, 305) and the new roster version (317).
> 2. During that time, the client would have received roster
>pushes related to roster versions 306, 307, 308, 309, 310,
>311, 312, 313, 314, 315, 316, and 317.
> 3. However, some of those roster pushes might have
>contained intermediate updates to the same roster item (e.g.,
>changes in the subscription state for [EMAIL PROTECTED]
>from "none" to "to" and from "to" to "both").
> 4. The roster result would not include all of the
>intermediate steps, only the final result of all changes
>applied while the client was in fact offline.
>
>--- clip ---
>
>Q: How many roster version have to server remember? In the xep
>0.0.3 there is clear spec for client site implementation, but
>server site is missing "server site requirements". If I keep
>changing the roster add, remove few times ("on/off
>relationship") --> how many roster versions have to be server
>must remember backwads 1...n?
>
>Cheer's
>
>
>>-Original Message-
>>From: [EMAIL PROTECTED]
>>[mailto:[EMAIL PROTECTED] On Behalf Of ext Alexander Gnauck
>>Sent: 05 March, 2008 22:45
>>To: standards@xmpp.org
>>Subject: Re: [Standards] Proposed XMPP Extension: Roster Versioning
>>
>>Peter Saint-Andre schrieb:
>>> Here is revised text:
>>>
>>>The value of the 'version' attribute SHOULD be a non-negative
>>>integer representing a strictly increasing sequence number that
>>>is increased with any change to the roster (whether or not the
>>>client supports this extension) but MAY be a unique
>identifer that
>>>is opaque to the client but understood by the server. In
>any case,
>>>the 'version' attribute contained in roster pushes MUST
>be unique.
>>
>>great, +1
>>
>>> Can we stop painting this bike shed and move on? :)
>>
>>yes
>>
>>Alex
>>
>>
Re: [Standards] Proposed XMPP Extension: Roster Versioning
Hi!
I readed the latest version 0.0.3 and I'm little bit confused of
statement (or I have
missed something):
--- clip ---
The "roster diff" can be understood as follows:
1. Imagine that the client had an active presence session for the
entire time between its cached roster version (in this case, 305) and
the new roster version (317).
2. During that time, the client would have received roster pushes
related to roster versions 306, 307, 308, 309, 310, 311, 312, 313, 314,
315, 316, and 317.
3. However, some of those roster pushes might have contained
intermediate updates to the same roster item (e.g., changes in the
subscription state for [EMAIL PROTECTED] from "none" to "to" and from
"to" to "both").
4. The roster result would not include all of the intermediate steps,
only the final result of all changes applied while the client was in
fact offline.
--- clip ---
Q: How many roster version have to server remember? In the xep 0.0.3
there is clear spec for client site implementation,
but server site is missing "server site requirements". If I keep
changing the roster add, remove few times
("on/off relationship") --> how many roster versions have to be server
must remember backwads 1...n?
Cheer's
>-Original Message-
>From: [EMAIL PROTECTED]
>[mailto:[EMAIL PROTECTED] On Behalf Of ext Alexander Gnauck
>Sent: 05 March, 2008 22:45
>To: standards@xmpp.org
>Subject: Re: [Standards] Proposed XMPP Extension: Roster Versioning
>
>Peter Saint-Andre schrieb:
>> Here is revised text:
>>
>>The value of the 'version' attribute SHOULD be a non-negative
>>integer representing a strictly increasing sequence number that
>>is increased with any change to the roster (whether or not the
>>client supports this extension) but MAY be a unique identifer that
>>is opaque to the client but understood by the server. In any case,
>>the 'version' attribute contained in roster pushes MUST be unique.
>
>great, +1
>
>> Can we stop painting this bike shed and move on? :)
>
>yes
>
>Alex
>
>
Re: [Standards] Proposed XMPP Extension: Roster Versioning
Peter Saint-Andre schrieb: Here is revised text: The value of the 'version' attribute SHOULD be a non-negative integer representing a strictly increasing sequence number that is increased with any change to the roster (whether or not the client supports this extension) but MAY be a unique identifer that is opaque to the client but understood by the server. In any case, the 'version' attribute contained in roster pushes MUST be unique. great, +1 Can we stop painting this bike shed and move on? :) yes Alex
Re: [Standards] Proposed XMPP Extension: Roster Versioning
Peter Saint-Andre wrote: > If we change it to a SHOULD will that make people happier? We do > something similar in server dialback (recommend an algorithm for key > generation using HMAC-SHA256, but say you can use something else). Here is revised text: The value of the 'version' attribute SHOULD be a non-negative integer representing a strictly increasing sequence number that is increased with any change to the roster (whether or not the client supports this extension) but MAY be a unique identifer that is opaque to the client but understood by the server. In any case, the 'version' attribute contained in roster pushes MUST be unique. Can we stop painting this bike shed and move on? :) Peter -- Peter Saint-Andre https://stpeter.im/ smime.p7s Description: S/MIME Cryptographic Signature
Re: [Standards] Proposed XMPP Extension: Roster Versioning
Richard Dobson wrote: >> You just use a 128-bit unsigned integer. There is no upper limit here >> - in particular, there is no upper limit specified anywhere in this >> document - XSD merely states that a xs:nonNegativeInteger is a >> sequence of digits, and has "countably infinite" cardinality. >> >> If you really and truly believe that practical limits of 64-bit >> unsigned integers can cause problems in the real world, I honestly >> don't know what to say except show you the figures - you could have >> thousands of updates every millisecond, and still last over half a >> million years - 574,542 roughly, assuming a fixed year length of >> 365.25 days. >> >> I'm all for designing for the future, but you have to draw the line >> somewhere, and besides, I figure we'll be on something bigger than >> 64-bit well before then - a jump to 128-bit gains us 10^25 years of >> breathing space, and I'd like to imagine we can think up a solution >> within that time, assuming that's prior to the heat death of the >> universe. > Sure you can keep increasing the bit size of your integer in your > implementation, but the spec still needs to dictate what happens once > you reach overflow In half a million years we'll be on XMPP version 168.9, won't we? ;-) Why are we designing for such an eventuality? > if its going to define that you have to implement it > that way as well as how long the integer should be, although I still > fail to see why strictly increasing numbers should be a MUST at the > protocol level, IMO this is an internal server implementation issue and > not a protocol one, im all for recommending that as a way for the server > to implement it, but still don't think it should be MUST only RECOMMENDED. If we change it to a SHOULD will that make people happier? We do something similar in server dialback (recommend an algorithm for key generation using HMAC-SHA256, but say you can use something else). More arguments for using "a non-negative integer representing a strictly increasing sequence number that is increased with any change to the roster"... 1. Code reuse 2. Server migration (yes, it happens, and it's painful enough as it is without having to figure out a way to migrate opaque GUIDs from your old implementation into strictly increasing sequence numbers in your new implementation) Peter -- Peter Saint-Andre https://stpeter.im/ smime.p7s Description: S/MIME Cryptographic Signature
Re: [Standards] Proposed XMPP Extension: Roster Versioning
You can't use timestamps - they're not strictly increasing, for various reasons. Why does it need to be strictly increasing? As already explained the version identifiers should IMO be opaque and just be a server implementation issue, I still can't see any reason why this needs to be set in stone the protocol as a MUST, should only be a RECOMMENDED so that server implementors have an idea of where to start from as a way to implement this. Firstly, two roster changes could happen at precisely the same moment. To be fair, by introducing cluster node identifiers, and having a strict strong ordering of them, you could avoid this. It could do yes, why is it a problem if they happen at the same time and are marked with the same timestamp?, it will just result in them both being pushed, how is that an issue?. Secondly, the clock on a computer can, and surprisingly often does, go backwards. That's a much harder problem to solve. Maybe so, do you have any more information on how prevalent this is? How long it lasts for etc? Thirdly, in a clustering situation, you'd have to ensure that the time on each cluster node was perfectly synchronized. No you wouldn't necessarily, not if the timestamping was happening at the central data storage layer (i.e. the database server), and again this is just an implementation issue and is easily overcome, not something that means its impossible. So the closest you can do would be a modified timestamp that had additional logic during generation to ensure it never went backwards, in which case you don't need the cluster identifier anymore, and that's effectively the same as having a strictly increasing integer sequence anyway, so it's easier to just do that. But even if you did want to use timestamps, just representing them as an integer is pretty trivial. Look at the definition of "modtime" in ACAP (RFC 2244), which defines a strictly increasing modified timestamp represented using digits. Yes I know I could represent them as integers, but id rather not if I don't have do, id prefer to have the flexibility to compress and shorten them to reduce bandwidth consumption as much as possible. It's useful for clients to be able to determine the ordering locally, on occasion. If we removed this, we'd also have to ensure that roster pushes were sent to the client in-order, which currently we don't mandate. (Making this a SHOULD is sane, but in the cluster case, it's quite hard). Well im pretty sure XMPP dictates in order processing of stanzas so surely the roster updates should thus be in order? Also you haven't really answered my concerns about allowing clients to determine meaning from the version identifier introducing the possibility of bugs and interoperability problems which IMO is a far more serious issue, and one that doesn't exist if the client just treats them as opaque strings. Also even if they were out of order (which I think would be unlikely because it would be only likely to happen when several roster updates were happening at the same time) it doesn't really cause much of an issue as far as I can see, its just that you might have one or two roster pushes that you have already cached pushed to you again, hardly the end of the world, and should be something the clients should be able to cope with, as what would happen if a servers database server crashed and needed to be restored from a backup and the most recent roster updates that have already been pushed arn't there, or the server now thinks it hasn't pushed the changes yet and ends up re-pushing changes, it shouldn't make any difference to the client, some method to re-synchronize needs to be in place to handle this, I think to solve this issue if the version id (be that timestamp or incrementing id) the client specifies is further on than any of the ones the server has you would need to re-push the entire list the server has invalidating the client list somehow (to ensure new now non-existent contacts that were created in between the db backup and the crash do not hang around). Plus, nobody can get it wrong. How exactly are they going to get it wrong if its an identifier that only the server is interpreting the meaning of? It's the server I'm worried about. :-) OK but that doesn't really answer my question. You just use a 128-bit unsigned integer. There is no upper limit here - in particular, there is no upper limit specified anywhere in this document - XSD merely states that a xs:nonNegativeInteger is a sequence of digits, and has "countably infinite" cardinality. If you really and truly believe that practical limits of 64-bit unsigned integers can cause problems in the real world, I honestly don't know what to say except show you the figures - you could have thousands of updates every millisecond, and still last over half a million years - 574,542 roughly, assuming a fixed year length of 365.25 days. I'm all for designing for the future, but yo
Re: [Standards] Proposed XMPP Extension: Roster Versioning
On Wed Mar 5 11:00:34 2008, Richard Dobson wrote: Sometimes flexibility comes back to bite you. I'd prefer to keep things simple if we can. What does the extra flexibility really do for us, and is it worth the cost? Well I think it is better to be flexible in this case otherwise its forcing server implementors to implement this in a particular way when there is no real need to, what if in a particular implementation its more efficient to implement it as GUIDs for example as Alex suggested because of how its clustered or how the database is implemented, in my case id want to implement this as a compressed timestamp rather than as an increasing integer, and as far as the spec is currently written and how it would work for clients it wouldn't make any difference what the version identifier is as it isn't (and IMO MUST NOT) take any meaning from the value of the version identifier so things stay nice and simple, as as soon as clients start taking any meaning of what the version identifier means you are introducing a whole raft of potential interoperability issues and bugs that need not exist. You can't use timestamps - they're not strictly increasing, for various reasons. Firstly, two roster changes could happen at precisely the same moment. To be fair, by introducing cluster node identifiers, and having a strict strong ordering of them, you could avoid this. Secondly, the clock on a computer can, and surprisingly often does, go backwards. That's a much harder problem to solve. Thirdly, in a clustering situation, you'd have to ensure that the time on each cluster node was perfectly synchronized. So the closest you can do would be a modified timestamp that had additional logic during generation to ensure it never went backwards, in which case you don't need the cluster identifier anymore, and that's effectively the same as having a strictly increasing integer sequence anyway, so it's easier to just do that. But even if you did want to use timestamps, just representing them as an integer is pretty trivial. Look at the definition of "modtime" in ACAP (RFC 2244), which defines a strictly increasing modified timestamp represented using digits. The only reason this scares me is that strictly increasing numeric sequences have proved useful in the IMAP world, because clients can spot when things go wrong much more easily. I think this its a very very bad idea for the clients to take any meaning from the version identifier as explained above, its just opening a pandora's box of potential bugs and issues, far better for it to just be a opaque string as far as the client in concerned, which also helps to keep things as simple as possible. It's useful for clients to be able to determine the ordering locally, on occasion. If we removed this, we'd also have to ensure that roster pushes were sent to the client in-order, which currently we don't mandate. (Making this a SHOULD is sane, but in the cluster case, it's quite hard). Plus, nobody can get it wrong. How exactly are they going to get it wrong if its an identifier that only the server is interpreting the meaning of? It's the server I'm worried about. :-) There's no way that even a 32-bit unsigned integer is going to overflow - if you did an update every second, it'd take 136 years - but if that still unnerves you (in case PSA turns into the undead, or something), use a 64-bit unsigned integer. Its possible even if unlikely, what if several updates were made in one second because of some kind of bulk update for example, at the very least the spec MUST define what should happen if the number is going to overflow, i.e. reseting to 0, even though it is unlikely. You just use a 128-bit unsigned integer. There is no upper limit here - in particular, there is no upper limit specified anywhere in this document - XSD merely states that a xs:nonNegativeInteger is a sequence of digits, and has "countably infinite" cardinality. If you really and truly believe that practical limits of 64-bit unsigned integers can cause problems in the real world, I honestly don't know what to say except show you the figures - you could have thousands of updates every millisecond, and still last over half a million years - 574,542 roughly, assuming a fixed year length of 365.25 days. I'm all for designing for the future, but you have to draw the line somewhere, and besides, I figure we'll be on something bigger than 64-bit well before then - a jump to 128-bit gains us 10^25 years of breathing space, and I'd like to imagine we can think up a solution within that time, assuming that's prior to the heat death of the universe. Dave. -- Dave Cridland - mailto:[EMAIL PROTECTED] - xmpp:[EMAIL PROTECTED] - acap://acap.dave.cridland.net/byowner/user/dwd/bookmarks/ - http://dave.cridland.net/ Infotrope Polymer - ACAP, IMAP, ESMTP, and Lemonade
Re: [Standards] Proposed XMPP Extension: Roster Versioning
yes I think we should recommend a an increasing integer. But should allow any string. So if some server vendor prefers hash codes or GUIDs for the versioning then this is fine for me too. Exactly. Sometimes flexibility comes back to bite you. I'd prefer to keep things simple if we can. What does the extra flexibility really do for us, and is it worth the cost? Well I think it is better to be flexible in this case otherwise its forcing server implementors to implement this in a particular way when there is no real need to, what if in a particular implementation its more efficient to implement it as GUIDs for example as Alex suggested because of how its clustered or how the database is implemented, in my case id want to implement this as a compressed timestamp rather than as an increasing integer, and as far as the spec is currently written and how it would work for clients it wouldn't make any difference what the version identifier is as it isn't (and IMO MUST NOT) take any meaning from the value of the version identifier so things stay nice and simple, as as soon as clients start taking any meaning of what the version identifier means you are introducing a whole raft of potential interoperability issues and bugs that need not exist. The only reason this scares me is that strictly increasing numeric sequences have proved useful in the IMAP world, because clients can spot when things go wrong much more easily. I think this its a very very bad idea for the clients to take any meaning from the version identifier as explained above, its just opening a pandora's box of potential bugs and issues, far better for it to just be a opaque string as far as the client in concerned, which also helps to keep things as simple as possible. Plus, nobody can get it wrong. How exactly are they going to get it wrong if its an identifier that only the server is interpreting the meaning of? There's no way that even a 32-bit unsigned integer is going to overflow - if you did an update every second, it'd take 136 years - but if that still unnerves you (in case PSA turns into the undead, or something), use a 64-bit unsigned integer. Its possible even if unlikely, what if several updates were made in one second because of some kind of bulk update for example, at the very least the spec MUST define what should happen if the number is going to overflow, i.e. reseting to 0, even though it is unlikely. Richard
Re: [Standards] Proposed XMPP Extension: Roster Versioning
On Tue Mar 4 23:48:53 2008, Pedro Melo wrote:
On Mar 4, 2008, at 10:27 PM, Dave Cridland wrote:
"The value of the 'version' attribute is an integer value which is
increased with any change to the roster, whether or not the
client supports this extension. In particular, the 'version'
attribute contained in roster pushes (see Section 2.5) MUST be
unique."
+1. I can't see any reason for the spec to require more than a
increasing version number.
The key phrase "strictly increasing sequence number" needs to be
present somewhere, to keep mathematicians happy. ("Monotonically
increasing" varies in its meaning in different parts of the world,
bizarrely, so there's a phrase to avoid).
Section 2.2:
Include some text saying that clients can set version to 0 if they
have no roster cached with a version - otherwise there's no
bootstrap.
Or omit the version maybe?
No, omitting the version means "I'd like the full roster and I do not
understand versions".
Section 2.4 needs to include a method for servers to indicate
whether this is an update, or a complete replacement.
Why? A complete replacement is just a full set of updates. The
final XML would be equivalent. From the client perspective, he
would replace all the cached local entries.
No, it isn't. There's that "removed" value for subscription. In the
case where a full update is needed, it's quite likely that one of the
reasons is because the server can't remember all (or any) deletions.
If a full update looks the same as a diff, then a client cannot know
to remove otherwise undead entries.
Finally, I'd include an assertion for roster sets - so that a
client can perform actions such as a "set if not changed since".
Dead easy for servers, I think.
I don't follow what this is. Can you elaborate with a use case?
It is, as PSA suggested, just a convenient place to put a potentially
cool feature. I'm far from wed to it, but it'd be nice to at least
keep it in mind.
But, let's say that you had a bot which processed all entries in it's
roster. Let's further complicate things by having multiple instances
of the bot, because it's a CPU heavy bot. Now, it can do work by:
1) Finding all entries not in the "Processed" group.
2) Doing a conditional modify of the roster, placing an entry into a
"Processing" group.
3) If that succeeds, process, then place it (unconditionally) into
the "Processed" group.
4) GOTO 1 (Gotta love that BASIC).
There's far simpler use-cases, such as adding a group. (The client
cannot add a group, it can merely change groups, so to add one
without potentially removing a newly added one, it must ensure
nothing else has changed the roster since it last operated.)
For both these use-cases, it would imply that we could specify this
slightly tighter, and insist that each roster entry has a sequence
value visible to the client. I can't think of a sane server
implementation that wouldn't do this anyway.
Dave.
--
Dave Cridland - mailto:[EMAIL PROTECTED] - xmpp:[EMAIL PROTECTED]
- acap://acap.dave.cridland.net/byowner/user/dwd/bookmarks/
- http://dave.cridland.net/
Infotrope Polymer - ACAP, IMAP, ESMTP, and Lemonade
Re: [Standards] Proposed XMPP Extension: Roster Versioning
On Wed Mar 5 07:07:20 2008, Alexander Gnauck wrote: Richard Dobson schrieb: +1. I can't see any reason for the spec to require more than a increasing version number. I would prefer if it were just an opaque string, certainly as far as the client in concerned there is no need for it to do anything other than store the most recent version identifier it has received and then return that to the server when required (i.e. at login), only the server needs to know what to do with it and this should certainly only be RECOMMENDED or SUGGESTED/MAYBE and not MUST. What would happen in cases were the version number needs to be reset to 0 because someone has such a busy roster that over time they exhast the maximum integer value?, or if maybe in future server implementors want to compress it somehow into hex or something similar, it would be far far better to leave this flexible and upto the server implementor on how they format the version identifier. yes I think we should recommend a an increasing integer. But should allow any string. So if some server vendor prefers hash codes or GUIDs for the versioning then this is fine for me too. The only reason this scares me is that strictly increasing numeric sequences have proved useful in the IMAP world, because clients can spot when things go wrong much more easily. Plus, nobody can get it wrong. There's no way that even a 32-bit unsigned integer is going to overflow - if you did an update every second, it'd take 136 years - but if that still unnerves you (in case PSA turns into the undead, or something), use a 64-bit unsigned integer. Dave. -- Dave Cridland - mailto:[EMAIL PROTECTED] - xmpp:[EMAIL PROTECTED] - acap://acap.dave.cridland.net/byowner/user/dwd/bookmarks/ - http://dave.cridland.net/ Infotrope Polymer - ACAP, IMAP, ESMTP, and Lemonade
Re: [Standards] Proposed XMPP Extension: Roster Versioning
Richard Dobson schrieb: +1. I can't see any reason for the spec to require more than a increasing version number. I would prefer if it were just an opaque string, certainly as far as the client in concerned there is no need for it to do anything other than store the most recent version identifier it has received and then return that to the server when required (i.e. at login), only the server needs to know what to do with it and this should certainly only be RECOMMENDED or SUGGESTED/MAYBE and not MUST. What would happen in cases were the version number needs to be reset to 0 because someone has such a busy roster that over time they exhast the maximum integer value?, or if maybe in future server implementors want to compress it somehow into hex or something similar, it would be far far better to leave this flexible and upto the server implementor on how they format the version identifier. yes I think we should recommend a an increasing integer. But should allow any string. So if some server vendor prefers hash codes or GUIDs for the versioning then this is fine for me too. Alex
Re: [Standards] Proposed XMPP Extension: Roster Versioning
Peter Saint-Andre wrote: >>> URL: http://www.xmpp.org/extensions/inbox/roster-versioning.html BTW version 0.0.2 is available. Reload as needed. /psa smime.p7s Description: S/MIME Cryptographic Signature
Re: [Standards] Proposed XMPP Extension: Roster Versioning
Richard Dobson wrote: >> +1. I can't see any reason for the spec to require more than a >> increasing version number. > > I would prefer if it were just an opaque string, certainly as far as the > client in concerned there is no need for it to do anything other than > store the most recent version identifier it has received and then return > that to the server when required (i.e. at login), only the server needs > to know what to do with it and this should certainly only be RECOMMENDED > or SUGGESTED/MAYBE and not MUST. Perhaps, I'll ponder that some more... > What would happen in cases were the > version number needs to be reset to 0 because someone has such a busy > roster that over time they exhast the maximum integer value? Even my roster isn't that busy. But yes it is *possible* I suppose. > , or if > maybe in future server implementors want to compress it somehow into hex > or something similar, it would be far far better to leave this flexible > and upto the server implementor on how they format the version identifier. Sometimes flexibility comes back to bite you. I'd prefer to keep things simple if we can. What does the extra flexibility really do for us, and is it worth the cost? Peter -- Peter Saint-Andre https://stpeter.im/ smime.p7s Description: S/MIME Cryptographic Signature
Re: [Standards] Proposed XMPP Extension: Roster Versioning
Dave Cridland wrote: > On Tue Mar 4 21:54:31 2008, XMPP Extensions Editor wrote: >> The XMPP Extensions Editor has received a proposal for a new XEP. >> >> Title: Roster Versioning >> >> Abstract: This specification proposes a modification to the XMPP >> roster management protocol to support versioning of rosters for more >> efficient downloading of the rost er information. >> >> URL: http://www.xmpp.org/extensions/inbox/roster-versioning.html > > "Note: This document is provided for discussion purposes in order to > clarify the usage of SASL ANONYMOUS in XMPP systems. It is not meant to > supersede the text in RFC 3920, RFC 4422, or RFC 4505. However, the > recommendations in this document may be folded into rfc3920bis." > > Good to know that's cleared up, anyway. :-) Yeah yeah yeah, copy and paste error, always working too fast... > "The value of the 'version' attribute MUST start at zero (0) and MUST be > incremented by one (1) for each change to the roster." > > I would personally prefer that this read: > > "The value of the 'version' attribute is an integer value which is > increased with any change to the roster, whether or not the client > supports this extension. In particular, the 'version' attribute > contained in roster pushes (see Section 2.5) MUST be unique." > > (Insert RFC 2119 language to suit). > > My reasoning is: > > a) Servers might need to increase it by more than 1 for some changes, if > they're not performed atomically. > > b) Servers may need to increase it for changes which are not > user-visible, such as the from-pending state. > > c) We might need to change it in the future for other purposes. Sure, that seems reasonable. > Section 2.2: > > Include some text saying that clients can set version to 0 if they have > no roster cached with a version - otherwise there's no bootstrap. Ah, good point. > Sections 2.4/2.5 contain no closing , and don't show groups, > etc. Both need to specify that the complete roster item is used. Right, I left out the groups because I'm lazy. I'll add a few in. > Section 2.4 needs to include a method for servers to indicate whether > this is an update, or a complete replacement. I defined a boolean 'diff' attribute for this purpose. > Finally, I'd include an assertion for roster sets - so that a client can > perform actions such as a "set if not changed since". Dead easy for > servers, I think. Is that really necessary for the intended purpose (optimization of roster gets)? Or is it just a cool feature that piggybacks on the optimization use case? Peter -- Peter Saint-Andre https://stpeter.im/ smime.p7s Description: S/MIME Cryptographic Signature
Re: [Standards] Proposed XMPP Extension: Roster Versioning
+1. I can't see any reason for the spec to require more than a increasing version number. I would prefer if it were just an opaque string, certainly as far as the client in concerned there is no need for it to do anything other than store the most recent version identifier it has received and then return that to the server when required (i.e. at login), only the server needs to know what to do with it and this should certainly only be RECOMMENDED or SUGGESTED/MAYBE and not MUST. What would happen in cases were the version number needs to be reset to 0 because someone has such a busy roster that over time they exhast the maximum integer value?, or if maybe in future server implementors want to compress it somehow into hex or something similar, it would be far far better to leave this flexible and upto the server implementor on how they format the version identifier. Richard
Re: [Standards] Proposed XMPP Extension: Roster Versioning
Hi, On Mar 4, 2008, at 10:27 PM, Dave Cridland wrote: On Tue Mar 4 21:54:31 2008, XMPP Extensions Editor wrote: The XMPP Extensions Editor has received a proposal for a new XEP. Title: Roster Versioning Abstract: This specification proposes a modification to the XMPP roster management protocol to support versioning of rosters for more efficient downloading of the rost er information. URL: http://www.xmpp.org/extensions/inbox/roster-versioning.html [...] "The value of the 'version' attribute MUST start at zero (0) and MUST be incremented by one (1) for each change to the roster." I would personally prefer that this read: "The value of the 'version' attribute is an integer value which is increased with any change to the roster, whether or not the client supports this extension. In particular, the 'version' attribute contained in roster pushes (see Section 2.5) MUST be unique." (Insert RFC 2119 language to suit). My reasoning is: a) Servers might need to increase it by more than 1 for some changes, if they're not performed atomically. b) Servers may need to increase it for changes which are not user- visible, such as the from-pending state. c) We might need to change it in the future for other purposes.\ +1. I can't see any reason for the spec to require more than a increasing version number. Section 2.2: Include some text saying that clients can set version to 0 if they have no roster cached with a version - otherwise there's no bootstrap. Or omit the version maybe? Sections 2.4/2.5 contain no closing , and don't show groups, etc. Both need to specify that the complete roster item is used. +1 for groups. Section 2.4 needs to include a method for servers to indicate whether this is an update, or a complete replacement. Why? A complete replacement is just a full set of updates. The final XML would be equivalent. From the client perspective, he would replace all the cached local entries. Finally, I'd include an assertion for roster sets - so that a client can perform actions such as a "set if not changed since". Dead easy for servers, I think. I don't follow what this is. Can you elaborate with a use case? Best regards, -- Pedro Melo Blog: http://www.simplicidade.org/notes/ XMPP ID: [EMAIL PROTECTED] Use XMPP!
Re: [Standards] Proposed XMPP Extension: Roster Versioning
On Tue Mar 4 21:54:31 2008, XMPP Extensions Editor wrote: The XMPP Extensions Editor has received a proposal for a new XEP. Title: Roster Versioning Abstract: This specification proposes a modification to the XMPP roster management protocol to support versioning of rosters for more efficient downloading of the rost er information. URL: http://www.xmpp.org/extensions/inbox/roster-versioning.html "Note: This document is provided for discussion purposes in order to clarify the usage of SASL ANONYMOUS in XMPP systems. It is not meant to supersede the text in RFC 3920, RFC 4422, or RFC 4505. However, the recommendations in this document may be folded into rfc3920bis." Good to know that's cleared up, anyway. :-) "The value of the 'version' attribute MUST start at zero (0) and MUST be incremented by one (1) for each change to the roster." I would personally prefer that this read: "The value of the 'version' attribute is an integer value which is increased with any change to the roster, whether or not the client supports this extension. In particular, the 'version' attribute contained in roster pushes (see Section 2.5) MUST be unique." (Insert RFC 2119 language to suit). My reasoning is: a) Servers might need to increase it by more than 1 for some changes, if they're not performed atomically. b) Servers may need to increase it for changes which are not user-visible, such as the from-pending state. c) We might need to change it in the future for other purposes. Section 2.2: Include some text saying that clients can set version to 0 if they have no roster cached with a version - otherwise there's no bootstrap. Sections 2.4/2.5 contain no closing , and don't show groups, etc. Both need to specify that the complete roster item is used. Section 2.4 needs to include a method for servers to indicate whether this is an update, or a complete replacement. Finally, I'd include an assertion for roster sets - so that a client can perform actions such as a "set if not changed since". Dead easy for servers, I think. Dave. -- Dave Cridland - mailto:[EMAIL PROTECTED] - xmpp:[EMAIL PROTECTED] - acap://acap.dave.cridland.net/byowner/user/dwd/bookmarks/ - http://dave.cridland.net/ Infotrope Polymer - ACAP, IMAP, ESMTP, and Lemonade
Re: [Standards] Proposed XMPP Extension: Roster Versioning
XMPP Extensions Editor wrote: > The XMPP Extensions Editor has received a proposal for a new XEP. > > Title: Roster Versioning > > Abstract: This specification proposes a modification to the XMPP > roster management protocol to support versioning of rosters for more > efficient downloading of the rost er information. > > URL: http://www.xmpp.org/extensions/inbox/roster-versioning.html This is the first of several proposals related to mobile optimization (and just general optimization) as discussed at the recent devcon. More to follow... Peter -- Peter Saint-Andre https://stpeter.im/ smime.p7s Description: S/MIME Cryptographic Signature
