Re: [Standards] Proposed XMPP Extension: Unique and stable message IDs
On an abstract level I feel like using the by attribute for the Message-ID-XEP will make the XEP more general purpose. Even though we might not necessarily see a benefit for multiple instances adding their own IDs now we might have a use case for them in the future. As for letting the client choose the MAM id vs agreeing on some kind of schema I'm all for the schema since it does not require for elaborate business rules on how to handle duplicate ids an unknowing or malicious clients might inject. Matthew is right in arguing that a remote entity might not know our stream id however those messages (let's face it we are talking about MUC here) will get reflected back to us anyway and thus allowing the remote entity to add their own message id. MUC is also a great example where the by attribute might come in handy. It allows for both the MUC component and the users server to both add their own message-id without requiring complicated business rules such as 'if its a normal message strip all message ids and add your own but don't do this when its a group chat message'. It also allows for MAM implementations to optionally save group messages on behalf of the server (current XEP says optionally) cheers Daniel P.S.: I don't have a strong opinion on the MAM id generation schema (And this doesn't belong to the message-id-discussion but should be in a separate MAM discussion since it has nothing to do with the actual message-id XEP besides the fact that it will be using this XEP when broadcasting messages via the MR2-XEP) but using something like SHA256(streamId+stanzaId) will have the benefit of not requiring and extra id and would also make all ids in the archive of the same length which might make it easier when looking at the database or an xml stream in a debugging context. 2015-06-07 23:14 GMT+02:00 Matthew Wild : > On 7 June 2015 at 21:39, Florian Schmaus wrote: > > On 07.06.2015 20:42, Georg Lukas wrote: > >> 3) Create a MID generation scheme that can be independently followed by > >> client and server, i.e. full-jid + stream-id + packet-id. > > > > I'd don't think we'll need the full-jid. The properties of the stream-id > > are sufficient to generate a unique, collision free MID by concatenating > > it with the stanza-id. > > > > We may not be able to use the stream-id though, because of the RFC > > requirement that it has to be kept secret. Thus requiring us to > > introduce a new public-stream-id which has the same properties as the > > stream-id, but without the 'secret' requirement. > > > >> This scheme is > >> very similar to #2, but the client has less options to game it, and I > >> don't see significant benefits over #2. > > > > "Less options to game it" is not a significant benefit? Also not > > requiring UUIDs as stanza-id is also a big plus: A dump client can still > > generate stanza-ids using a simple counter and still participate. > > > > > >> I think these options need to be weighted and one of them chosen before > >> we can proceed with the MID XEP. > > > > I'm all for 3. It solves the "how can a client know the MAM archive ID > > in advance" problem in an elegant and efficient way. > > I like it too, and I hope Dave can throw together the proposal he's > been promising for it :) > > I feel it still has some shortcomings though: > > - It only works for server-local archives (as remote entities won't > know your stream id) > > - It removes the ability for the server to use its own ids for > messages, which may be more efficient > > - If it's based on stanza ids, we have to verify uniqueness somehow > (and gracefully fail on non-uniqueness) > > - If it's based on counters then uniqueness is not a problem, but > XEP-0198 implementation experience has demonstrated that simple > counters apparently aren't so simple after all > > - It removes the need for reflection of all sent messages, but there > is potentially benefit in that practice (people generally like it in > MUC) > > Regards, > Matthew >
Re: [Standards] Proposed XMPP Extension: Unique and stable message IDs
On 7 June 2015 at 21:39, Florian Schmaus wrote: > On 07.06.2015 20:42, Georg Lukas wrote: >> 3) Create a MID generation scheme that can be independently followed by >> client and server, i.e. full-jid + stream-id + packet-id. > > I'd don't think we'll need the full-jid. The properties of the stream-id > are sufficient to generate a unique, collision free MID by concatenating > it with the stanza-id. > > We may not be able to use the stream-id though, because of the RFC > requirement that it has to be kept secret. Thus requiring us to > introduce a new public-stream-id which has the same properties as the > stream-id, but without the 'secret' requirement. > >> This scheme is >> very similar to #2, but the client has less options to game it, and I >> don't see significant benefits over #2. > > "Less options to game it" is not a significant benefit? Also not > requiring UUIDs as stanza-id is also a big plus: A dump client can still > generate stanza-ids using a simple counter and still participate. > > >> I think these options need to be weighted and one of them chosen before >> we can proceed with the MID XEP. > > I'm all for 3. It solves the "how can a client know the MAM archive ID > in advance" problem in an elegant and efficient way. I like it too, and I hope Dave can throw together the proposal he's been promising for it :) I feel it still has some shortcomings though: - It only works for server-local archives (as remote entities won't know your stream id) - It removes the ability for the server to use its own ids for messages, which may be more efficient - If it's based on stanza ids, we have to verify uniqueness somehow (and gracefully fail on non-uniqueness) - If it's based on counters then uniqueness is not a problem, but XEP-0198 implementation experience has demonstrated that simple counters apparently aren't so simple after all - It removes the need for reflection of all sent messages, but there is potentially benefit in that practice (people generally like it in MUC) Regards, Matthew
Re: [Standards] Proposed XMPP Extension: Unique and stable message IDs
On 07.06.2015 20:42, Georg Lukas wrote: > 3) Create a MID generation scheme that can be independently followed by > client and server, i.e. full-jid + stream-id + packet-id. I'd don't think we'll need the full-jid. The properties of the stream-id are sufficient to generate a unique, collision free MID by concatenating it with the stanza-id. We may not be able to use the stream-id though, because of the RFC requirement that it has to be kept secret. Thus requiring us to introduce a new public-stream-id which has the same properties as the stream-id, but without the 'secret' requirement. > This scheme is > very similar to #2, but the client has less options to game it, and I > don't see significant benefits over #2. "Less options to game it" is not a significant benefit? Also not requiring UUIDs as stanza-id is also a big plus: A dump client can still generate stanza-ids using a simple counter and still participate. > I think these options need to be weighted and one of them chosen before > we can proceed with the MID XEP. I'm all for 3. It solves the "how can a client know the MAM archive ID in advance" problem in an elegant and efficient way. - Florian signature.asc Description: OpenPGP digital signature
Re: [Standards] Proposed XMPP Extension: Unique and stable message IDs
* Daniel Gultsch [2015-06-05 13:01]: > And I'm currently not really seeing the point of a client adding an id > since the client can already set the stanza id. Besides of the MUC-eating-your-message-ID use case there is another one that was heavily discussed as a possible motivation for client-id: when a MAM-capable client sends out a message, it needs to somehow obtain the MAM-ID for this message. Several imperfect approaches are possible: 1) Let the server (or MAM entity) reflect the message to the sender, with a MID element added. This can happen in the form of a full message, a carbon, or as a stub message only containing the MID and the original message id, so the client can relate them. Either way, this will duplicate the number of messages for local senders. 2) Let the client create the MID element, e.g. by writing into the spec that the MID element MUST be a UUID. That way, the MAM entity will be able to just add the message to its archive with the client-generated MID. However, an evil client could create MID collisions, degenerate hash-tables to O(N), or simply be broken due to accessing random.org without HTTPS. Besides, letting the MAM entity generate MIDs might allow more efficient implementations (i.e. autoincrement-indices). 3) Create a MID generation scheme that can be independently followed by client and server, i.e. full-jid + stream-id + packet-id. This scheme is very similar to #2, but the client has less options to game it, and I don't see significant benefits over #2. I think these options need to be weighted and one of them chosen before we can proceed with the MID XEP. Georg -- || http://op-co.de ++ GCS d--(++) s: a C+++ UL+++ !P L+++ !E W+++ N ++ || gpg: 0x962FD2DE || o? K- w---() O M V? PS+ PE-- Y++ PGP+ t+ 5 R+ || || Ge0rG: euIRCnet || X(+++) tv+ b+(++) DI+++ D- G e h- r++ y? || ++ IRCnet OFTC OPN ||_|| signature.asc Description: Digital signature
Re: [Standards] Proposed XMPP Extension: Unique and stable message IDs
On 04.06.2015 12:56, Kurt Zeilenga wrote: > It’s not clear to me that the problems that this extensions proposes to > address can be addressed through use an extension to XMPP. Extensions > ought to be truly optional. This ProtoXEP appears to be making mandates > which are more appropriate made, if the community were to support them > being made, in a revision of the XMPP base specification. Possible, but a revision of the XMPP RFC changing the semantic and requirements of the 'id' attribute of stanzas in such a way won't ever happen I'd say. Yes, XEP-MID breaks if entities strip it out. But I can live with that. > I have a big problem with one ID to rule them all… and a bigger problem > with the last ID being that ID (the overrride requirement). To the > extent IDs are needed, they need to “stable”. This means they cannot be > overridden. As far as the use case is concerned where the ID attribute would get "overridden", it's not really overriding an existing ID. Maybe the XEP should use other terminology here. I full agree that stable IDs cannot be overridden. > For operational and security reasons, it unwise for one entity to rely > on IDs for services it provides (such as MAM) provide by some other > entity. MAM IDs are never provided by another entity then the MAM archive itself. At least that is my understanding of how MAM will/should work. > It’s not clear to me how this id provided by this extension would or > could be used in message type=error stanzas. That's simple: MIDs are not at all used for 'error' type messages. > It’s not clear to me why > this extension proposes a only solution, when it seems likely > that stanza id= problems are not necessarily specific to message stanzas. It was intentionally designed to be a solution for message stanzas only. I can't think why would want unique and stable messages IDs, like they are required to build an archive of stanza, for IQs and presences. IQ are simple request-response mechanisms and presence are ... Hmm, ok so maybe there is a reason one would archive presences in MAM (if we leave the privacy implications aside). That's actually a good point. What do the others think? I'm happy to s/message-id/stanza-id/. > It seems that there’s many security risks here associated with ID > forgery, replay, etc., but I’m not sure what, if any, mitigation is needed. Spoofing is actually a valid security concern. That is why XEP-MID states those strict business rules. > For the MUC use case, I rather we solve this as part of the MUC2 effort… I don't want to wait for MUC2, I want to solve cant-track-my-own-messages-in-muc issue right now, as it's pretty simple to solve. - Florian signature.asc Description: OpenPGP digital signature
Re: [Standards] Proposed XMPP Extension: Unique and stable message IDs
On 05.06.2015 13:26, Matthew Wild wrote: > On 5 June 2015 at 12:07, Daniel Gultsch wrote: >> 2015-06-05 13:01 GMT+02:00 Daniel Gultsch : >>> >>> And I'm currently not really seeing the point of a client adding an id >>> since the client can already set the stanza id. With an the by attribute we >>> don't have to decide on whether or not there is a use case for the >>> client-id. >> >> fwiw I just thought of a use case for a client setting the message-id. A >> client can use that to identify it's own message in a MUC when the muc >> server changes the stanza id. > > Right, this is basically the only case I have seen. And I would *much* > rather see the MUC XEP and the single implementation that I know with > this behaviour, fixed, instead of writing off the 'id' attribute as > useless for what it's designed for - a means of tracking messages. A few weeks ago, I was all for changing xep45 so that the ID of reflected messages SHOULD be preserved (like it was suggested by Georg on standards@ a few months ago). Now, I'm unsure if this is really possible: How do you handle duplicate IDs? How do you handle error stanzas send back to the MUC service, caused by a stanza with a duplicate ID? That's why I think the only way to possible solve this is with 'client-id', as described in http://geekplace.eu/xeps/xep-muc-extensions/xep-muc-extensions.html#mid - Florian signature.asc Description: OpenPGP digital signature
Re: [Standards] Proposed XMPP Extension: Unique and stable message IDs
On 03.06.2015 17:02, XMPP Extensions Editor wrote: > The XMPP Extensions Editor has received a proposal for a new XEP. > > Title: Unique and stable message IDs > > Abstract: This specification describes unique and stable IDs for message > stanzas. > > URL: http://xmpp.org/extensions/inbox/mid.html > > The XMPP Council will decide in the next two weeks whether to accept this > proposal as an official XEP. Thanks for the feedback everyone so far. I try to respond to most concerns and comments individually. I'd like to explain the rationale for the current design, since adding information about which entity the MID belongs to, was mentioned multiple times, I intentionally did not include the information about the related XMPP entity allocating the MID in , because my idea is that MAM, or MAM-like, archives won't contain 'groupchat' messages. This makes it possible to infer the XMPP entity the MID belongs to: If it's of not of type 'groupchat' it's a MID of the users MAM archive, if it's of type 'groupchat' it's a MID of the MUC service. The motivation was the principle, that a protocol is well designed if there is nothing left that can be removed without decreasing its power. But on the other hand, I've no strong objections adding a 'by' attribute to , if the XMPP community thinks it's absolutely required. -Florian signature.asc Description: OpenPGP digital signature
Re: [Standards] Proposed XMPP Extension: Unique and stable message IDs
On 5 June 2015 at 12:07, Daniel Gultsch wrote: > > > 2015-06-05 13:01 GMT+02:00 Daniel Gultsch : >> >> >> And I'm currently not really seeing the point of a client adding an id >> since the client can already set the stanza id. With an the by attribute we >> don't have to decide on whether or not there is a use case for the >> client-id. >> > > fwiw I just thought of a use case for a client setting the message-id. A > client can use that to identify it's own message in a MUC when the muc > server changes the stanza id. Right, this is basically the only case I have seen. And I would *much* rather see the MUC XEP and the single implementation that I know with this behaviour, fixed, instead of writing off the 'id' attribute as useless for what it's designed for - a means of tracking messages. Regards, Matthew
Re: [Standards] Proposed XMPP Extension: Unique and stable message IDs
2015-06-05 13:01 GMT+02:00 Daniel Gultsch : > > And I'm currently not really seeing the point of a client adding an id > since the client can already set the stanza id. With an the by attribute we > don't have to decide on whether or not there is a use case for the > client-id. > > fwiw I just thought of a use case for a client setting the message-id. A client can use that to identify it's own message in a MUC when the muc server changes the stanza id. - Daniel
Re: [Standards] Proposed XMPP Extension: Unique and stable message IDs
2015-06-04 11:26 GMT+02:00 Matthew Wild : > > This revision says that there must only be a single id at any point in > time. This would prevent you from learning the id that a remote MUC > service (or any other entity) assigned it (in case you want to query a > MUC's archive). > > I would probably go for the same approach as the original > element in MAM, and have a 'by' attribute (similar also to ): > > > > This would also allow clients to use it if they really wanted to, by > putting their JID in the 'by'. > agreed. Having multiple message-id tags each with their own by attribute makes for a more salable solution than having attributes for each individual propose like client-id, muc-server-id, server-id. On top of that the delay tag has the same attribute which gives us some consistency across different XEPs. And I'm currently not really seeing the point of a client adding an id since the client can already set the stanza id. With an the by attribute we don't have to decide on whether or not there is a use case for the client-id. cheers Daniel
Re: [Standards] Proposed XMPP Extension: Unique and stable message IDs
On 4 June 2015 at 11:56, Kurt Zeilenga wrote: > For the MAM case, to the extent what it offers in the current Experimental > XEP is not sufficient, I suggest MAM XEP be modified to provide for reliable > identification of archived content. Just so you know, this XEP grew out of MAM because we intend to include archived message ids in carbons. The consensus at the summit was that specifying ids as a new XEP removed the need for Carbons to depend on MAM, and would also be reusable in other specifications without depending on MAM. Regards, Matthew
Re: [Standards] Proposed XMPP Extension: Unique and stable message IDs
I'm in broad agreement with what Kurt writes below. I think the only point where we differ is that the "does MUC issue new ids" question is really one we could answer now (and if not directly answer, certainly fix). I would also add that using the stanza's id and from attributes, plus a (possibly server-provided) public tracking identifier for the stream, should be unique - and in at least most cases where we care we can mandate it. On 4 June 2015 at 11:56, Kurt Zeilenga wrote: > It’s not clear to me that the problems that this extensions proposes to > address can be addressed through use an extension to XMPP. Extensions > ought to be truly optional. This ProtoXEP appears to be making mandates > which are more appropriate made, if the community were to support them > being made, in a revision of the XMPP base specification. > > For instance, the ProtoXEP not only says "XMPP entities, which are > routing stanzas, MUST NOT strip the message-id element from message > stanzas” but seems to be rely on all servers adhering to this MUST NOT. > That’s never going to be case… certainly not for implementations not > purporting to implement this extension… and likely not for some > implementations which might purport to implement this extension. This > spec needs to consider and discuss what happens in face of elements it > suggests be added are stripped by another entity. > > I have a big problem with one ID to rule them all… and a bigger problem > with the last ID being that ID (the overrride requirement). To the extent > IDs are needed, they need to “stable”. This means they cannot be > overridden. > > For operational and security reasons, it unwise for one entity to rely on > IDs for services it provides (such as MAM) provide by some other entity. > To the extent IDs are needed, we need per-entity IDs… possibly even > per-enitty handling IDs (that is, when an IM handles a stanza for the > second time (such as after being handled by an MUC service, it might need > to separately ID the stanza). This because stanza content can and will > change as its handled by other entities. > > It’s not clear to me how this id provided by this extension would or could > be used in message type=error stanzas. It’s not clear to me why this > extension proposes a only solution, when it seems likely that > stanza id= problems are not necessarily specific to message stanzas. > > It seems that there’s many security risks here associated with ID forgery, > replay, etc., but I’m not sure what, if any, mitigation is needed. > > I suggest instead an element allowing entities to provide IDs in stanzas > they originate or handle. > > > > where ID is some opaque ID and JID is the providers JID. Servers which > need to provide different IDs in different contexts can use different JIDs. > (or possibly better to allow a context attribute or something). > > I’m not convinced having an extension providing an ID (whether by the > ProtoXEP suggestion or mine or other) reliably solves any problem I’m faced > with. > > For the MUC use case, I rather we solve this as part of the MUC2 effort… > > For the MAM case, to the extent what it offers in the current Experimental > XEP is not sufficient, I suggest MAM XEP be modified to provide for > reliable identification of archived content. > > — Kurt >
Re: [Standards] Proposed XMPP Extension: Unique and stable message IDs
It’s not clear to me that the problems that this extensions proposes to address can be addressed through use an extension to XMPP. Extensions ought to be truly optional. This ProtoXEP appears to be making mandates which are more appropriate made, if the community were to support them being made, in a revision of the XMPP base specification. For instance, the ProtoXEP not only says "XMPP entities, which are routing stanzas, MUST NOT strip the message-id element from message stanzas” but seems to be rely on all servers adhering to this MUST NOT. That’s never going to be case… certainly not for implementations not purporting to implement this extension… and likely not for some implementations which might purport to implement this extension. This spec needs to consider and discuss what happens in face of elements it suggests be added are stripped by another entity. I have a big problem with one ID to rule them all… and a bigger problem with the last ID being that ID (the overrride requirement). To the extent IDs are needed, they need to “stable”. This means they cannot be overridden. For operational and security reasons, it unwise for one entity to rely on IDs for services it provides (such as MAM) provide by some other entity. To the extent IDs are needed, we need per-entity IDs… possibly even per-enitty handling IDs (that is, when an IM handles a stanza for the second time (such as after being handled by an MUC service, it might need to separately ID the stanza). This because stanza content can and will change as its handled by other entities. It’s not clear to me how this id provided by this extension would or could be used in message type=error stanzas. It’s not clear to me why this extension proposes a only solution, when it seems likely that stanza id= problems are not necessarily specific to message stanzas. It seems that there’s many security risks here associated with ID forgery, replay, etc., but I’m not sure what, if any, mitigation is needed. I suggest instead an element allowing entities to provide IDs in stanzas they originate or handle. where ID is some opaque ID and JID is the providers JID. Servers which need to provide different IDs in different contexts can use different JIDs. (or possibly better to allow a context attribute or something). I’m not convinced having an extension providing an ID (whether by the ProtoXEP suggestion or mine or other) reliably solves any problem I’m faced with. For the MUC use case, I rather we solve this as part of the MUC2 effort… For the MAM case, to the extent what it offers in the current Experimental XEP is not sufficient, I suggest MAM XEP be modified to provide for reliable identification of archived content. — Kurt
Re: [Standards] Proposed XMPP Extension: Unique and stable message IDs
On 3 June 2015 at 16:02, XMPP Extensions Editor wrote: > The XMPP Extensions Editor has received a proposal for a new XEP. > > Title: Unique and stable message IDs > > Abstract: This specification describes unique and stable IDs for message > stanzas. > > URL: http://xmpp.org/extensions/inbox/mid.html Yay, thanks Florian! My feedback so far: ## client-id This is much better than the first draft I saw containing 'client-id'. However I'm still unconvinced that it belongs here. It has different uses, and different semantics. If it stays in this XEP (which I'm ok with), I think it should be in another element, which the server doesn't have to even care about. However for the use-cases that I understand it to be solving, I don't believe it should even be necessary. The id attribute is the client's id and it should be preserved. Duplicating it in a content element is a silly hack in my opinion. ## replacing message-id's This revision says that there must only be a single id at any point in time. This would prevent you from learning the id that a remote MUC service (or any other entity) assigned it (in case you want to query a MUC's archive). I would probably go for the same approach as the original element in MAM, and have a 'by' attribute (similar also to ): This would also allow clients to use it if they really wanted to, by putting their JID in the 'by'. Regards, Matthew
Re: [Standards] Proposed XMPP Extension: Unique and stable message IDs
On 3 Jun 2015, at 16:02, XMPP Extensions Editor wrote: > Title: Unique and stable message IDs > > URL: http://xmpp.org/extensions/inbox/mid.html Hoorah again for short XEPs. 3.1 - I think SHOULD is too strong for UUIDs here, isn’t it? There’s no interoperability concern - so I’d just go with ‘suggested’. 3.2 - I think MUST here is weird when combined with ‘some use cases’ being unnamed. In general, is stamping multiple message-ids, together with who did the stamping, most elegant here? /K
[Standards] Proposed XMPP Extension: Unique and stable message IDs
The XMPP Extensions Editor has received a proposal for a new XEP. Title: Unique and stable message IDs Abstract: This specification describes unique and stable IDs for message stanzas. URL: http://xmpp.org/extensions/inbox/mid.html The XMPP Council will decide in the next two weeks whether to accept this proposal as an official XEP.
