Re: [Standards] EXI extension proposal
(2013/03/20 5:47), Peter Waher wrote:
Dear Yusuke
Thanks for your mail. Regarding your questions and comments:
Ok, so I think I need to write another proposal. Would you mind if
I re-use the basic idea on 'EXI encoding part' of your proposal to
make better interoperability?
By all means. It would be excellent if you did. It would allow
devices to be able to connect freely between servers supporting
either of the two versions.
Thanks :-)
The proposal supports different EXI versions. It's part of the
negotiation, using the version attribute.
I see, then may I assume you have no intention to make 'yet another
EXI for XMPP' such as sessionWideBuffer option?
I'm not sure what you mean with 'yet another EXI for XMPP'. Where do
you find another proposal for EXI with option negotiation?
Also, we plan to let the sessionWideBuffer to stand for the time
being. I believe it's an important aspect of the integration of EXI
into XMPP. And as I showed using my examples (which are in no ways
extreme, they are in fact two common use cases in sensor networks)
sometimes one option is better than the other. One option is not the
best for all use cases.
I would say 'yet another EXI for XMPP' for EXI streams not be able to be
decoded by current EXI-1.0 compatible implementations, such as EXIP,
OpenEXI, and EXIficient. Your proposal seems to expect somewhat
different handling in EXI stream structure, including sessionWideBuffer.
I agree re-use of string table is important in terms encoding
efficiency. However, how to make it needs careful consideration.
Inter-stream session management could be a good idea worth standardized
in future version of EXI (however, this may involve out-of-EXI session
negotiation such as something like EXI-cookie option to find identical
session state). If you have long term point of view, it's better for
XMPP folks and EXI folks to make some discussion to solve.
(BTW, my proposal to handle this is to add NOP rule to let an element be
pushed out even in bit-packed stream)
On the other hand, if you need something you can use NOW, I believe you
need to stay in compatible stream structure. (You may think differently,
this is just my opinion).
Added a note in §2.7. regarding this, since it was unclear.
is only written at the start of a stream, and when the
stream is closed, with stanzas inbetween. When entering
EXI-compressed mode, the and should be omitted.
I'm not quite sure ... You mean something like this?
...
SD
SE("iq")
... (contents of iq)
EE
ED
(fflush(), PSH)
SD
SE("message")
... (contents of message)
EE
ED
(fflush(), PSH)
Regards,
// Yusuke DOI
Re: [Standards] EXI extension proposal
Dear Yusuke Thanks for your mail. Regarding your questions and comments: > Ok, so I think I need to write another proposal. Would you mind if I re-use > the basic idea on 'EXI encoding part' of your proposal to make better > interoperability? By all means. It would be excellent if you did. It would allow devices to be able to connect freely between servers supporting either of the two versions. >> The proposal supports different EXI versions. It's part of the negotiation, >> using the version attribute. > >I see, then may I assume you have no intention to make 'yet another EXI for >XMPP' such as sessionWideBuffer option? I'm not sure what you mean with 'yet another EXI for XMPP'. Where do you find another proposal for EXI with option negotiation? Also, we plan to let the sessionWideBuffer to stand for the time being. I believe it's an important aspect of the integration of EXI into XMPP. And as I showed using my examples (which are in no ways extreme, they are in fact two common use cases in sensor networks) sometimes one option is better than the other. One option is not the best for all use cases. > In my (yet) shallow understandings, your approach to send a stanza looks like > following (correct me if I'm wrong) > > > > > (padding, fflush(), PSH) > > > > (again, padding, fflush(), PSH) > > For me, it looks different from the semantics of tag defined in > RFC6120 section 4.4: > > If the parties are using either two streams over a single TCP connection or > > two streams over two TCP connections, the entity that sends the closing > > stream tag MUST behave as follows: > > > > 1.Wait for the other party to also close its outbound stream before > > terminating the underlying TCP connection(s); this gives the other party an > > opportunity to finish transmitting any outbound data to the closing entity > > before the termination of the TCP > connection(s). > > 2.Refrain from sending any further data over its outbound stream to the > > other entity, but continue to process data received from the other entity > > (and, if necessary, process such data). > > 3.Consider both streams to be void if the other party does not send its > > closing stream tag within a reasonable amount of time (where the definition > > of "reasonable" is a matter of implementation or deployment). > > 4.After receiving a reciprocal closing stream tag from the other party > > or waiting a reasonable amount of time with no response, terminate the > > underlying TCP connection(s). Added a note in §2.7. regarding this, since it was unclear. is only written at the start of a stream, and when the stream is closed, with stanzas inbetween. When entering EXI-compressed mode, the and should be omitted. Sincerely, Peter Waher -Original Message- From: Yusuke DOI [mailto:yusuke@toshiba.co.jp] Sent: den 19 mars 2013 05:07 To: Peter Waher Cc: Peter Saint-Andre; XMPP Standards; Joachim Lindborg (joachim.lindb...@sust.se); Takuki Kamiya (tkam...@us.fujitsu.com) Subject: Re: EXI extension proposal Peter, (sorry for multiple mail, I did wanted to split technical details and requirement discussions) (2013/03/18 22:53), Peter Waher wrote: >> Stepping back to the requirements. For example, what I need is binary-only >> bootstrap mechanism for EXI (not like XEP-0138) with least >> negotiation/propose of compression parameter. Does your proposal cover such >> use case? > > No. This XEP covers EXI negotiation in a way compliant with XEP-0138. Ok, so I think I need to write another proposal. Would you mind if I re-use the basic idea on 'EXI encoding part' of your proposal to make better interoperability? >> Let me ask a question: which strategy are you taking? >> >> 1) short target: find a way to encode XMPP stanza with existing >> EXI format 1.0 >> 2) long target: find a way to encode XMPP stanza ideally with >> proposed currently-nonexistent ideal EXI 1.1 (or 2.0) > > The proposal supports different EXI versions. It's part of the negotiation, > using the version attribute. I see, then may I assume you have no intention to make 'yet another EXI for XMPP' such as sessionWideBuffer option? >> My concern on a-1 is about XML semantics. Does current server implementation >> okay to have modified XML-level structure? On the other hand, re-use of >> string table is possible only in a-3 (discussion below). > > Sorry, I don't understand your question. In my (yet) shallow understandings, your approach to send a stanza looks like following (correct me if I'm wrong) (padding, fflush(), PSH) (again, padding, fflush(), PSH) For me, it looks different from the semantics of tag defined in RFC6120 section 4.4: > If the parties are using either two streams over a single TCP connection or > two streams over two TCP connections, the entity that sends the closing > stream tag MUST behave as follows: > > 1.Wait for the other party to also close its out
Re: [Standards] EXI extension proposal
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 3/19/13 3:39 AM, Yusuke DOI wrote: > Dear Peter, > > (2013-03-19 17:22), Peter Saint-Andre wrote: >> -BEGIN PGP SIGNED MESSAGE- If I understand correctly, a >> "binary-only bootstrap mechanism for EXI" would involve using a >> separate port and would effectively define a new transport >> binding for XMPP (in addition to the existing bindings = TCP, >> BOSH, and WebSocket). > > Thank you for giving the right direction. I think what to be > desgined is following three. > > 1) EXI in compression mechanism : Peter Waher's proposal 2) > Encoding detail : also in Peter's proposal, could be a common > ground 3) EXI as alternative transport binding > > I have no much time until the next week but after that I'll make up > my (our? is there anybody who is interested in EXI-as-binding?) > proposal. Based on what I know about constrained networks and devices (e.g., low power, low RAM, perhaps also intermittent connectivity or sleepy nodes), I can see the need for an alternative transport binding. So I think it is worth spending some time and effort on understanding the types of networks and devices we're designing for here. But personally I won't be able to do that until next week, or even the week after that. Peter - -- Peter Saint-Andre https://stpeter.im/ -BEGIN PGP SIGNATURE- Version: GnuPG/MacGPG2 v2.0.18 (Darwin) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQIcBAEBAgAGBQJRSHZ0AAoJEOoGpJErxa2p/qIP/0vkINtxfHdxxzrTNyy6b1wa TEc/mKyoNDMynEju48L2J1OpUGHIl20iMAZsOxHZLB8zVtFZDM/XbFh6EPAZD//7 gF56KpyBjMm15aRY6wonB5i2I+7jIrGjp7eIg5AdR1z7NZHDh9Xn6qiiibHIaYkR /WMpdBuYuS/v8Kf2Eu5FOAVqRXZuNI55vdFKzTE4EjsR6B3y/Pf11FEySqNJJwAS NlwZT/rTvhnyHf4/70taFNSPmmAbxqunXzN9jHwax3fBix5daqS2gHLobxOIP9l0 55omxwfWaqwdh6dpQ1NbykCnLtnrCuhKoUabGMEI+URlGB6r6VBBN/fu0IQJqbi9 Lu2BBbl6Y3pgAGXpqNON1fLBw9oeFYGGRYNdLhC1ZZFsD/UfkgRqUF53bZQgQcLW AQlI+ByeHDm4hooWlI/benDT9xHmiYatiYqgE8Lx4sGqFBMaZLVJtmD2BotnuUrz 70G09xOd2u0ephFzESjhq6j24xXmv/iGyWDwI+fn4eez6sffnIy/pUBBuKBBZcT6 0H5WKWyGvEQUaoiaUHUWA1byftjSngduCuugm61hEhzsea8aZfjewR1P+DwCDbdK 4LHNwIHqMTqBw8BpumyhoTsKqjNxSlLpl46la5Lmvics2RweRn30IrdqXoba/omy vg8r07DSzaSetJ+ahzpS =7fxz -END PGP SIGNATURE-
Re: [Standards] EXI extension proposal
Dear Peter, (2013-03-19 17:22), Peter Saint-Andre wrote: -BEGIN PGP SIGNED MESSAGE- If I understand correctly, a "binary-only bootstrap mechanism for EXI" would involve using a separate port and would effecitvely define a new transport binding for XMPP (in addition to the existing bindings = TCP, BOSH, and WebSocket). Thank you for giving the right direction. I think what to be desgined is following three. 1) EXI in compression mechanism : Peter Waher's proposal 2) Encoding detail : also in Peter's proposal, could be a common ground 3) EXI as alternative transport binding I have no much time until the next week but after that I'll make up my (our? is there anybody who is interested in EXI-as-binding?) proposal. Regards, Yusuke
Re: [Standards] EXI extension proposal
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 3/19/13 1:06 AM, Yusuke DOI wrote: > Peter, (sorry for multiple mail, I did wanted to split technical > details and requirement discussions) > > (2013/03/18 22:53), Peter Waher wrote: >>> Stepping back to the requirements. For example, what I need is >>> binary-only bootstrap mechanism for EXI (not like XEP-0138) >>> with least negotiation/propose of compression parameter. Does >>> your proposal cover such use case? >> >> No. This XEP covers EXI negotiation in a way compliant with >> XEP-0138. > > Ok, so I think I need to write another proposal. Would you mind if > I re-use the basic idea on 'EXI encoding part' of your proposal to > make better interoperability? If I understand correctly, a "binary-only bootstrap mechanism for EXI" would involve using a separate port and would effecitvely define a new transport binding for XMPP (in addition to the existing bindings = TCP, BOSH, and WebSocket). Peter - -- Peter Saint-Andre https://stpeter.im/ -BEGIN PGP SIGNATURE- Version: GnuPG/MacGPG2 v2.0.18 (Darwin) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQIcBAEBAgAGBQJRSCBeAAoJEOoGpJErxa2pmGMP+wZnlQQ5vgdoUB9kjPRESH23 4T1VM4H2Bx1qBOxChdiCZuFRKjRdIpNjIrYJ+AHlXA/fQ6WHNmeAXwIpVYWFoL2z DhQL7hGr+Wh10RG8+E/bhE86oP6yOyQwjoEaywvNtB2hIdeEceNc8dVWtFobstX0 UvjL/6lNrRypyKuOgbOBRnk4qZPFNoiKpbSrP/DFOMxDNd63NQYNRiOmmHZ74jGm hW7zh6EMNyqWjHA4Wy87cKxrdY9p1R9Onj+oz74VrGqk32jx4jk8Ekm4TIxRpkzc EfJEZPHUwIu/gfQnLdR5oEznrMn5ZsUCSHUF+b8mgpX5KxUcFbDiG90m8KKupva/ AoBHTH9GYQg5BMD/b8Ju01h0Z2pgBivnCzoU6woRLUj/hwdlZlgpaC2gmgjPsWJl w8uo5jA2iDGdm3QmlpepREMB/JTDhG1W0mLPHk7eKz2spphBPGjuhkS9yo3nQ/7v cd98hWXC+4mKkaZi9r97dED7BbNl5VIFp36yrof4/CqUP9XbkoVECiXDm4XRmIhm BICwTJ9tIpZLoXw2EbQcMY4wrPlq3TNaJSGcmdFeniLyhY5xIq/vfPTAbg6pzaQM qXH+xJ68jguyfoVZxm7cAy/Nrs91niyvYpxqCl8YpT+mc/PxCixp4eIJGNZcA9uW j/0PX1Zsfhqr0wb5/gma =PMur -END PGP SIGNATURE-
Re: [Standards] EXI extension proposal
Peter, (sorry for multiple mail, I did wanted to split technical details and requirement discussions) (2013/03/18 22:53), Peter Waher wrote: Stepping back to the requirements. For example, what I need is binary-only bootstrap mechanism for EXI (not like XEP-0138) with least negotiation/propose of compression parameter. Does your proposal cover such use case? No. This XEP covers EXI negotiation in a way compliant with XEP-0138. Ok, so I think I need to write another proposal. Would you mind if I re-use the basic idea on 'EXI encoding part' of your proposal to make better interoperability? Let me ask a question: which strategy are you taking? 1) short target: find a way to encode XMPP stanza with existing EXI format 1.0 2) long target: find a way to encode XMPP stanza ideally with proposed currently-nonexistent ideal EXI 1.1 (or 2.0) The proposal supports different EXI versions. It's part of the negotiation, using the version attribute. I see, then may I assume you have no intention to make 'yet another EXI for XMPP' such as sessionWideBuffer option? My concern on a-1 is about XML semantics. Does current server implementation okay to have modified XML-level structure? On the other hand, re-use of string table is possible only in a-3 (discussion below). Sorry, I don't understand your question. In my (yet) shallow understandings, your approach to send a stanza looks like following (correct me if I'm wrong) (padding, fflush(), PSH) (again, padding, fflush(), PSH) For me, it looks different from the semantics of tag defined in RFC6120 section 4.4: If the parties are using either two streams over a single TCP connection or two streams over two TCP connections, the entity that sends the closing stream tag MUST behave as follows: 1.Wait for the other party to also close its outbound stream before terminating the underlying TCP connection(s); this gives the other party an opportunity to finish transmitting any outbound data to the closing entity before the termination of the TCP connection(s). 2.Refrain from sending any further data over its outbound stream to the other entity, but continue to process data received from the other entity (and, if necessary, process such data). 3.Consider both streams to be void if the other party does not send its closing stream tag within a reasonable amount of time (where the definition of "reasonable" is a matter of implementation or deployment). 4.After receiving a reciprocal closing stream tag from the other party or waiting a reasonable amount of time with no response, terminate the underlying TCP connection(s). For dynamic grammars, your proposal "reusing the same options as used by the stream" (section 3.2) may not be adequate, because this means all stanzas should be in the same schema and does not allow introduction of new schema on the fly. However, I agree a > constrained node seldom updates its functionality so fixed set of schema on C2S pair should be okay (for S2S communication it's not good). I don't see why this should require all stanzas to be in the same schema. Schemas are handled by namespace, and a collection of schemas can be used in compression. Which schema namespaces are to be included is part of the negotiation. Sorry, I mean 'the set of schema given in the negotiation phase.' S2S connection may be required to re-negotiated if a new client connect to a server with schemas not included in the running S2S communication. Regards, Yusuke
Re: [Standards] EXI extension proposal
Dear Yusuke, Regarding your questions: > Is a mail with the subject 'Requirement Clarification' sent to the list? I've not received one. > Stepping back to the requirements. For example, what I need is binary-only > bootstrap mechanism for EXI (not like XEP-0138) with least > negotiation/propose of compression parameter. Does your proposal cover such > use case? No. This XEP covers EXI negotiation in a way compliant with XEP-0138. >Let me ask a question: which strategy are you taking? > > 1) short target: find a way to encode XMPP stanza with existing > EXI format 1.0 > 2) long target: find a way to encode XMPP stanza ideally with > proposed currently-nonexistent ideal EXI 1.1 (or 2.0) The proposal supports different EXI versions. It's part of the negotiation, using the version attribute. > My concern on a-1 is about XML semantics. Does current server implementation > okay to have modified XML-level structure? On the other hand, re-use of > string table is possible only in a-3 (discussion below). Sorry, I don't understand your question. > For dynamic grammars, your proposal "reusing the same options as used by the > stream" (section 3.2) may not be adequate, because this means all stanzas > should be in the same schema and does not allow introduction of new schema on > the fly. However, I agree a > constrained node seldom updates its > functionality so fixed set of schema on C2S pair should be okay (for S2S > communication it's not good). I don't see why this should require all stanzas to be in the same schema. Schemas are handled by namespace, and a collection of schemas can be used in compression. Which schema namespaces are to be included is part of the negotiation. Sincerely, Peter Waher -Original Message- From: Yusuke DOI [mailto:yusuke@toshiba.co.jp] Sent: den 16 mars 2013 01:10 To: Peter Waher Cc: Peter Saint-Andre; XMPP Standards; Joachim Lindborg (joachim.lindb...@sust.se); Takuki Kamiya (tkam...@us.fujitsu.com) Subject: Re: EXI extension proposal Dear Peter, (2013/03/15 12:53), Peter Waher wrote: >> I believe we need to clarify some of 'requirements' first. Maybe, >> there could be several approaches for EXI1.0 or maybe we need to >> propose something to EXI1.x(maybe x=1), upon such requirement >> discussion. >> >> Topics may involve: - Minimal client requirement - Minimal server >> requirement - Schema management - Efficiency requirement - ... >> >> I'll make another thread on requirement discussion following this >> mail. > > Sounds like a very good idea. I'll wait your coming mail, and then > create a new section for this. Is a mail with the subject 'Requirement Clarification' sent to the list? I have had mailer trouble so I'm not sure I've successfully sent out the mail. Stepping back to the requirements. For example, what I need is binary-only bootstrap mechanism for EXI (not like XEP-0138) with least negotiation/propose of compression parameter. Does your proposal cover such use case? >> Agreed. BTW, is the schema itself encoded in EXI or plaintext? The >> example seems to be in plaintext but it could be EXI-encoded binary. > > Do you mean with the uploadSchema command? In this case, it's simply > base64-encoded plain text. > >> Then upload in xs:base64binary will become straitforward Binary type >> upload, far more efficient. > > I agree. If packet size is of importance (which it probably is for > memory constrained devices), embedding the schemas might not even be > possible. In this case, downloadSchema is an option that requires very > few bytes. EXI-encoded schema exchange should be also good for EXI-only nodes. Anyway, if a node does not be capable of handling EXI, schema download/upload is useless. >>> The problem with this approach, is that name seldom change, >>> especially during development. And a slight change, a new attribute, >>> a new element, etc., will completely change the compression. >>> Furthermore, errors produced in this way will be extremely difficult >>> to debug. An efficient and fool-proof way to communicate using >>> different schema versions (having the same namespace and schema IDs) >>> is necessary. >> >>> §2.4 also proposes the possibility to install such schema files >>> manually on the server. The XEP allows for different scenarios. > >> Agreed. We need different mechanism for development stage and >> deployment stage. I believe schemaLocation and hash is sufficient >> enough. > > It will probably be enough. However, I used target namespace instead > of schema location, as the schema may be available in many places, and > it's the target namespace that is used during validation and > compression. I also included the byte size as a reference. I don't think the number of bytes is required in this case. Usually schema file itself has targetNameSpace so namespace is also redundunt. With schemaLocation, the other end of the node may choose to download the schema from the lo
Re: [Standards] EXI extension proposal
Dear Peter, (2013/03/15 12:53), Peter Waher wrote: I believe we need to clarify some of 'requirements' first. Maybe, there could be several approaches for EXI1.0 or maybe we need to propose something to EXI1.x(maybe x=1), upon such requirement discussion. Topics may involve: - Minimal client requirement - Minimal server requirement - Schema management - Efficiency requirement - ... I'll make another thread on requirement discussion following this mail. Sounds like a very good idea. I'll wait your coming mail, and then create a new section for this. Is a mail with the subject 'Requirement Clarification' sent to the list? I have had mailer trouble so I'm not sure I've successfully sent out the mail. Stepping back to the requirements. For example, what I need is binary-only bootstrap mechanism for EXI (not like XEP-0138) with least negotiation/propose of compression parameter. Does your proposal cover such use case? Agreed. BTW, is the schema itself encoded in EXI or plaintext? The example seems to be in plaintext but it could be EXI-encoded binary. Do you mean with the uploadSchema command? In this case, it's simply base64-encoded plain text. Then upload in xs:base64binary will become straitforward Binary type upload, far more efficient. I agree. If packet size is of importance (which it probably is for memory constrained devices), embedding the schemas might not even be possible. In this case, downloadSchema is an option that requires very few bytes. EXI-encoded schema exchange should be also good for EXI-only nodes. Anyway, if a node does not be capable of handling EXI, schema download/upload is useless. The problem with this approach, is that name seldom change, especially during development. And a slight change, a new attribute, a new element, etc., will completely change the compression. Furthermore, errors produced in this way will be extremely difficult to debug. An efficient and fool-proof way to communicate using different schema versions (having the same namespace and schema IDs) is necessary. §2.4 also proposes the possibility to install such schema files manually on the server. The XEP allows for different scenarios. Agreed. We need different mechanism for development stage and deployment stage. I believe schemaLocation and hash is sufficient enough. It will probably be enough. However, I used target namespace instead of schema location, as the schema may be available in many places, and it's the target namespace that is used during validation and compression. I also included the byte size as a reference. I don't think the number of bytes is required in this case. Usually schema file itself has targetNameSpace so namespace is also redundunt. With schemaLocation, the other end of the node may choose to download the schema from the location, instead of returning a 'I don't have the schema please upload with our tiny sensor network bandwidth' error. I had to rethink this a bit, after comments made by Takuki Kamiya. I've rewritten §3.2 to clarify this. Is it clear from what I've written? The EXI header and options are omitted in the communication, since they are agreed upon during initial setup. Only EXI bodies are written. Furthermore, I changed the operation to exclude the SD/ED events and include a description of how to perform padding after each last End Element EE at the end of each stanza, so EXI bodies are sent at byte boundaries. Again, as Taki mentioned, we'd better not to alter EXI spec. Let me ask a question: which strategy are you taking? 1) short target: find a way to encode XMPP stanza with existing EXI format 1.0 2) long target: find a way to encode XMPP stanza ideally with proposed currently-nonexistent ideal EXI 1.1 (or 2.0) If you're taking strategy (2), I stongly argue it's too early to step into detailed spec. Even you and I does not have common view on this problem space. For strategy (1), there are at least two problems (let me start again): a) padding b) dynamic schema/grammar For padding problem there are three solutions, a-1) encode a stanza in a time, as you proposed a-2) use selfContained per stanza a-3) use byte-aligned My concern on a-1 is about XML semantics. Does current server implementation okay to have modified XML-level structure? On the other hand, re-use of string table is possible only in a-3 (discussion below). For dynamic grammars, your proposal "reusing the same options as used by the stream" (section 3.2) may not be adequate, because this means all stanzas should be in the same schema and does not allow introduction of new schema on the fly. However, I agree a constrained node seldom updates its functionality so fixed set of schema on C2S pair should be okay (for S2S communication it's not good). I think this makes far worse compression ratio (according to my experiment). This may depend on type of information sent. Example: ... 100 times (inly Energy, MWh) May be better compressed if str
Re: [Standards] EXI extension proposal
Dear Peter, I believe we need to clarify some of 'requirements' first. Maybe, there could be several approaches for EXI1.0 or maybe we need to propose something to EXI1.x(maybe x=1), upon such requirement discussion. Topics may involve: - Minimal client requirement - Minimal server requirement - Schema management - Efficiency requirement - ... I'll make another thread on requirement discussion following this mail. (2013/03/13 12:26), Peter Waher wrote: §3.3 in the XEP handles this. The server is free to add cache rules to avoid explosion of number of derived schemas. The server is also free to reject uploading or downloading requests, for any reason (§2.4) Agreed. BTW, is the schema itself encoded in EXI or plaintext? The example seems to be in plaintext but it could be EXI-encoded binary. Then upload in xs:base64binary will become straitforward Binary type upload, far more efficient. The problem with this approach, is that name seldom change, especially during development. And a slight change, a new attribute, a new element, etc., will completely change the compression. Furthermore, errors produced in this way will be extremely difficult to debug. An efficient and fool-proof way to communicate using different schema versions (having the same namespace and schema IDs) is necessary. §2.4 also proposes the possibility to install such schema files manually on the server. The XEP allows for different scenarios. Agreed. We need different mechanism for development stage and deployment stage. I believe schemaLocation and hash is sufficient enough. It is supposed that the EXI compression engine works in XML fragment mode, where each stanza is compressed separately. I don't see that self contained elements would not be required in this case. Could you elaborate this? Are you thinking something like this? [EXI Header][EXI Option] SD SE(stream) SE(some-stanza) ... EE(some-stanza) // single stanza EE(stream) ED (padding) SD SE(stream) SE(some-stanza) ... EE(some-stanza) EE(stream) ED or [EXI Header][EXI Option] SD SE(some-stanza) ... EE(some-stanza) ED (padding) SD SE(some-stanza) ... EE(some-stanza) ED or something else? (for those who are not familiar with EXI, SD/ED: start/end document, SE/EE: start/end element) To be honest, I have never implemented EXI Fragment Document Grammar so my understanding may wrong. Tables should not be reused between stanzas, since tables can be very small, but, as you point out in your example, number of possible strings may be large (for instance many different JIDs). However, possible strings within a message are much smaller, making also references to tables shorter within the bit-packed message. I think this makes far worse compression ratio (according to my experiment). Finally it's up to the implementor how to setup the EXI compression engine. Some may feel bit-packed is better, some that byte-packed is better. I'll add an implementation note in the XEP regarding this. In my experience in SEP2[1], we didn't need full option for it. You may interested in EXI Profile[2], too. [1] SEP2: smart energy profile 2 http://www.zigbee.org/Standards/ZigBeeSmartEnergy/Version20Documents.aspx SEP2 editors have had long discussion on how to use EXI in constrained devices and agreed to use it in schema-informed, non-strict grammar with schemaID and schema negotiation for linear extensibility. [2] http://www.w3.org/TR/exi-profile/ This is a vital aspect of this proposal. For sensor networks and IoT, especially wireless sensor networks, buffer size is first priority. Therefore, EXI compression should be done with as much information about schemas as possible. Agreed. We also see this as an important aspect of this proposal: Most sensors will have pre-compiled code, often (semi-)automatically generated from schema files, for compression and decompression of EXI content. Therefore, the proposal Includes the possibility for the client to reject the connection if parameters are not as expected. Agreed, but I think there should be some greatest common divisor (or 'please implement at least this functionarity'-set) for better interoperability. (from the other mail) I've thought more about your proposition regarding session-wide string tables vs. having to clear tables and buffers between stanzas. Both have merits, depending on use case. So, I added an option controlling this behavior: sessionWideBuffers, as well as descriptions describing the differences in how they work. As well as a note regarding flushing between stanzas. We need to be careful to add something on existing standards. In other words, existing EXI processors should be applicable for this proposal. Best Regards, // Yusuke DOI
Re: [Standards] EXI extension proposal
Dear Yusuke. Thank you for your constructive comments. I'll try to address them one at a time: > I think this idea of schema exchange is interesting. On the other hand, it > may make confusion on management (explosion of the number of derived schemas). §3.3 in the XEP handles this. The server is free to add cache rules to avoid explosion of number of derived schemas. The server is also free to reject uploading or downloading requests, for any reason (§2.4) > If there is a way to name a XML schema defined in XEP, servers and clients > can share them by the names. The problem with this approach, is that name seldom change, especially during development. And a slight change, a new attribute, a new element, etc., will completely change the compression. Furthermore, errors produced in this way will be extremely difficult to debug. An efficient and fool-proof way to communicate using different schema versions (having the same namespace and schema IDs) is necessary. > Of course (1) this does not eliminate needs to upload XML schema because the > end device may have non-XEP vendor specific extensions (2) we need secure > channel to download XEP-defined schemas to avoid attacks. §2.4 also proposes the possibility to install such schema files manually on the server. The XEP allows for different scenarios. > Another pitfall: if we want to use bit-packed, we need to make stanzas > encoded as self contained elements. Otherwise you cannot do 'fflush()' at > the end of element of a stanza. However, self contained elements do not allow > an encoder to re-use compression context (string tables) between outside of > the element and the element itself. This means the encoder need to re-encode > JID strings as is (otherwise you can just encode a string with few bytes of > reference). It is supposed that the EXI compression engine works in XML fragment mode, where each stanza is compressed separately. I don't see that self contained elements would not be required in this case. Tables should not be reused between stanzas, since tables can be very small, but, as you point out in your example, number of possible strings may be large (for instance many different JIDs). However, possible strings within a message are much smaller, making also references to tables shorter within the bit-packed message. Finally it's up to the implementor how to setup the EXI compression engine. Some may feel bit-packed is better, some that byte-packed is better. I'll add an implementation note in the XEP regarding this. > Note: the results do not use schema-informed grammars to encode XEP-based > elements, so compression ratio of Peter's proposal should be much better -- > in my best scenario with schema-informed EXI, it will be 809 bytes (22% of original XML). This is a vital aspect of this proposal. For sensor networks and IoT, especially wireless sensor networks, buffer size is first priority. Therefore, EXI compression should be done with as much information about schemas as possible. > BTW, my initial idea is somewhat different. What I want to make is > constrained XMPP clients (and if technically possible, servers) with static > set of pre-compiled EXI grammars and without ability to talk with regular > XML-based XMPP. This enables nodes with batteries to speak sensor data with > narrow wireless link such as 15.4 or with 3G link charged by quantity. Maybe > this idea is oriented towards SRV-based negotiation. We also see this as an important aspect of this proposal: Most sensors will have pre-compiled code, often (semi-)automatically generated from schema files, for compression and decompression of EXI content. Therefore, the proposal Includes the possibility for the client to reject the connection if parameters are not as expected. Sincerely, Peter Waher -Original Message- From: Yusuke DOI [mailto:yusuke@toshiba.co.jp] Sent: den 13 mars 2013 12:00 To: Peter Waher Cc: Peter Saint-Andre; XMPP Standards; Joachim Lindborg (joachim.lindb...@sust.se) Subject: Re: EXI extension proposal Dear Peter, (2013/03/13 23:10), Peter Waher wrote: > Anybody interested in EXI & XMPP, please review. Any feedback is most welcome. I think this idea of schema exchange is interesting. On the other hand, it may make confusion on management (explosion of the number of derived schemas). If there is a way to name a XML schema defined in XEP, servers and clients can share them by the names. Of course (1) this does not eliminate needs to upload XML schema because the end device may have non-XEP vendor specific extensions (2) we need secure channel to download XEP-defined schemas to avoid attacks. Another pitfall: if we want to use bit-packed, we need to make stanzas encoded as self contained elements. Otherwise you cannot do 'fflush()' at the end of element of a stanza. However, self contained elements do not allow an encoder to re-use compression context (string tables) between outside of
Re: [Standards] EXI extension proposal
Dear Peter, (2013/03/13 23:10), Peter Waher wrote: Anybody interested in EXI & XMPP, please review. Any feedback is most welcome. I think this idea of schema exchange is interesting. On the other hand, it may make confusion on management (explosion of the number of derived schemas). If there is a way to name a XML schema defined in XEP, servers and clients can share them by the names. Of course (1) this does not eliminate needs to upload XML schema because the end device may have non-XEP vendor specific extensions (2) we need secure channel to download XEP-defined schemas to avoid attacks. Another pitfall: if we want to use bit-packed, we need to make stanzas encoded as self contained elements. Otherwise you cannot do 'fflush()' at the end of element of a stanza. However, self contained elements do not allow an encoder to re-use compression context (string tables) between outside of the element and the element itself. This means the encoder need to re-encode JID strings as is (otherwise you can just encode a string with few bytes of reference). This may make the XMPP/EXI stream more inefficient compared to byte-aligned streams. My preliminary experiment shows following results. [The number of bytes] bytes -+--- XML | 3681 selfContained, bit-packed| 1589 byte-aligned | 1358 -+--- Note: the results do not use schema-informed grammars to encode XEP-based elements, so compression ratio of Peter's proposal should be much better -- in my best scenario with schema-informed EXI, it will be 809 bytes (22% of original XML). BTW, my initial idea is somewhat different. What I want to make is constrained XMPP clients (and if technically possible, servers) with static set of pre-compiled EXI grammars and without ability to talk with regular XML-based XMPP. This enables nodes with batteries to speak sensor data with narrow wireless link such as 15.4 or with 3G link charged by quantity. Maybe this idea is oriented towards SRV-based negotiation. For long-targetted apporach, I think I can propose some update to EXI spec itself (I recently joined to W3C EXI working group). Now EXI working group are open to collect requirements for EXI2.0 (I already raised fflush() issue). I believe this kind of collaboration should be very important to let more constrained IoT devices join the network. Regards, Yusuke
Re: [Standards] EXI extension proposal
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 3/13/13 10:10 AM, Peter Waher wrote: > Hello Peter > > Here's the first draft of a new EXI extension. Could you give it a > number? We used to work that way, but nowadays the XMPP Council needs to approve of publication first. I will, however, add it to the inbox: http://xmpp.org/extensions/inbox/ I have a few other proposals to process at the same time, and the Council will figure out what to do with them at its next meeting (a week from now). BTW, Council meetings are open to the public, add http://xmpp.org/calendar/xsf-council.ics to your calendar if you want to know when they happen. Peter - -- Peter Saint-Andre https://stpeter.im/ -BEGIN PGP SIGNATURE- Version: GnuPG/MacGPG2 v2.0.18 (Darwin) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQIcBAEBAgAGBQJRQIt2AAoJEOoGpJErxa2pBdgP/06KI5B1Ux2fBLLUQuUASsvb LyotYLDHLnQ81fkDwwXtCIlRI4APW4pme0UNE9zwnQiuG9ic4vsXdqAAFe2HunNc vLwlD7Q14MpZ9sw4yzmG5msfCc/rbakl7KgMvqdYJM64KiG1FcIjCQKp7j6KgKKK jZApj/wASw7P4YK23g8wMlzD7oLZi6cItCzF1/vcPS/nK8V6LZ41sW6FReeTm5V/ maNCXxWZ9U4P+MbwOs5bznW2fJDZEO8onzJ9sZJHe+evCWLrcWeGUwShWi6Im87j yLWteInEGFu8VwFNoHOWdvszrskYNx6t9Gdr4jWDcIjgCjuS9eOfKyIlzgUcARAT rK7EY1m+DA/wvm09i3+ftFrM42QJuVYEdmoYAWlY79q8BYfaRzx32kJkAvNNsF4h QgbDx9indHsKpenb5w99cKyHm2Dvogj3pjw+5khXQRPc7pdZq/Ps/yvZ+6A6B3l3 WROSQkUHI06+BoLDs55Pb6X5106RNkJiRfbCcbyIRed3i2ujX4Ge4QUrIzlcYQuN ipnlHfHGcv4hbFT8Gt1ZvcoHrwk+h+nm8fUTdi+qxDzu42NqpNS6vzBpBrIsZhZi yTuMXbF9eH5dzjasV9GfAjMy1bpJCz4+Jb0wAX5JAVBzbOlS97w2gukseMcdbQXN VxphMvUltlt1v6t3I2wh =25AJ -END PGP SIGNATURE-
