Re: [Standards] EXI extension proposal

[Yusuke DOI]

Dear Peter,

(2013/03/15 12:53), Peter Waher wrote:

I believe we need to clarify some of 'requirements' first. Maybe,
there could be several approaches for EXI1.0 or maybe we need to
propose something to EXI1.x(maybe x=1), upon such requirement
discussion.

Topics may involve: - Minimal client requirement - Minimal server
requirement - Schema management - Efficiency requirement - ...

I'll make another thread on requirement discussion following this
mail.


Sounds like a very good idea. I'll wait your coming mail, and then
create a new section for this.


Is a mail with the subject 'Requirement Clarification' sent to the list?
I have had mailer trouble so I'm not sure I've successfully sent out the
mail.

Stepping back to the requirements. For example, what I need is
binary-only bootstrap mechanism for EXI (not like XEP-0138) with least
negotiation/propose of compression parameter. Does your proposal cover
such use case?


Agreed. BTW, is the schema itself encoded in EXI or plaintext? The
example seems to be in plaintext but it could be EXI-encoded
binary.


Do you mean with the uploadSchema command? In this case, it's simply
base64-encoded plain text.


Then upload in xs:base64binary will become straitforward Binary
type upload, far more efficient.


I agree. If packet size is of importance (which it probably is for
memory constrained devices), embedding the schemas might not even be
possible. In this case, downloadSchema is an option that requires
very few bytes.


EXI-encoded schema exchange should be also good for EXI-only nodes.
Anyway, if a node does not be capable of handling EXI, schema
download/upload is useless.


The problem with this approach, is that name seldom change,
especially during development. And a slight change, a new
attribute, a new element, etc., will completely change the
compression. Furthermore, errors produced in this way will be
extremely difficult to debug. An efficient and fool-proof way to
communicate using different schema versions (having the same
namespace and schema IDs) is necessary.



§2.4 also proposes the possibility to install such schema files
 manually on the server. The XEP allows for different scenarios.



Agreed. We need different mechanism for development stage and
deployment stage. I believe schemaLocation and hash is sufficient
enough.


It will probably be enough. However, I used target namespace instead
of schema location, as the schema may be available in many places,
and it's the target namespace that is used during validation and
compression. I also included the byte size as a reference.


I don't think the number of bytes is required in this case. Usually
schema file itself has targetNameSpace so namespace is also redundunt.
With schemaLocation, the other end of the node may choose to download
the schema from the location, instead of returning a 'I don't have the
schema please upload with our tiny sensor network bandwidth' error.


I had to rethink this a bit, after comments made by Takuki Kamiya.
I've rewritten §3.2 to clarify this. Is it clear from what I've
written?

The EXI header and options are omitted in the communication, since
they are agreed upon during initial setup. Only EXI bodies are
written. Furthermore, I changed the operation to exclude the SD/ED
events and include a description of how to perform padding after
each last End Element EE at the end of each stanza, so EXI bodies are
sent at byte boundaries.


Again, as Taki mentioned, we'd better not to alter EXI spec.

Let me ask a question: which strategy are you taking?

 1) short target: find a way to encode XMPP stanza with existing
EXI format 1.0
 2) long target: find a way to encode XMPP stanza ideally with
proposed currently-nonexistent ideal EXI 1.1 (or 2.0)

If you're taking strategy (2), I stongly argue it's too early to step
into detailed spec. Even you and I does not have common view on this
problem space.

For strategy (1), there are at least two problems (let me start again):

 a) padding
 b) dynamic schema/grammar

For padding problem there are three solutions,
 a-1) encode a stanza in a time, as you proposed
 a-2) use selfContained per stanza
 a-3) use byte-aligned

My concern on a-1 is about XML semantics. Does current server
implementation okay to have modified XML-level structure? On the other
hand, re-use of string table is possible only in a-3 (discussion below).

For dynamic grammars, your proposal "reusing the same options as used by
the stream" (section 3.2) may not be adequate, because this means all
stanzas should be in the same schema and does not allow introduction of
new schema on the fly. However, I agree a constrained node seldom
updates its functionality so fixed set of schema on C2S pair should be
okay (for S2S communication it's not good).


I think this makes far worse compression ratio (according to my
experiment).


This may depend on type of information sent.

Example:

  ... 100 times (inly Energy, MWh) 

May be better compressed if str

[Standards] Question regarding data URI in XEP-0292

[Todd Herman]

XEP-0292 uses the XML format defined in RFC 6351.  Section 11.3 has an example 
vCard.  Near the bottom is the "logo" element with a child "uri" element.  RFC 
6352 defines the logo element as containing a child element of value-uri.  This 
is defined as an element named "uri" that is any URI type.  In the example, a 
data URI is being used.

I am not overly familiar with the data URI format but when I looked it up it 
says the format requires you to specify the encoding.  The example in section 
11.3 of the XEP does not do this.  It has the following:

data:image/jpeg;{data}

Where "{data} is the base 64 encoded image.  My research shows that the data 
URI should look more like this:

data:image/jpeg;base64,{data}

Can someone let me know which is correct?  I thought maybe it would just 
default to base 64 if omitted but the article I read said it defaults to 
something else entirely.

On a slightly related not, the examples in XEP-0292 show the "n" element as 
containing a child element of "middle".  The xCard specification (RFC 6351) 
indicates only the following children (surname, given, additional, prefix, 
suffix) all of which are mandatory (but can be empty).  This seems to be a 
conflict.

I know XEP-0292 is currently deferred but we have a need for this.  I wouldn't 
mind any insight on whether I am just reading them wrong as that is always a 
possibility.  I would also be happy to provide any additional feedback or make 
additional suggestions related to this XEP in order to bring it out of 
"deferred" status.

Re: [Standards] Proposal for including EXI in XMPP

[rtur...@amalfisystems.com]

Hi

Yes...I understood from the previous message that, for a constrained device, 
the implementation would essentially be locked down, to support only the 
specific xml protocol snippets that the device would support, which could 
significantly reduce the code ROM requirements, compared with a full EXI 
implementation that could support a more dynamic and flexible xml/schema 
processing model


Sent from my Verizon Wireless 4G LTE Smartphone

 Original message 
From: Peter Waher  
Date: 03/15/2013  11:09 AM  (GMT-08:00) 
To: XMPP Standards  
Cc: public-...@w3.org,Randy Turner  
Subject: RE: [Standards] Proposal for including EXI in XMPP 
 
Dear Randy

Constrained here, apart from the normal English definition, could mean any 
reason why they would not otherwise be able to use XMPP as a transport protocol.

One constraint could be allowable packet size. Wireless sensor networks (for 
example over 6LowPan) can only send small packets. So, it might well be that 
compressing the stanzas would enable them to send control messages vs. not 
being able to send them (successfully). Partitioning small messages into 
several parts in otherwise congested WSN might not be a practical option.

Another constraint may be available RAM for buffers. While they may actually 
have plenty of ROM to store code, schema files, etc., RAM might be limited.

Furthermore, efficient code can be automatically generated from schema files, 
so writing and reading EXI can be surprisingly efficient (both in size and 
processing power), if strict schema compression is used.

Sincerely,
Peter Waher


-Original Message-
From: Randy Turner [mailto:rtur...@amalfisystems.com] 
Sent: den 15 mars 2013 12:08
To: XMPP Standards
Cc: public-...@w3.org
Subject: Re: [Standards] Proposal for including EXI in XMPP


I was curious what the definition of "constrained" is ?  

EXI does produce a compact representation of XML (which is good if 
"constrained" is meant to apply to the amount of any output XML representation)

But I think the executable code size of an EXI implementation might not be 
appropriate for a "light switch", "low-power sensor"  or other similarly 
constrained memory device.

But I guess it depends on the definition of "constrained" memory?  (both RAM 
and non-volatile code space) -- especially since there may be a dual-stack 
IPv4/IPv6 stack, HTTP module, operating system, and other code already assumed 
to be on the constrained device.

Randy


On Mar 15, 2013, at 3:39 AM, Peter Saint-Andre  wrote:

> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA1
> 
> On 3/14/13 6:50 PM, John Schneider wrote:
>> Peter,
>> 
>> Its really great to see some momentum on this! Thanks very much for 
>> the energy you are putting into it. This is something in which I've 
>> had a long-running interest. In fact, I think Peter and I first 
>> talked about it back in 2005 (yikes!). Although I'm not sure Peter 
>> was completely sold on the idea at the time (broccoli ice cream 
>> anyone? [1]), he was gracious enough to help us get the XMPP EXI use 
>> case together [2] and even started an early draft [3]. Thanks Peter! 
>> ;-)
> 
> One must keep an open mind.
> 
>> Here at work, we have incorporated EXI into various XMPP solutions to 
>> support our Efficient XML users (e.g., to enable XMPP on aircraft). 
>> So, I'm very interested in following your progress.
>> Please keep me in the loop as you move forward. I'd be very happy to 
>> help if I can.
> 
> Yesterday at IETF 86 I had a productive chat with Yusuke Doi on this 
> topic. He helped me see that, just as we've defined HTTP bindings 
> (BOSH / WebSocket) for web endpoints like browsers, an EXI binding 
> would expand the universe of XMPP usage to constrained devices of the 
> kind that might otherwise use protocols like CoAP.
> 
> Although I've not yet had time to read the new EXI proposal in detail, 
> I shall do so as soon as possible and provide comments on the 
> standards@xmpp.org list.
> 
> Peter
> 
> - --
> Peter Saint-Andre
> https://stpeter.im/
> 
> 
> -BEGIN PGP SIGNATURE-
> Version: GnuPG/MacGPG2 v2.0.18 (Darwin)
> Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
> 
> iQIcBAEBAgAGBQJRQvpOAAoJEOoGpJErxa2pcfMQAKPeGWf+02AZmrGff3hA+H+I
> g9INO8/62pukrj7utK1BuBRkersMXLxJUm3TIWnfQuqBXe/zgbkcvms42GHvu9+R
> J5/37i4Mq+lKHdI/XyPtLnn6/3YjriotGwl2ZKpvxvD4b66F0BL0jUaCoZx6jpPm
> N7QNYtX51uzpU7ofWfWf/IhzXOKNgFaB4u/EVcJp4+Gu8UInnualtkeq/ZHQwEiG
> SkpE36LfuCy31cXEd4Oankv30ywOwUmh2ETwzLyeDPzPfhFVdjgNkabnJr1J/H/n
> 8GnZNIEgzx/hFTGUetHkhMaHQfqGAVtEjWsFIYStugQnBRS6pP/V+fEME4DEMw64
> FLf9sF8sJBSP4/e62Z8myog/eVWrYiGNjGRu8qvo+fNmD4Fn+/qhHA/SvYcN+ZXN
> yqeqzAJty+A+oxduPCN+bbP93grroSjs1qmN7ybsu+bO9hJiDCs3IIsJPOBZrr9Y
> uQ+an33QWIAEsPRO4VKQlFYxHmh6QRjUPTCeFyFemuD7dyLk3UBXNikMUrvcFwBa
> uj0jAyC0tIf0w/Qm5RPtzVIV02On1jKRjHGoNLjEnjf0nusmdqzEZMq3mIL95PzL
> g1WYu5ASmDVAzWUuwvhYYMGraAQbqrnj5+QgiaAEHulpjl43ut7FHYslD7uyPvFp
> E8efql+LqrpSRCjgvZ8e
> =jNUK
> -END PGP SIGNATURE--

Re: [Standards] Proposal for including EXI in XMPP

[Peter Waher]

Dear Randy

Constrained here, apart from the normal English definition, could mean any 
reason why they would not otherwise be able to use XMPP as a transport protocol.

One constraint could be allowable packet size. Wireless sensor networks (for 
example over 6LowPan) can only send small packets. So, it might well be that 
compressing the stanzas would enable them to send control messages vs. not 
being able to send them (successfully). Partitioning small messages into 
several parts in otherwise congested WSN might not be a practical option.

Another constraint may be available RAM for buffers. While they may actually 
have plenty of ROM to store code, schema files, etc., RAM might be limited.

Furthermore, efficient code can be automatically generated from schema files, 
so writing and reading EXI can be surprisingly efficient (both in size and 
processing power), if strict schema compression is used.

Sincerely,
Peter Waher


-Original Message-
From: Randy Turner [mailto:rtur...@amalfisystems.com] 
Sent: den 15 mars 2013 12:08
To: XMPP Standards
Cc: public-...@w3.org
Subject: Re: [Standards] Proposal for including EXI in XMPP


I was curious what the definition of "constrained" is ?  

EXI does produce a compact representation of XML (which is good if 
"constrained" is meant to apply to the amount of any output XML representation)

But I think the executable code size of an EXI implementation might not be 
appropriate for a "light switch", "low-power sensor"  or other similarly 
constrained memory device.

But I guess it depends on the definition of "constrained" memory?  (both RAM 
and non-volatile code space) -- especially since there may be a dual-stack 
IPv4/IPv6 stack, HTTP module, operating system, and other code already assumed 
to be on the constrained device.

Randy


On Mar 15, 2013, at 3:39 AM, Peter Saint-Andre  wrote:

> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA1
> 
> On 3/14/13 6:50 PM, John Schneider wrote:
>> Peter,
>> 
>> Its really great to see some momentum on this! Thanks very much for 
>> the energy you are putting into it. This is something in which I've 
>> had a long-running interest. In fact, I think Peter and I first 
>> talked about it back in 2005 (yikes!). Although I'm not sure Peter 
>> was completely sold on the idea at the time (broccoli ice cream 
>> anyone? [1]), he was gracious enough to help us get the XMPP EXI use 
>> case together [2] and even started an early draft [3]. Thanks Peter! 
>> ;-)
> 
> One must keep an open mind.
> 
>> Here at work, we have incorporated EXI into various XMPP solutions to 
>> support our Efficient XML users (e.g., to enable XMPP on aircraft). 
>> So, I'm very interested in following your progress.
>> Please keep me in the loop as you move forward. I'd be very happy to 
>> help if I can.
> 
> Yesterday at IETF 86 I had a productive chat with Yusuke Doi on this 
> topic. He helped me see that, just as we've defined HTTP bindings 
> (BOSH / WebSocket) for web endpoints like browsers, an EXI binding 
> would expand the universe of XMPP usage to constrained devices of the 
> kind that might otherwise use protocols like CoAP.
> 
> Although I've not yet had time to read the new EXI proposal in detail, 
> I shall do so as soon as possible and provide comments on the 
> standards@xmpp.org list.
> 
> Peter
> 
> - --
> Peter Saint-Andre
> https://stpeter.im/
> 
> 
> -BEGIN PGP SIGNATURE-
> Version: GnuPG/MacGPG2 v2.0.18 (Darwin)
> Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
> 
> iQIcBAEBAgAGBQJRQvpOAAoJEOoGpJErxa2pcfMQAKPeGWf+02AZmrGff3hA+H+I
> g9INO8/62pukrj7utK1BuBRkersMXLxJUm3TIWnfQuqBXe/zgbkcvms42GHvu9+R
> J5/37i4Mq+lKHdI/XyPtLnn6/3YjriotGwl2ZKpvxvD4b66F0BL0jUaCoZx6jpPm
> N7QNYtX51uzpU7ofWfWf/IhzXOKNgFaB4u/EVcJp4+Gu8UInnualtkeq/ZHQwEiG
> SkpE36LfuCy31cXEd4Oankv30ywOwUmh2ETwzLyeDPzPfhFVdjgNkabnJr1J/H/n
> 8GnZNIEgzx/hFTGUetHkhMaHQfqGAVtEjWsFIYStugQnBRS6pP/V+fEME4DEMw64
> FLf9sF8sJBSP4/e62Z8myog/eVWrYiGNjGRu8qvo+fNmD4Fn+/qhHA/SvYcN+ZXN
> yqeqzAJty+A+oxduPCN+bbP93grroSjs1qmN7ybsu+bO9hJiDCs3IIsJPOBZrr9Y
> uQ+an33QWIAEsPRO4VKQlFYxHmh6QRjUPTCeFyFemuD7dyLk3UBXNikMUrvcFwBa
> uj0jAyC0tIf0w/Qm5RPtzVIV02On1jKRjHGoNLjEnjf0nusmdqzEZMq3mIL95PzL
> g1WYu5ASmDVAzWUuwvhYYMGraAQbqrnj5+QgiaAEHulpjl43ut7FHYslD7uyPvFp
> E8efql+LqrpSRCjgvZ8e
> =jNUK
> -END PGP SIGNATURE-
> 

Re: [Standards] Proposal for including EXI in XMPP

[DOI Yusuke]

Dear Randy,

From: Randy Turner 
Subject: Re: [Standards] Proposal for including EXI in XMPP
Date: Fri, 15 Mar 2013 08:08:25 -0700

> I was curious what the definition of "constrained" is ?  

You're right. Full-spec EXI does require certain amount of codes, but
one can implement dedicated EXI processor for a protocol if we select
appropriate set of options, schema, and some other aspects such as
schema evolution management scheme. In my experience, SEP2 works on
that way to make home appliances work with constrained hardware
resource.

Advantage of such EXI use against proprietary message format is
messages on the wire is still compatible with regular full-spec EXI
processor and you're free to construct servers with some regular EXI
processors.

// Yusuke DOI , TOSHIBA R&D Center

Re: [Standards] Proposal for including EXI in XMPP

[Randy Turner]

I was curious what the definition of "constrained" is ?  

EXI does produce a compact representation of XML (which is good if 
"constrained" is meant to apply to the amount of any output XML representation)

But I think the executable code size of an EXI implementation might not be 
appropriate for a "light switch", "low-power sensor"  or other similarly 
constrained memory device.

But I guess it depends on the definition of "constrained" memory?  (both RAM 
and non-volatile code space) -- especially since there may be a dual-stack 
IPv4/IPv6 stack, HTTP module, operating system, and other code already assumed 
to be on the constrained device.

Randy


On Mar 15, 2013, at 3:39 AM, Peter Saint-Andre  wrote:

> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA1
> 
> On 3/14/13 6:50 PM, John Schneider wrote:
>> Peter,
>> 
>> Its really great to see some momentum on this! Thanks very much for
>> the energy you are putting into it. This is something in which I've
>> had a long-running interest. In fact, I think Peter and I first
>> talked about it back in 2005 (yikes!). Although I'm not sure Peter
>> was completely sold on the idea at the time (broccoli ice cream
>> anyone? [1]), he was gracious enough to help us get the XMPP EXI
>> use case together [2] and even started an early draft [3]. Thanks
>> Peter! ;-)
> 
> One must keep an open mind.
> 
>> Here at work, we have incorporated EXI into various XMPP solutions
>> to support our Efficient XML users (e.g., to enable XMPP on
>> aircraft). So, I'm very interested in following your progress.
>> Please keep me in the loop as you move forward. I'd be very happy
>> to help if I can.
> 
> Yesterday at IETF 86 I had a productive chat with Yusuke Doi on this
> topic. He helped me see that, just as we've defined HTTP bindings
> (BOSH / WebSocket) for web endpoints like browsers, an EXI binding
> would expand the universe of XMPP usage to constrained devices of the
> kind that might otherwise use protocols like CoAP.
> 
> Although I've not yet had time to read the new EXI proposal in detail,
> I shall do so as soon as possible and provide comments on the
> standards@xmpp.org list.
> 
> Peter
> 
> - -- 
> Peter Saint-Andre
> https://stpeter.im/
> 
> 
> -BEGIN PGP SIGNATURE-
> Version: GnuPG/MacGPG2 v2.0.18 (Darwin)
> Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
> 
> iQIcBAEBAgAGBQJRQvpOAAoJEOoGpJErxa2pcfMQAKPeGWf+02AZmrGff3hA+H+I
> g9INO8/62pukrj7utK1BuBRkersMXLxJUm3TIWnfQuqBXe/zgbkcvms42GHvu9+R
> J5/37i4Mq+lKHdI/XyPtLnn6/3YjriotGwl2ZKpvxvD4b66F0BL0jUaCoZx6jpPm
> N7QNYtX51uzpU7ofWfWf/IhzXOKNgFaB4u/EVcJp4+Gu8UInnualtkeq/ZHQwEiG
> SkpE36LfuCy31cXEd4Oankv30ywOwUmh2ETwzLyeDPzPfhFVdjgNkabnJr1J/H/n
> 8GnZNIEgzx/hFTGUetHkhMaHQfqGAVtEjWsFIYStugQnBRS6pP/V+fEME4DEMw64
> FLf9sF8sJBSP4/e62Z8myog/eVWrYiGNjGRu8qvo+fNmD4Fn+/qhHA/SvYcN+ZXN
> yqeqzAJty+A+oxduPCN+bbP93grroSjs1qmN7ybsu+bO9hJiDCs3IIsJPOBZrr9Y
> uQ+an33QWIAEsPRO4VKQlFYxHmh6QRjUPTCeFyFemuD7dyLk3UBXNikMUrvcFwBa
> uj0jAyC0tIf0w/Qm5RPtzVIV02On1jKRjHGoNLjEnjf0nusmdqzEZMq3mIL95PzL
> g1WYu5ASmDVAzWUuwvhYYMGraAQbqrnj5+QgiaAEHulpjl43ut7FHYslD7uyPvFp
> E8efql+LqrpSRCjgvZ8e
> =jNUK
> -END PGP SIGNATURE-
> 

Re: [Standards] Proposal for including EXI in XMPP

[Yusuke DOI]

Dear folks,

As Peter mentioned, I believe XMPP and EXI should be a good oppotunity 
for both parties to make Internet-of-Things with good richness of data 
representation (i.e. end-to-end XML datamodel), responsiveness (i.e. 
XMPP IM/PubSub), and integration to real-world use cases (i.e. 
integration with everyone's IM environment).


As Takuki pointed out, the current proposal may have some impedance 
mismatch against current EXI specification. However, I believe the two 
groups can work togather to solve the problem. Let me contribute to the 
effort.


Best Regards,

// Yusuke DOI 

Re: [Standards] EXI extension proposal

[Yusuke DOI]

Dear Peter,

I believe we need to clarify some of 'requirements' first. Maybe, there
could be several approaches for EXI1.0 or maybe we need to propose
something to EXI1.x(maybe x=1), upon such requirement discussion.

Topics may involve:
- Minimal client requirement
- Minimal server requirement
- Schema management
- Efficiency requirement
- ...

I'll make another thread on requirement discussion following this mail.

(2013/03/13 12:26), Peter Waher wrote:

§3.3 in the XEP handles this. The server is free to add cache rules
to avoid explosion of number of derived schemas. The server is also
free to reject uploading or downloading requests, for any reason
(§2.4)


Agreed. BTW, is the schema itself encoded in EXI or plaintext? The 
example seems to be in plaintext but it could be EXI-encoded binary. 
Then upload in xs:base64binary will become straitforward Binary type 
upload, far more efficient.



The problem with this approach, is that name seldom change,
especially during development. And a slight change, a new attribute,
a new element, etc., will completely change the compression.
Furthermore, errors produced in this way will be extremely difficult
to debug. An efficient and fool-proof way to communicate using
different schema versions (having the same namespace and schema IDs)
is necessary.



§2.4 also proposes the possibility to install such schema files
manually on the server. The XEP allows for different scenarios.


Agreed. We need different mechanism for development stage and deployment
stage. I believe schemaLocation and hash is sufficient enough.


It is supposed that the EXI compression engine works in XML fragment
mode, where each stanza is compressed separately. I don't see that
self contained elements would not be required in this case.


Could you elaborate this? Are you thinking something like this?

[EXI Header][EXI Option]
SD
 SE(stream)
  SE(some-stanza) ... EE(some-stanza) // single stanza
 EE(stream)
ED
(padding)
SD
 SE(stream)
  SE(some-stanza) ... EE(some-stanza)
 EE(stream)
ED

or

[EXI Header][EXI Option]
SD
 SE(some-stanza) ... EE(some-stanza)
ED
(padding)
SD
 SE(some-stanza) ... EE(some-stanza)
ED

or something else?
(for those who are not familiar with EXI, SD/ED: start/end document, 
SE/EE: start/end element)


To be honest, I have never implemented EXI Fragment Document Grammar so 
my understanding may wrong.



Tables
should not be reused between stanzas, since tables can be very
small, but, as you point out in your example, number of possible
strings may be large (for instance many different JIDs). However,
possible strings within a message are much smaller, making also
references to tables shorter within the bit-packed message.


I think this makes far worse compression ratio (according to my experiment).


Finally it's up to the implementor how to setup the EXI compression
engine. Some may feel bit-packed is better, some that byte-packed is
better.

I'll add an implementation note in the XEP regarding this.


In my experience in SEP2[1], we didn't need full option for it. You may 
interested in EXI Profile[2], too.


[1] SEP2: smart energy profile 2
http://www.zigbee.org/Standards/ZigBeeSmartEnergy/Version20Documents.aspx

 SEP2 editors have had long discussion on how to use EXI in constrained 
devices and agreed to use it in schema-informed, non-strict grammar with 
schemaID and schema negotiation for linear extensibility.


[2] http://www.w3.org/TR/exi-profile/


This is a vital aspect of this proposal. For sensor networks and
IoT, especially wireless sensor networks, buffer size is first
priority. Therefore, EXI compression should be done with as much
information about schemas as possible.


Agreed.


We also see this as an important aspect of this proposal: Most
sensors will have pre-compiled code, often (semi-)automatically
generated from schema files, for compression and decompression of
EXI content. Therefore, the proposal Includes the possibility for
the client to reject the connection if parameters are not as
expected.


Agreed, but I think there should be some greatest common divisor (or 
'please implement at least this functionarity'-set) for better 
interoperability.


(from the other mail)

I've thought more about your proposition regarding session-wide
string tables vs. having to clear tables and buffers between stanzas.
Both have merits, depending on use case.

So, I added an option controlling this behavior: sessionWideBuffers,
as well as descriptions describing the differences in how they work.
As well as a note regarding flushing between stanzas.


We need to be careful to add something on existing standards. In other 
words, existing EXI processors should be applicable for this proposal.


Best Regards,

// Yusuke DOI 

[Standards] Requirement Clarification (Re: EXI extension proposal)

[Yusuke DOI]

Dear all,

I believe combination of XMPP and EXI should have great synergy and can 
extend the world of XMPP far more. However, because this space is very 
wide and I think it's better to clarify requirements on this 
combination, mainly towards IoT/M2M/sensor network use case of XMPP.


Let me start with enumeration of problem spaces.

- Minimal client requirement
How small a client implementation should be?
- Minimal server requirement
How small a server implementation should be?
- Message Efficiency requirement
How small a message should be?
- Schema management (in a communication and in the whole system)
How do we manage schema and its variants?
- (maybe more axis should be there)

In this mail, let me start with client/server requirements.
I think it is clear:

server requirements >= client requirements.

to make things work.

Choice would be something like this:

- Should a node accept plain text XML, in addition to EXI?
- Should a node accept introduction of new schema-informed grammar?
- Should a node accept various set of options? Which option should be 
selectable per session?


Combination of client and server requirement could be something like the 
followings:


[Proposal just for communication efficiency]
- Server and client should accept various EXI options and dynamic schema 
installation in addition to plain XML communication handler.


[Proposal for extremely constrained clients]
- Client may implement only EXI-based, fixed schema set. The client may 
be able to notify/upload the implemented schema set to its corresponding 
server.
- Server should implement both EXI and XML with dynamic schema 
installation with appropriate limited set of options.


Many other combination should be considered. Even binary-only servers 
could be possible. Of course, interoperability between current XMPP 
implementations will be lost without further consideration like 
Server-to-Server proxy.


Summary:
I think EXI is so flexible and thinking about 'what we really want to 
do' will greatly help to cut down the huge problem space.


Regards,

// Yusuke DOI 

Re: [Standards] Proposal for including EXI in XMPP

[Peter Waher]

Hello John,

Thanks for your mail. It sounds like very interesting projects you're working 
with. I would be happy to hear your thoughts, ideas, visions and comments 
regarding this. I'll also make sure I include you in any discussions regarding 
EXI.

Sincerely,
Peter Waher

From: John Schneider [mailto:john.schnei...@agiledelta.com]
Sent: den 14 mars 2013 19:51
To: Peter Waher; FABLET Youenn; public-...@w3.org
Cc: Joachim Lindborg (joachim.lindb...@sust.se); Yusuke DOI; Peter Saint-Andre; 
XMPP Standards; mact-...@att.net; Stephen Williams
Subject: Re: Proposal for including EXI in XMPP

Peter,

Its really great to see some momentum on this! Thanks very much for the energy 
you are putting into it. This is something in which I've had a long-running 
interest. In fact, I think Peter and I first talked about it back in 2005 
(yikes!). Although I'm not sure Peter was completely sold on the idea at the 
time (broccoli ice cream anyone? [1]), he was gracious enough to help us get 
the XMPP EXI use case together [2] and even started an early draft [3]. Thanks 
Peter! ;-)

Here at work, we have incorporated EXI into various XMPP solutions to support 
our Efficient XML users (e.g., to enable XMPP on aircraft). So, I'm very 
interested in following your progress. Please keep me in the loop as you move 
forward. I'd be very happy to help if I can.

  All the best!,

  John


  AgileDelta, Inc.

  
john.schnei...@agiledelta.com

  http://www.agiledelta.com


[1] http://pooh-99hi.blogspot.com/2005/11/peter-saint-andre-broccoliafter.html
[2] http://www.w3.org/TR/xbc-use-cases/#xmpp
[3] 
file:///jcs/Work/Standards/XMPP/xep--Exi.html



From: Peter Waher mailto:peter.wa...@clayster.com>>
Date: Thursday, March 14, 2013 7:38 AM
To: FABLET Youenn 
mailto:youenn.fab...@crf.canon.fr>>, 
"public-...@w3.org" 
mailto:public-...@w3.org>>
Cc: "Joachim Lindborg 
(joachim.lindb...@sust.se)" 
mailto:joachim.lindb...@sust.se>>, Yusuke DOI 
mailto:yusuke@toshiba.co.jp>>, Peter Saint-Andre 
mailto:stpe...@stpeter.im>>, XMPP Standards 
mailto:standards@xmpp.org>>, 
"mact-...@att.net" 
mailto:mact-...@att.net>>, Stephen Williams 
mailto:s...@lig.net>>
Subject: RE: Proposal for including EXI in XMPP
Resent-From: mailto:public-...@w3.org>>
Resent-Date: Thu, 14 Mar 2013 14:39:02 +

Dear Youenn,

Thank you for your response.

We also think EXI is exceptionally interesting and want to use it for other 
technologies as well. For instance, we are also looking at EXI compression in 
REST web services, especially for use with mobile devices hoping to improve 
latency by decreasing packet sizes.

Regarding using the preexisting EXI option header, I was also thinking the same 
thing in the beginning, but changed my mind thinking there are options that are 
not readily applicable in the XMPP case (for instance the Document/Fragment 
option), as well as options that are necessary in the EXI/XMPP case that are 
not applicable and available in the more general EXI case, such as the 
sessionWideBuffers option, added in revision 0.0.2. (I've attached the latest 
revision, with a note regarding this option §3.2).

Thank you for your time, and looking forward to your comments,
Sincerely,
Peter Waher


From: FABLET Youenn [mailto:youenn.fab...@crf.canon.fr]
Sent: den 14 mars 2013 11:00
To: Peter Waher; public-...@w3.org
Cc: Joachim Lindborg (joachim.lindb...@sust.se)
Subject: RE: Proposal for including EXI in XMPP

Hi,

That is an interesting piece of work.
My first thought when going through it is that EXI option negotiation may be a 
feature that is useful to more than XMPP.
That is very good that you bring it to the table.

Reading the spec quickly, I wonder whether the already-defined EXI header 
option element could be used for that purpose, especially in the case it can be 
serialized as EXI.

Anyway, I think the EXI WG will review this and get back to you.
Regards,
Youenn

From: Peter Waher [mailto:peter.wa...@clayster.com]
Sent: mercredi 13 mars 2013 17:43
To: public-...@w3.org
Cc: Joachim Lindborg (joachim.lindb...@sust.se)
Subject: Proposal for including EXI in XMPP

Hello

We have made a first draft of a proposal for incorporating EXI into XMPP 
networks. (See attached files.)

Anybody with an interest in EXI & XMPP are welcome to join us in this effort, 
please contact us. Any comments, suggestions, etc., on the contents of these 
documents are warmly welcomed and appreciated.

Sincerely,
Peter Waher

Re: [Standards] Proposal for including EXI in XMPP

[Joachim Lindborg]

We are in our projects using www.contiki-os.org/ created by www.sics.se and
constrained nodes and the aim is to use EXI and XMPP over IPv6 according to
the new XEP's with Sensor Networks.

/Joachim

2013/3/15 Peter Saint-Andre 

> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA1
>
> On 3/14/13 6:50 PM, John Schneider wrote:
> > Peter,
> >
> > Its really great to see some momentum on this! Thanks very much for
> > the energy you are putting into it. This is something in which I've
> > had a long-running interest. In fact, I think Peter and I first
> > talked about it back in 2005 (yikes!). Although I'm not sure Peter
> > was completely sold on the idea at the time (broccoli ice cream
> > anyone? [1]), he was gracious enough to help us get the XMPP EXI
> > use case together [2] and even started an early draft [3]. Thanks
> > Peter! ;-)
>
> One must keep an open mind.
>
> > Here at work, we have incorporated EXI into various XMPP solutions
> > to support our Efficient XML users (e.g., to enable XMPP on
> > aircraft). So, I'm very interested in following your progress.
> > Please keep me in the loop as you move forward. I'd be very happy
> > to help if I can.
>
> Yesterday at IETF 86 I had a productive chat with Yusuke Doi on this
> topic. He helped me see that, just as we've defined HTTP bindings
> (BOSH / WebSocket) for web endpoints like browsers, an EXI binding
> would expand the universe of XMPP usage to constrained devices of the
> kind that might otherwise use protocols like CoAP.
>
> Although I've not yet had time to read the new EXI proposal in detail,
> I shall do so as soon as possible and provide comments on the
> standards@xmpp.org list.
>
> Peter
>
> - --
> Peter Saint-Andre
> https://stpeter.im/
>
>
> -BEGIN PGP SIGNATURE-
> Version: GnuPG/MacGPG2 v2.0.18 (Darwin)
> Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
>
> iQIcBAEBAgAGBQJRQvpOAAoJEOoGpJErxa2pcfMQAKPeGWf+02AZmrGff3hA+H+I
> g9INO8/62pukrj7utK1BuBRkersMXLxJUm3TIWnfQuqBXe/zgbkcvms42GHvu9+R
> J5/37i4Mq+lKHdI/XyPtLnn6/3YjriotGwl2ZKpvxvD4b66F0BL0jUaCoZx6jpPm
> N7QNYtX51uzpU7ofWfWf/IhzXOKNgFaB4u/EVcJp4+Gu8UInnualtkeq/ZHQwEiG
> SkpE36LfuCy31cXEd4Oankv30ywOwUmh2ETwzLyeDPzPfhFVdjgNkabnJr1J/H/n
> 8GnZNIEgzx/hFTGUetHkhMaHQfqGAVtEjWsFIYStugQnBRS6pP/V+fEME4DEMw64
> FLf9sF8sJBSP4/e62Z8myog/eVWrYiGNjGRu8qvo+fNmD4Fn+/qhHA/SvYcN+ZXN
> yqeqzAJty+A+oxduPCN+bbP93grroSjs1qmN7ybsu+bO9hJiDCs3IIsJPOBZrr9Y
> uQ+an33QWIAEsPRO4VKQlFYxHmh6QRjUPTCeFyFemuD7dyLk3UBXNikMUrvcFwBa
> uj0jAyC0tIf0w/Qm5RPtzVIV02On1jKRjHGoNLjEnjf0nusmdqzEZMq3mIL95PzL
> g1WYu5ASmDVAzWUuwvhYYMGraAQbqrnj5+QgiaAEHulpjl43ut7FHYslD7uyPvFp
> E8efql+LqrpSRCjgvZ8e
> =jNUK
> -END PGP SIGNATURE-
>



-- 
Regards

Joachim Lindborg
CTO, systems architect

Sustainable Innovation AB
Adress: Box 55998 102 16 Stockholm
Besöksadress: Storgatan 31 (Malmgården)
Email: joachim.lindb...@sust.se, www.sust.se
Tel +46 706-442270

Re: [Standards] Proposal for including EXI in XMPP

[Peter Saint-Andre]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

On 3/14/13 6:50 PM, John Schneider wrote:
> Peter,
> 
> Its really great to see some momentum on this! Thanks very much for
> the energy you are putting into it. This is something in which I've
> had a long-running interest. In fact, I think Peter and I first
> talked about it back in 2005 (yikes!). Although I'm not sure Peter
> was completely sold on the idea at the time (broccoli ice cream
> anyone? [1]), he was gracious enough to help us get the XMPP EXI
> use case together [2] and even started an early draft [3]. Thanks
> Peter! ;-)

One must keep an open mind.

> Here at work, we have incorporated EXI into various XMPP solutions
> to support our Efficient XML users (e.g., to enable XMPP on
> aircraft). So, I'm very interested in following your progress.
> Please keep me in the loop as you move forward. I'd be very happy
> to help if I can.

Yesterday at IETF 86 I had a productive chat with Yusuke Doi on this
topic. He helped me see that, just as we've defined HTTP bindings
(BOSH / WebSocket) for web endpoints like browsers, an EXI binding
would expand the universe of XMPP usage to constrained devices of the
kind that might otherwise use protocols like CoAP.

Although I've not yet had time to read the new EXI proposal in detail,
I shall do so as soon as possible and provide comments on the
standards@xmpp.org list.

Peter

- -- 
Peter Saint-Andre
https://stpeter.im/


-BEGIN PGP SIGNATURE-
Version: GnuPG/MacGPG2 v2.0.18 (Darwin)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQIcBAEBAgAGBQJRQvpOAAoJEOoGpJErxa2pcfMQAKPeGWf+02AZmrGff3hA+H+I
g9INO8/62pukrj7utK1BuBRkersMXLxJUm3TIWnfQuqBXe/zgbkcvms42GHvu9+R
J5/37i4Mq+lKHdI/XyPtLnn6/3YjriotGwl2ZKpvxvD4b66F0BL0jUaCoZx6jpPm
N7QNYtX51uzpU7ofWfWf/IhzXOKNgFaB4u/EVcJp4+Gu8UInnualtkeq/ZHQwEiG
SkpE36LfuCy31cXEd4Oankv30ywOwUmh2ETwzLyeDPzPfhFVdjgNkabnJr1J/H/n
8GnZNIEgzx/hFTGUetHkhMaHQfqGAVtEjWsFIYStugQnBRS6pP/V+fEME4DEMw64
FLf9sF8sJBSP4/e62Z8myog/eVWrYiGNjGRu8qvo+fNmD4Fn+/qhHA/SvYcN+ZXN
yqeqzAJty+A+oxduPCN+bbP93grroSjs1qmN7ybsu+bO9hJiDCs3IIsJPOBZrr9Y
uQ+an33QWIAEsPRO4VKQlFYxHmh6QRjUPTCeFyFemuD7dyLk3UBXNikMUrvcFwBa
uj0jAyC0tIf0w/Qm5RPtzVIV02On1jKRjHGoNLjEnjf0nusmdqzEZMq3mIL95PzL
g1WYu5ASmDVAzWUuwvhYYMGraAQbqrnj5+QgiaAEHulpjl43ut7FHYslD7uyPvFp
E8efql+LqrpSRCjgvZ8e
=jNUK
-END PGP SIGNATURE-