Re: [Standards] LAST CALL: XEP-0322 (Efficient XML Interchange (EXI) Format)

[Yusuke DOI]

Sorry for late response...

> Please consider the following questions during this Last Call and send your 
> feedback to the standards@xmpp.org discussion list:
> 
> 1. Is this specification needed to fill gaps in the XMPP protocol stack or to 
> clarify an existing protocol?

I believe yes.

> 2. Does the specification solve the problem stated in the introduction and 
> requirements?

Yes. IoT-related items can work very efficiently over the protocol.

> 3. Do you plan to implement this specification in your code? If not, why not?

Partially yes. A student's open-sourced code (though, not fully-working
yet) is on https://github.com/akari-h/nex

> 4. Do you have any security concerns related to this specification?

I've not noticed at the point of writings. I'd read the comments
carefully...

> 5. Is the specification accurate and clearly written?

I welcome any comments.

Yusuke

Re: [Standards] IETF 89 and the XMPP WG meeting - are you going?

[Yusuke DOI]

Thank you!

Yusuke

(2014-02-20 10:04), bear wrote:

Yusuke,

I have created an account for you and a temporary password has been sent to you 
via email.


On Wed, Feb 19, 2014 at 7:58 PM, Yusuke DOI mailto:yusuke@toshiba.co.jp>> wrote:

(2014-02-13 02:39), Mike Taylor wrote:

Please do list your name on the wiki page if you are going so we can
start getting organized for this.


I'll do, but I need Wiki account to do so.
May I ask somebody to create my account on Wiki?
(I prefer doi, ydoi, or d0i as my username)

Thanks,

Yusuke




--
Bear

b...@xmpp.org <mailto:b...@xmpp.org> (email)
bea...@gmail.com <mailto:bea...@gmail.com> (xmpp, email)
b...@code-bear.com <mailto:b...@code-bear.com> (xmpp, email)
http://code-bear.com/bearlog (weblog)

PGP Fingerprint = 9996 719F 973D B11B E111  D770 9331 E822 40B3 CD29

Re: [Standards] IETF 89 and the XMPP WG meeting - are you going?

[Yusuke DOI]

(2014-02-13 02:39), Mike Taylor wrote:

Please do list your name on the wiki page if you are going so we can
start getting organized for this.


I'll do, but I need Wiki account to do so.
May I ask somebody to create my account on Wiki?
(I prefer doi, ydoi, or d0i as my username)

Thanks,

Yusuke

Re: [Standards] Event Logging over XMPP

[Yusuke DOI]

Peter,

(2013-11-11 0:43), Peter Waher wrote:


XML has many features that are very nice: Plugability and versioning
using namespaces, validation (schema), swearch (xpath) and
transformations/conversions (xslt).


I don't think it's beneficial to start JSON vs. XMPP discussion here. If 
we want to hand logs to JSON-based systems, this doesn't mean we should 
use JSON as-is. For example, we can still use JSON-ish XML instead. In 
the case, if we use complex namespace and XML-ish versioning, XML-JSON 
binding may become broken.


What I would like to clarify is use cases.


'logging' is a too broad word I think. What framework/logging
facilities in your mind?


I was thinking of network/admin logging, for instance like or based
on Syslog semantics (also IETF). Could be used to log events of
interest to system administrators, network architects, etc. Can also
be used to find problems or learn how IoT networks based on XMPP
work.


Then, I guess netconf/netmod(YANG) and SNMP will be a first choices. 
FYI, as IoT-related disucussion there are proposal on CoAP-based 
management interface (not adopted by IETF yet).


http://www.ietf.org/mail-archive/web/core/current/msg05102.html

--
// Yusuke DOI 

Re: [Standards] Event Logging over XMPP

[Yusuke DOI]

Hi Peter,

How are you doing?

I think it could be beneficial to aligned with existing logging 
facilities in mind. A 'formal' choice is netconf/netmod in IETF. In 
practice, there are many frameworks built over JSON.


'logging' is a too broad word I think. What framework/logging facilities 
in your mind?


(2013-11-10 3:24), Peter Waher wrote:

Hello

Is anybody aware of event logging extensions for XMPP? XEP 0163, 207 and
316 all relate to publish/subscript (personal) events as I can see. What
I’m looking for is system and network events for system administrators,
like Syslog, for instance, but over XMPP.

Best regards,

Peter Waher




--
// Yusuke DOI 

Re: [Standards] jabber-id email header

[Yusuke DOI]

+1 from Japan as well :-)

Yusuke

(2013-09-21 07:22), Joachim Lindborg wrote:

+1 from sweden as well.


2013/9/20 Alexey Melnikov mailto:alexey.melni...@isode.com>>

On 20/09/2013 03:44, Peter Saint-Andre wrote:

Back in 2007 we defined an email header for advertising your Jabber ID
in your email messages:

http://tools.ietf.org/id/__draft-saintandre-jabberid-08.__txt 


For various reasons we never got this header registered with IANA, but
on re-reading RFC 3864 I've concluded that we could add it to the
provisional register through a XEP instead of an RFC.

Can you do an Independent submission to RFC Editor? You already have a 
draft...

Is there still interest in registering this header?

Yes yes yes :-).




--
Regards

Joachim Lindborg
CTO, systems architect

Sustainable Innovation AB
Adress: Box 55998 102 16 Stockholm
Besöksadress: Storgatan 31 (Malmgården)
Email: joachim.lindb...@sust.se , www.sust.se 

linkedin: http://www.linkedin.com/in/joachimlindborg
Tel +46 706-442270

Re: [Standards] repository of xmpp academic papers & white papers

[Yusuke DOI]

Hi,

Let me volunteer to make an initial list on mendeley. If the list looks good, 
it can be exported to xmpp.org or anywhere else.
(Actually, the list is open and anybody can contribute on it)

http://www.mendeley.com/groups/3516891/xmpp/

Regards,

Yusuke





(2013-07-16 06:33), Joachim Lindborg wrote:

Isn't the wiki a good place to keep links and also upload such docs?

Den 15 jul 2013 22:52 skrev "Peter Saint-Andre" mailto:stpe...@stpeter.im>>:

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

On 7/15/13 9:57 AM, Peter Waher wrote:
 > Hello
 >
 >
 >
 > Is there a repository of known academic papers & white papers
 > related to XMPP somewhere? If not, would it not be of interest to
 > have such a repository with such papers or links to them? Perhaps
 > in some logical order.

We don't have such a repository. We've talked before about starting a
repo of whitepapers, but we never seem to have the energy to launch it.

Peter

- --
Peter Saint-Andre
https://stpeter.im/


-BEGIN PGP SIGNATURE-
Version: GnuPG/MacGPG2 v2.0.19 (Darwin)
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQIcBAEBAgAGBQJR5GEbAAoJEOoGpJErxa2p3+QP/3SXU5jCiwf4EnDGl2zk4jSp
cI4DoKBA5CptY8rsAT7ouIV2+/oCdAuouYX10FCE+kDn7GmdmzzVsz4tOj9ocObA
i2HuV9lZM0p7EWH3xoCNWsj+Ck16XJYfKmfUNKF0l8hjlhJGwY50vseO0DxHB9qX
0Dhxz7I+fU6mtHMdeKISPYXmM4UOZuRhGaDVDkNux/N70/y8MZ3LCDyASZhTlmB1
rLJUykpCKTMJX9g1lsplZqaTG++J+3M8sp3HSYrcUEs9ujeQond6PqrqA7iI3bVy
ratMyXZDUY8fEiEcazY5gXB83s5+uqoXWj8nFU64z9kpGmT9K+SVJ1g1bEwPRsVe
UCney25QxlMsAAaTJYGeKQFFujpXnw91pdkrMqrElrhR3R/60lnBEIhZZUiDWxoX
mKBKjXzczaXdFVwn7ZhFFsKmNNxD4Si/4Q8wfHcDWQUpllWNtLJk63J1h815SYUx
wyjtan33cImWe5EPIpdAhDkfwAEsGPy20fOsbWfQWeRtfm4aGJ//zloIRCJdtYiJ
LMLEIZnXw2Io104k0bySahMXGPi2iriFA5W0FRa/eO1hTou6+cMG4KQyscG3VDVD
+tfIMF9AbZhD17tpqBjOnSNt9FCqcsF5kkKRw+mCX/cT78eO4XpH2dsYJay7Z2DV
7hkAucrUZHCIG37TNKEB
=62Vy
-END PGP SIGNATURE-

Re: [Standards] NetConf over XMPP

[Yusuke DOI]

Dier Michal,

I'm interested in this activity in terms of machine-to-machine 
communication/configuration over XMPP.

Regards,

Yusuke


(2013-05-30 16:04), Michal 'vorner' Vaner wrote:

Hello

I'm working with protocol called NetConf (RFC 6241) now. The protocol can be
used to configure devices remotely over something more standardized than web
interface or ssh & command line utilities (which is good for people but not for
automated scripts).

The protocol uses XML messages and defines several transport layers to work on
top of (SSH, TLS socket, …). It seems like a reasonable idea to define a
transport over XMPP, which has several advantages (like you don't need 1000 ssh
sessions to configure 1000 devices on your client, there's single point where we
could define who is allowed to do the configuration, etc).

Also, it could be used to configure software as well (XMPP bots, for example),
with possibly better UI than the ad-hoc commands.

I brought the idea with my boss today that I'd like to do write the XEP and some
experimental implementation. I got an answer in the meaning of „That sounds like
a cool idea, though we need to finish the current project first“.

So, I will probably find the time to write the proposal XEP some time in the
future. But if anyone is interested in a way to configure something remotely
(either a device or a software), please contact me, we may want to exchange some
ideas.

Thank you

PS.: It seems the email didn't get through when sent from company email, so I'm
retrying with a personal one (which is subscribed to the list). Please ignore
any possible duplicates.

Re: [Standards] Using .well-known/ to supplement XEP-0156

[Yusuke DOI]

Peter,

(2013-05-19 03:13), Peter Waher wrote:

What about the UPnP method? Using SSDP? Creating some well defined
UPnP Device interface for XMPP Servers & XMPP Clients, perhaps
exposing the features of each device at the same time? Saves time as
you don't have to do service discovery on each device.


UPnP is not for browsers and I think this is not what Lance needs.

Regards,

Yusuke

Re: [Standards] Using .well-known/ to supplement XEP-0156

[Yusuke DOI]

Dear Lance,

I don't think ordinary browsers will use binary XML (EXI) very soon (even DNS 
may be omitted on IoT devices), but if you're going to make entries of 
alternate transport, please make room for XMPP over EXI as alternate transport 
binding. Next version of XEP-0322 will include the idea.

Regards,

Yusuke

(2013-05-18 11:12), Lance Stout wrote:

So XEP-0156 details how to use DNS records to advertise alternative ways to 
connect to an XMPP server, namely BOSH and soon WebSocket. That's great, except 
that in the browser you can't make arbitrary DNS requests, which severely 
limits the usefulness of this method.

Any interest in standardizing some .well-known/ document to expose alternative 
connection endpoints? There's probably something like this that already exists 
that we can profile or register with, right?

-- Lance

Re: [Standards] EXI proposal objections/responses

[Yusuke DOI]

oops


Maybe this is my comment (sent: Thu, 21 Mar 2013 01:11:49 +0900).I need to 
clarify two points before I agree on this point.


text in parenthes should be removed (this is timestamp of the e-mail I sent to 
state the problem). Please ignore.

Yusuke

Re: [Standards] EXI proposal objections/responses

[Yusuke DOI]

Hi Peter,


(2013-03-21 07:18), Peter Waher wrote:

* Some concerns have been raised that this proposal has slight 
variations from regular EXI:

I would not say so. The compression/decompression part is the same. EXI options 
are the same, except a few that have been removed since they do not work well 
together with XMPP. Also, an XMPP-specific (relating to how the XMPP server 
should use the EXI compression/decompression engine) option has been added 
(which the server can reject if it wants to).


Maybe this is my comment (sent: Thu, 21 Mar 2013 01:11:49 +0900).I need to 
clarify two points before I agree on this point.

1) streamWideBuffer (I think we share the problem, I don't restate it here).
2) stream structure (see below)

(copied from the previous mail)

Added a note in §2.7. regarding this, since it was unclear. 
is only written at the start of a stream, and  when the
stream is closed, with stanzas inbetween. When entering
EXI-compressed mode, the  and  should be omitted.


I'm not quite sure ... You mean something like this?

...

 
SD
SE("iq")
 ... (contents of iq)
EE
ED
 (fflush(), PSH)
SD
SE("message")
 ... (contents of message)
EE
ED
 (fflush(), PSH)
(/copied)

This is not an EXI stream, this is a set of EXI streams on the same TCP session (I'm 
not opposing on it, it's a good idea). However, this structure changes overall 
document structure: no  tag on EXI part. XML equivalent is:

[from here EXI stream starts]
 ...  
 ...  
[End of TCP session]

I'm not the person who can say it's good or not good (I'm not XMPP spec geek, 
yet).

Yusuke

Re: [Standards] Requirement Clarification (Re: EXI extension proposal)

[Yusuke DOI]

Dear Peter,

(2013/03/20 4:41), Peter Waher wrote:



- Minimal client requirement How small a client implementation
should be?


In what way would this affect the XEP? I see this as an
implementation detail that can be left to product developers.


- Minimal server requirement How small a server implementation
should be?


This is also an implementation detail left to product developers.


In general, this requirement should be reflected to decision of 
mandatory items in a proposal. For example, 'all EXI options should be 
supported in every client' makes client much larger than 'option 
document option is only mandatory option'. A client may start 
negotiation in EXI instead of XML may reduce the size of the 
implementation furthermore.


I think your 'left to developers (no assumptions)' is a possible choice.


- Message Efficiency requirement How small a message should be?


This would depend on use case, overall IT architecture, etc. What's
small to one application might be huge for another.


Ah, I should say: 'how small message (compared to uncompressed XML 
message) should be worth working on this topic?' In addition, any other 
characteristics (such as implementation simplicity) could be goal of a 
proposal.



- Schema management (in a communication and in the whole system)
How do we manage schema and its variants?


The XEP only defines that different versions of schemas with the same
namespace must be handled. It also provides a small implementation
note how this could be achieved on the server side.


I see. I mean there could be different approach on different management 
style. I'm trying to state the problem space we have.



- (maybe more axis should be there)


A couple of interesting requirements arose as a response to input
from Rumen and Takuki, regarding uploading pre-compressed schema
files. Requirements on pre-set options for such pre-compressed schema
files and what is supposed to be a minimum of requirements for the
server.


I believe it's good to have a default set of options.
For precompression and identification thing, you mentiond variation on 
encoding in your proposal. For XML it has the same problem

(You may find 'canonical XML' interesting)


server requirements >= client requirements.


True. In effect, the entire process is client driven, since it is
assumed the client is the weak link.


I agree in general.


- Should a node accept plain text XML, in addition to EXI?


With node, I assume you mean client? In this case, at least
sufficiently to perform the negotiation. It's the client that
decides. (This is noted in the XEP.)


I mean this 'should' in context of specification. If client have choice 
not to implement XML, I would say a node may (not 'should') use XML. 
Your XEP seems for me (correct me if I'm wrong) clients and servers must 
be able to speak XML to (at least) start the XEP-138 negotiation.



- Should a node accept introduction of new schema-informed
grammar?


If node=client, then the client never receives schemas from the
server. The client knows at compile time what namespaces it can
interpret, and should provide those schemas. So, the only probable
reason for not having a specific schema, is if something it does not
understand is being sent. Why should the client provide logic and
space for handling this?


Agree. I mean this is enumeration of decisions to clarify the problem 
space. Even it seems to be clear I think worth clarify.



- Should a node accept various set of options? Which option should
be selectable per session?


Finally it's up to the client to decide. Takuki wrote a mail on the
same topic where he proposed an order of the different options,
whereby if a party knows how to handle a "difficult" option, it
should be able to handle "easier" options.


Again, 'should' is in context of the specification. Letting everything 
up to client's implementor could be a choice. However, a mandated 
'greatest common divisor' option (as you stated in exchange of 
compressed schema) works good for interoperability and I mean that is 
the option a node 'should accept'.



Combination of client and server requirement could be something
like the followings:

[Proposal just for communication efficiency] - Server and client
should accept various EXI options and dynamic schema installation
in addition to plain XML communication handler.


It might be valid for the client to only support one particular set
of options. However, the server should be able to understand all
options for that EXI version. However, for embedded XMPP servers, the
domain of allowable EXI options might have to be restricted.


I agree your point but your statement is not for 'proposal just for 
communication efficiency'.



[Proposal for extremely constrained clients] - Client may implement
only EXI-based, fixed schema set. The client may be able to
notify/upload the implemented schema set to its corresponding
server. - Server should implement both EXI and XML with dynamic
schema 

Re: [Standards] EXI extension proposal

[Yusuke DOI]

(2013/03/20 5:47), Peter Waher wrote:

Dear Yusuke

Thanks for your mail. Regarding your questions and comments:


Ok, so I think I need to write another proposal. Would you mind if
I re-use the basic idea on 'EXI encoding part' of your proposal to
make better interoperability?


By all means. It would be excellent if you did. It would allow
devices to be able to connect freely between servers supporting
either of the two versions.


Thanks :-)


The proposal supports different EXI versions. It's part of the
negotiation, using the version attribute.


I see, then may I assume you have no intention to make 'yet another
EXI for XMPP' such as sessionWideBuffer option?


I'm not sure what you mean with 'yet another EXI for XMPP'. Where do
you find another proposal for EXI with option negotiation?

Also, we plan to let the sessionWideBuffer to stand for the time
being. I believe it's an important aspect of the integration of EXI
into XMPP. And as I showed using my examples (which are in no ways
extreme, they are in fact two common use cases in sensor networks)
sometimes one option is better than the other. One option is not the
best for all use cases.


I would say 'yet another EXI for XMPP' for EXI streams not be able to be 
decoded by current EXI-1.0 compatible implementations, such as EXIP, 
OpenEXI, and EXIficient. Your proposal seems to expect somewhat 
different handling in EXI stream structure, including sessionWideBuffer.


I agree re-use of string table is important in terms encoding 
efficiency. However, how to make it needs careful consideration. 
Inter-stream session management could be a good idea worth standardized 
in future version of EXI (however, this may involve out-of-EXI session 
negotiation such as something like EXI-cookie option to find identical 
session state). If you have long term point of view, it's better for 
XMPP folks and EXI folks to make some discussion to solve.


(BTW, my proposal to handle this is to add NOP rule to let an element be 
pushed out even in bit-packed stream)


On the other hand, if you need something you can use NOW, I believe you 
need to stay in compatible stream structure. (You may think differently, 
this is just my opinion).



Added a note in §2.7. regarding this, since it was unclear. 
is only written at the start of a stream, and  when the
stream is closed, with stanzas inbetween. When entering
EXI-compressed mode, the  and  should be omitted.


I'm not quite sure ... You mean something like this?

...

 
SD
SE("iq")
 ... (contents of iq)
EE
ED
 (fflush(), PSH)
SD
SE("message")
 ... (contents of message)
EE
ED
 (fflush(), PSH)

Regards,

// Yusuke DOI 

Re: [Standards] Proposal for including EXI in XMPP

[Yusuke DOI]

Peter,

(2013/03/20 7:06), Peter Waher wrote:

What EXI options should we require the XMPP server to support (MUST)? If
we could agree on a set of options the server must implement we can make
the negotiation simpler, and clients can assume these options to be
supported.



I'd rather like to limit single or very limited set of options for 
constrained devices with (as Takuki said) somewhat of 'sequential 
constrained-level'. One candidate is the option used for EXI option 
documents.


Yusuke

Re: [Standards] EXI extension proposal

[Yusuke DOI]

Dear Peter,

(2013-03-19 17:22), Peter Saint-Andre wrote:

-BEGIN PGP SIGNED MESSAGE-
If I understand correctly, a "binary-only bootstrap mechanism for EXI"
would involve using a separate port and would effecitvely define a new
transport binding for XMPP (in addition to the existing bindings =
TCP, BOSH, and WebSocket).


Thank you for giving the right direction. I think what to be desgined is 
following three.

1) EXI in compression mechanism : Peter Waher's proposal
2) Encoding detail : also in Peter's proposal, could be a common ground
3) EXI as alternative transport binding

I have no much time until the next week but after that I'll make up my (our? is 
there anybody who is interested in EXI-as-binding?) proposal.

Regards,

Yusuke

Re: [Standards] EXI extension proposal

[Yusuke DOI]

Peter,
(sorry for multiple mail, I did wanted to split technical details and 
requirement discussions)


(2013/03/18 22:53), Peter Waher wrote:

Stepping back to the requirements. For example, what I need is binary-only 
bootstrap mechanism for EXI (not like XEP-0138) with least negotiation/propose 
of compression parameter. Does your proposal cover such use case?


No. This XEP covers EXI negotiation in a way compliant with XEP-0138.


Ok, so I think I need to write another proposal. Would you mind if I 
re-use the basic idea on 'EXI encoding part' of your proposal to make 
better interoperability?



Let me ask a question: which strategy are you taking?

  1) short target: find a way to encode XMPP stanza with existing
 EXI format 1.0
  2) long target: find a way to encode XMPP stanza ideally with
 proposed currently-nonexistent ideal EXI 1.1 (or 2.0)


The proposal supports different EXI versions. It's part of the negotiation, 
using the version attribute.


I see, then may I assume you have no intention to make 'yet another EXI 
for XMPP' such as sessionWideBuffer option?



My concern on a-1 is about XML semantics. Does current server implementation 
okay to have modified XML-level structure? On the other hand, re-use of string 
table is possible only in a-3 (discussion below).


Sorry, I don't understand your question.


In my (yet) shallow understandings, your approach to send a stanza looks 
like following (correct me if I'm wrong)



 

(padding, fflush(), PSH)

 

(again, padding, fflush(), PSH)

For me, it looks different from the semantics of  tag defined 
in RFC6120 section 4.4:

If the parties are using either two streams over a single TCP connection or two 
streams over two TCP connections, the entity that sends the closing stream tag 
MUST behave as follows:

1.Wait for the other party to also close its outbound stream before 
terminating the underlying TCP connection(s); this gives the other party an 
opportunity to finish transmitting any outbound data to the closing entity 
before the termination of the TCP connection(s).
2.Refrain from sending any further data over its outbound stream to the 
other entity, but continue to process data received from the other entity (and, 
if necessary, process such data).
3.Consider both streams to be void if the other party does not send its closing 
stream tag within a reasonable amount of time (where the definition of 
"reasonable" is a matter of implementation or deployment).
4.After receiving a reciprocal closing stream tag from the other party or 
waiting a reasonable amount of time with no response, terminate the underlying 
TCP connection(s).




For dynamic grammars, your proposal "reusing the same options as used by the 
stream" (section 3.2) may not be adequate, because this means all stanzas should be in 
the same schema and does not allow introduction of new schema on the fly. However, I agree a 
> constrained node seldom updates its functionality so fixed set of schema on C2S pair 
should be okay (for S2S communication it's not good).


I don't see why this should require all stanzas to be in the same schema. 
Schemas are handled by namespace, and a collection of schemas can be used in 
compression. Which schema namespaces are to be included is part of the 
negotiation.


Sorry, I mean 'the set of schema given in the negotiation phase.' S2S 
connection may be required to re-negotiated if a new client connect to a 
server with schemas not included in the running S2S communication.


Regards,

Yusuke

[Standards] Requirement Clarification (Re: EXI extension proposal)

[Yusuke DOI]

Dear Peter and all,

(2013/03/18 22:53), Peter Waher wrote:

Is a mail with the subject 'Requirement Clarification' sent to the list?


I've not received one.


Oops, It seems to be sent to different list? Let me re-send one.

  ---

I believe combination of XMPP and EXI should have great synergy and can 
extend the world of XMPP far more. However, because this space is very 
wide and I think it's better to clarify requirements on this 
combination, mainly towards IoT/M2M/sensor network use case of XMPP.


Let me start with enumeration of problem spaces.

- Minimal client requirement
How small a client implementation should be?
- Minimal server requirement
How small a server implementation should be?
- Message Efficiency requirement
How small a message should be?
- Schema management (in a communication and in the whole system)
How do we manage schema and its variants?
- (maybe more axis should be there)

In this mail, let me start with client/server requirements.
I think it is clear:

server requirements >= client requirements.

to make things work.

Choice would be something like this:

- Should a node accept plain text XML, in addition to EXI?
- Should a node accept introduction of new schema-informed grammar?
- Should a node accept various set of options? Which option should be 
selectable per session?


Combination of client and server requirement could be something like the 
followings:


[Proposal just for communication efficiency]
- Server and client should accept various EXI options and dynamic schema 
installation in addition to plain XML communication handler.


[Proposal for extremely constrained clients]
- Client may implement only EXI-based, fixed schema set. The client may 
be able to notify/upload the implemented schema set to its corresponding 
server.
- Server should implement both EXI and XML with dynamic schema 
installation with appropriate limited set of options.


Many other combination should be considered. Even binary-only servers 
could be possible. Of course, interoperability between current XMPP 
implementations will be lost without further consideration like 
Server-to-Server proxy.


Summary:
I think EXI is so flexible and thinking about 'what we really want to 
do' will greatly help to cut down the huge problem space.


Regards,

// Yusuke DOI 

Re: [Standards] EXI extension proposal

[Yusuke DOI]

epend on type of information sent.

Example:

  ... 100 times (inly Energy, MWh) 

May be better compressed if string tables are renewed, since bit
field lengths decrease and strings are short, rather than if it has
to share space with, for instance:

  ...

20 times, with different field names and units. 

Depends on use case. So manufacturer may have to create tests
depending on the use of the device, and schemas used.


I agree we need test but I don't get your point in the example (looks
like an extreme case with just slight better compression -- 40 of
labels makes 6-bit string table identifier. The 100 records equiv. to 75
characters = re-use of 6-10 JIDs will compensate the cost).

But anyway,

We need to be careful to add something on existing standards. In
other words, existing EXI processors should be applicable for this
proposal.


Correct. This option is for the XMPP implementation, on how to
use/configure existing EXP processors. It's not an option for the
EXI processor itself.


I strongly oppose to this idea (sessionWideBuffer) as long as your
proposal is to encode each stanza as standalone EXI document. Re-use of
string table is not allowed in the EXI spec.

Best Regards,

// Yusuke DOI 

Re: [Standards] Proposal for including EXI in XMPP

[Yusuke DOI]

Dear folks,

As Peter mentioned, I believe XMPP and EXI should be a good oppotunity 
for both parties to make Internet-of-Things with good richness of data 
representation (i.e. end-to-end XML datamodel), responsiveness (i.e. 
XMPP IM/PubSub), and integration to real-world use cases (i.e. 
integration with everyone's IM environment).


As Takuki pointed out, the current proposal may have some impedance 
mismatch against current EXI specification. However, I believe the two 
groups can work togather to solve the problem. Let me contribute to the 
effort.


Best Regards,

// Yusuke DOI 

Re: [Standards] EXI extension proposal

[Yusuke DOI]

Dear Peter,

I believe we need to clarify some of 'requirements' first. Maybe, there
could be several approaches for EXI1.0 or maybe we need to propose
something to EXI1.x(maybe x=1), upon such requirement discussion.

Topics may involve:
- Minimal client requirement
- Minimal server requirement
- Schema management
- Efficiency requirement
- ...

I'll make another thread on requirement discussion following this mail.

(2013/03/13 12:26), Peter Waher wrote:

§3.3 in the XEP handles this. The server is free to add cache rules
to avoid explosion of number of derived schemas. The server is also
free to reject uploading or downloading requests, for any reason
(§2.4)


Agreed. BTW, is the schema itself encoded in EXI or plaintext? The 
example seems to be in plaintext but it could be EXI-encoded binary. 
Then upload in xs:base64binary will become straitforward Binary type 
upload, far more efficient.



The problem with this approach, is that name seldom change,
especially during development. And a slight change, a new attribute,
a new element, etc., will completely change the compression.
Furthermore, errors produced in this way will be extremely difficult
to debug. An efficient and fool-proof way to communicate using
different schema versions (having the same namespace and schema IDs)
is necessary.



§2.4 also proposes the possibility to install such schema files
manually on the server. The XEP allows for different scenarios.


Agreed. We need different mechanism for development stage and deployment
stage. I believe schemaLocation and hash is sufficient enough.


It is supposed that the EXI compression engine works in XML fragment
mode, where each stanza is compressed separately. I don't see that
self contained elements would not be required in this case.


Could you elaborate this? Are you thinking something like this?

[EXI Header][EXI Option]
SD
 SE(stream)
  SE(some-stanza) ... EE(some-stanza) // single stanza
 EE(stream)
ED
(padding)
SD
 SE(stream)
  SE(some-stanza) ... EE(some-stanza)
 EE(stream)
ED

or

[EXI Header][EXI Option]
SD
 SE(some-stanza) ... EE(some-stanza)
ED
(padding)
SD
 SE(some-stanza) ... EE(some-stanza)
ED

or something else?
(for those who are not familiar with EXI, SD/ED: start/end document, 
SE/EE: start/end element)


To be honest, I have never implemented EXI Fragment Document Grammar so 
my understanding may wrong.



Tables
should not be reused between stanzas, since tables can be very
small, but, as you point out in your example, number of possible
strings may be large (for instance many different JIDs). However,
possible strings within a message are much smaller, making also
references to tables shorter within the bit-packed message.


I think this makes far worse compression ratio (according to my experiment).


Finally it's up to the implementor how to setup the EXI compression
engine. Some may feel bit-packed is better, some that byte-packed is
better.

I'll add an implementation note in the XEP regarding this.


In my experience in SEP2[1], we didn't need full option for it. You may 
interested in EXI Profile[2], too.


[1] SEP2: smart energy profile 2
http://www.zigbee.org/Standards/ZigBeeSmartEnergy/Version20Documents.aspx

 SEP2 editors have had long discussion on how to use EXI in constrained 
devices and agreed to use it in schema-informed, non-strict grammar with 
schemaID and schema negotiation for linear extensibility.


[2] http://www.w3.org/TR/exi-profile/


This is a vital aspect of this proposal. For sensor networks and
IoT, especially wireless sensor networks, buffer size is first
priority. Therefore, EXI compression should be done with as much
information about schemas as possible.


Agreed.


We also see this as an important aspect of this proposal: Most
sensors will have pre-compiled code, often (semi-)automatically
generated from schema files, for compression and decompression of
EXI content. Therefore, the proposal Includes the possibility for
the client to reject the connection if parameters are not as
expected.


Agreed, but I think there should be some greatest common divisor (or 
'please implement at least this functionarity'-set) for better 
interoperability.


(from the other mail)

I've thought more about your proposition regarding session-wide
string tables vs. having to clear tables and buffers between stanzas.
Both have merits, depending on use case.

So, I added an option controlling this behavior: sessionWideBuffers,
as well as descriptions describing the differences in how they work.
As well as a note regarding flushing between stanzas.


We need to be careful to add something on existing standards. In other 
words, existing EXI processors should be applicable for this proposal.


Best Regards,

// Yusuke DOI 

[Standards] Requirement Clarification (Re: EXI extension proposal)

[Yusuke DOI]

Dear all,

I believe combination of XMPP and EXI should have great synergy and can 
extend the world of XMPP far more. However, because this space is very 
wide and I think it's better to clarify requirements on this 
combination, mainly towards IoT/M2M/sensor network use case of XMPP.


Let me start with enumeration of problem spaces.

- Minimal client requirement
How small a client implementation should be?
- Minimal server requirement
How small a server implementation should be?
- Message Efficiency requirement
How small a message should be?
- Schema management (in a communication and in the whole system)
How do we manage schema and its variants?
- (maybe more axis should be there)

In this mail, let me start with client/server requirements.
I think it is clear:

server requirements >= client requirements.

to make things work.

Choice would be something like this:

- Should a node accept plain text XML, in addition to EXI?
- Should a node accept introduction of new schema-informed grammar?
- Should a node accept various set of options? Which option should be 
selectable per session?


Combination of client and server requirement could be something like the 
followings:


[Proposal just for communication efficiency]
- Server and client should accept various EXI options and dynamic schema 
installation in addition to plain XML communication handler.


[Proposal for extremely constrained clients]
- Client may implement only EXI-based, fixed schema set. The client may 
be able to notify/upload the implemented schema set to its corresponding 
server.
- Server should implement both EXI and XML with dynamic schema 
installation with appropriate limited set of options.


Many other combination should be considered. Even binary-only servers 
could be possible. Of course, interoperability between current XMPP 
implementations will be lost without further consideration like 
Server-to-Server proxy.


Summary:
I think EXI is so flexible and thinking about 'what we really want to 
do' will greatly help to cut down the huge problem space.


Regards,

// Yusuke DOI 

Re: [Standards] Proposed XMPP Extension: Using Efficient XML Interchange (EXI) Format in XMPP

[Yusuke DOI]

Dear Evgeniy,

EXIP is the only known C-implemented library available as open-source.
If you're going to make a product, AgileDelta may help (not free).

http://exip.sourceforge.net/
http://www.agiledelta.com/

Regards,

Yusuke

(2013/03/14 0:48), Evgeniy Khramtsov wrote:

Are there any fast and robust C-implemented libraries for exi recommended?

Re: [Standards] EXI extension proposal

[Yusuke DOI]

Dear Peter,

(2013/03/13 23:10), Peter Waher wrote:

Anybody interested in EXI & XMPP, please review. Any feedback is most welcome.


I think this idea of schema exchange is interesting. On the other hand, 
it may make confusion on management (explosion of the number of derived 
schemas). If there is a way to name a XML schema defined in XEP, servers 
and clients can share them by the names. Of course (1) this does not 
eliminate needs to upload XML schema because the end device may have 
non-XEP vendor specific extensions (2) we need secure channel to 
download XEP-defined schemas to avoid attacks.


Another pitfall: if we want to use bit-packed, we need to make stanzas 
encoded as self contained elements. Otherwise you cannot do 'fflush()' 
at the end of element of a stanza. However, self contained elements do 
not allow an encoder to re-use compression context (string tables) 
between outside of the element and the element itself. This means the 
encoder need to re-encode JID strings as is (otherwise you can just 
encode a string with few bytes of reference). This may make the XMPP/EXI 
stream more inefficient compared to byte-aligned streams. My preliminary 
experiment shows following results.


[The number of bytes]
bytes
-+---
XML  |  3681
selfContained, bit-packed|  1589
byte-aligned |  1358
-+---

Note: the results do not use schema-informed grammars to encode 
XEP-based elements, so compression ratio of Peter's proposal should be 
much better -- in my best scenario with schema-informed EXI, it will be 
809 bytes (22% of original XML).




BTW, my initial idea is somewhat different. What I want to make is 
constrained XMPP clients (and if technically possible, servers) with 
static set of pre-compiled EXI grammars and without ability to talk with 
regular XML-based XMPP. This enables nodes with batteries to speak 
sensor data with narrow wireless link such as 15.4 or with 3G link 
charged by quantity. Maybe this idea is oriented towards SRV-based 
negotiation.


For long-targetted apporach, I think I can propose some update to EXI 
spec itself (I recently joined to W3C EXI working group). Now EXI 
working group are open to collect requirements for EXI2.0 (I already 
raised fflush() issue). I believe this kind of collaboration should be 
very important to let more constrained IoT devices join the network.


Regards,

Yusuke

Re: [Standards] Question on valid XMPP

[Yusuke DOI]

Dear Peter,

Yes, for sensor networks such as one on 802.15.4, XMPP/EXI should be a good 
choice (if we can implement it efficiently). Please let me join your effort. 
Thanks!

// Yusuke DOI  Corporate R&D Center, TOSHIBA Corp.


(2013-03-07 20:55), Peter Waher wrote:

Hello Yusuke
We have been experimenting with EXI and find it an exceptional and efficient 
way to compress XML. We want to use it in sensor networks, where RAM and packet 
size is an issue. We hope to, within our effort to create XEPs for sensor 
networks, to include a XEP for EXI enablement of XMPP communication. If you 
want to join us in this work, you're welcome.
Note to XSF members: Anybody interested in participating in such an EXI XEP, 
please contact me.
Sincerely,
Peter Waher

Från: Yusuke DOI mailto:yusuke@toshiba.co.jp>>
Datum: 6 mars 2013 07:32:55 CET
Till: XMPP Standards mailto:standards@xmpp.org>>
Ämne: [Standards] Question on valid XMPP
Svara till: XMPP Standards mailto:standards@xmpp.org>>
Hi,

My name is Yusuke Doi. I'm wondering if it is possible to put EXI[1] with XMPP.

During my experiment to encode some XML instances captured from my tcpdump to 
EXI, I have got unique particle attribution (UPA) problem as described in [2].
Although EXI works with schemaless XML, it's far better to use schema-informed 
EXI in terms of efficiency and validation. Schema-informed EXI is for valid 
XML, but current spec does not allow validation due to UPA. Changing local 
schema (suggested by Peter in [2]) breaks interoperability of schema-informed 
EXI.

Is there any possible way to make a 'valid XMPP spec' with some 
(backword-compatible) specification update? I guess there are three choices.

1) change XMPP schema/spec to avoid UPA. This may break inteoperability.
2) use different XMPP schema for EXI communications. This looks awkward.
3) use XSD 1.1 for weak wildcard. I'm not sure XMPP community wants/hates 
'cool' features of XSD 1.1.

[1] http://www.w3.org/TR/exi/
[2] http://mail.jabber.org/pipermail/jdev/2012-June/089069.html

Regards,

// Yusuke DOI mailto:yusuke@toshiba.co.jp>> Corporate 
R&D Center, TOSHIBA Corp.



--
Hälsningar

Joachim Lindborg
Teknisk Chef

Sustainable Innovation AB
Adress: Box 55998 102 16 Stockholm
Besöksadress: Storgatan 31 (Malmgården)
Email: joachim.lindb...@sust.se<mailto:joachim.lindb...@sust.se>, 
www.sust.se<http://www.sust.se>
Tel +46 706-442270

[Standards] Question on valid XMPP

[Yusuke DOI]

Hi,

My name is Yusuke Doi. I'm wondering if it is possible to put EXI[1] with XMPP.

During my experiment to encode some XML instances captured from my tcpdump to 
EXI, I have got unique particle attribution (UPA) problem as described in [2]. 
Although EXI works with schemaless XML, it's far better to use schema-informed 
EXI in terms of efficiency and validation. Schema-informed EXI is for valid 
XML, but current spec does not allow validation due to UPA. Changing local 
schema (suggested by Peter in [2]) breaks interoperability of schema-informed 
EXI. 

Is there any possible way to make a 'valid XMPP spec' with some 
(backword-compatible) specification update? I guess there are three choices.

1) change XMPP schema/spec to avoid UPA. This may break inteoperability.
2) use different XMPP schema for EXI communications. This looks awkward.
3) use XSD 1.1 for weak wildcard. I'm not sure XMPP community wants/hates 
'cool' features of XSD 1.1.

[1] http://www.w3.org/TR/exi/
[2] http://mail.jabber.org/pipermail/jdev/2012-June/089069.html

Regards,

// Yusuke DOI  Corporate R&D Center, TOSHIBA Corp.