Re: [Standards] XEP-0313: why it is *really* not a good idea to use MAM with Pubsub
Hello, I think it is a good idea for there to be a search extension for pubsub. One thing to keep in mind would be is that the extension could become really complicated depending on the search fields that you are going to have and the type of filter you want. If there are user defined fields, then special care should be had not to make the stanza cumbersome. An idea you can implement for instance, to make it more compact, is to use fields defined in other namespaces, so in essence you'd be searching a particular XEP namespace (for exampleurn:xmpp:jingle:apps:file-transfer:4) and you could reuse to fields instead of redefining then. Of course you would still need to have user defined fields as I don't think that searching namespaces would handle all use cases. Good luck, For reference you can have a look at: XEP-055 On 06/01/16 07:08, Goffi wrote: G'day, MAM is a great tool which solves several problems for messages management. It also offers the ability to get items from a PubSub node when the "node" attribute is used. We have implemented this feature in our PubSub/PEP component, and I haven't seen any other implementation for PubSub so far (if you know any, please tell me). But I have to say that MAM is really badly adapted to PubSub, here are the major reasons: - All items a returned in separate stanza, wrapped in a element, one item per stanza. This both is a waste of bandwidth and make the task more difficult for the client as it must track each and the result to known when a page has been received. A simple query like for a PubSub items retrieval would be much more better. - Requests are made on one node. But it is desirable to be able to do requests on several nodes, or on nodes which match a pattern. For instance, in XEP-0277 comments node are in the form "urn:xmpp:microblog:0:comments/dd88c9bc58886fce0049ed050df0c5f2" and it would be usefull to request all items from a node starting with "urn:xmpp:microblog:0:comments". With MAM I can't request all comments published by Romeo. - this one could be easily fixed, but currently we can't do filtering on PEP without requesting a particular jid. With microblog, we want to be able to request e.g. all items with the category/tag "XMPP" regardless of the author. - There is no way when a service offer MAM both for message and PubSub (e.g.: a MUC component with PubSub abilities (MUC 2 ?), or the server itself when it offers PEP) to know if the filtering fields apply to messages, or PubSub, or both. Look at section 4.1.5 "Retrieving form fields", how can I know if "urn:example:xmpp:free-text-search" can be used for PubSub or not? - section 4.2 says that "The archive results MUST be sorted in chronological order", that totally make sense for message archives, but in the case of PubSub this is incoherent with the classic items retrieval ordering (most recent item first), and we may want to sort on other fields than publication date: for instance item updating date vs publishing date, or size of files tracked with pubsub. Of course we can reverse order easily with RSM, but though it's not natural, and we can't sort on other fields. - overall, PubSub already manages archives by design, but it is lacking a good searching tool. Even if it is tempting to use MAM with PubSub because we can have filtering "for free", I really think it is not adapted at all, and PubSub deserve a real dedicated searching/filtering tool. If other people are interested, I would like to work on a "PubSub searching" protoXEP. PubSub will probably be the core of many major features in XMPP in the future, so we need a good, generic, and extendable way to search/filter items. Regards Goffi ___ Standards mailing list Info: http://mail.jabber.org/mailman/listinfo/standards Unsubscribe: standards-unsubscr...@xmpp.org ___ ___ Standards mailing list Info: http://mail.jabber.org/mailman/listinfo/standards Unsubscribe: standards-unsubscr...@xmpp.org ___
Re: [Standards] XEP-0234 Jingle File Transfer 0.16
I vote for not moving towards supporting :4. There isn't a compelling reason to do it. So it is better to wait until (or if) there features are supported in the next version or in some other XEP. On 20/06/15 16:13, Daniel Gultsch wrote: 2015-06-20 21:59 GMT+02:00 Yann Leboulanger aste...@lagaule.org mailto:aste...@lagaule.org: That will also break compatibility with clients that supports only :3 namespace, but currently we don't work with clients that have :4 namespace implemented. implementing several version of a protocol is quite complexe. Thats basically the reason I'm asking. My client Conversations currently only supports :3 and I'm not going to support both however I would upgrade to :4 if gajim were to do so as well and thus making Conversations compatible with Swift. (Gajim, Conversations and Swift - to my knowledge - are the only clients supporting Jingle FT) cheers Daniel
Re: [Standards] wiki for xeps
Just as an experiment to see how the procedure could be improved. Of course, xsf could implement it if you are inclined to do it. Although, you'd have to be aware of the costs of doing that. On 30/03/15 16:55, Kevin Smith wrote: On 29 Mar 2015, at 02:36, Jefry Lagrange jefry.re...@gmail.com wrote: Hello everyone, I found myself thinking about the process of proposing and editing xeps that the xsf implements. I think there are some improvements that could be made in order to make the process easier. These are the main areas which I think are problematic with the current way of doing things: • git has a learning curve. • xml has a learning curve. • scripts are necessary to convert the xml to readable documents. • Everything is tracked in the same project (scripts, templates, xeps). XEPs aren't tracked as individual projects. These problems aren't that grave, after all, most of us are developers. However, I think it is worthwhile to explore alternatives to make it easier. I spent the past week working on a wiki page that closely resembles the look and feel of xeps. Here is the url: http://jlagrange.net/wiki/index.php/Category:Extensions In that week, I learned that by developing your own mediawiki skins and extensions, you are able to make a wiki look however you like. Because wikis were made specifically to track and modify documents, they have some features could be useful for us. • view and compare history of changes online • user management to allow authors to modify their xeps and nothing more. Or to allow trusted users to help Peter. • rss of changes • easier to edit and see the results without using scripts to convert it to html • discussion pages for each xep At the very least it could serve a way to make quick prototypes and get feedback. Hi Jefry, What’s your expected response to this? That is - are you posting it as a FYI thing for folks to look at, or are you proposing this as an update to XSF procedures? /K
Re: [Standards] wiki for xeps
On 28/03/15 23:16, Mathieu Pasquet wrote: On Sat, Mar 28, 2015 at 09:36:50PM -0400, Jefry Lagrange wrote: Hello everyone, I found myself thinking about the process of proposing and editing xeps that the xsf implements. I think there are some improvements that could be made in order to make the process easier. These are the main areas which I think are problematic with the current way of doing things: * git has a learning curve. * xml has a learning curve. * scripts are necessary to convert the xml to readable documents. * Everything is tracked in the same project (scripts, templates, xeps). XEPs aren't tracked as individual projects. These problems aren't that grave, after all, most of us are developers. However, I think it is worthwhile to explore alternatives to make it easier. I spent the past week working on a wiki page that closely resembles the look and feel of xeps. Here is the url: http://jlagrange.net/wiki/index.php/Category:Extensions In that week, I learned that by developing your own mediawiki skins and extensions, you are able to make a wiki look however you like. Because wikis were made specifically to track and modify documents, they have some features could be useful for us. * view and compare history of changes online * user management to allow authors to modify their xeps and nothing more. Or to allow trusted users to help Peter. * rss of changes * easier to edit and see the results without using scripts to convert it to html * discussion pages for each xep At the very least it could serve a way to make quick prototypes and get feedback. Hello Jefry, Making the XEP creation/edition process more accessible is a worthwhile endeavour in my opinion. However, while the wiki format does have its uses and advantages, I am not sure it would be a great fit for XEPs. I want to address several points: - git has a learning curve: yes, but git is not required at all for proposing and editing xeps, you only need to clone the repo and everything afterwards can be done without git. - XML has a learning curve, but I would argue that people submitting XMPP extensions are already past the required level ;). More to the point, the only XML required while writing a XEP is some bare-bones XHTML, and writing the metadata. - Everything is tracked in the same project. Yes, I agree, but the consensus seems to be that this isn’t an issue, as there aren’t too many changes happening, and maintaining a separate history for each XEP would be only more maintenance overhead, currently. - XEPs already have online diffs What I find lacking, or counter-intuitive, in a wiki-powered solution: - The very ability to get all XEP sources in plain text, like we currently can. This is possible either by extensions or by using third party converters. I think this is actually a common use case for a lot of mediawiki users. - Reviewing changes: wikis have this feature, but this would require to create several pages per XEP, in order to have the author edit a copy of the extension, then having people review them, and finally editors can move the copy to the “final” page. This is important for the publishing process, because non-editors shouldn’t be able to edit published documents. Actually I think you could configure mediawiki to have a queue of changes where the changes have to be approved by someone i.e. the editor. - Mandatory web interface (this is the biggest issue, for me), both for reviews, discussions, and submissions That's a feature not a bug =D. What I could appreciate in a wiki solution: - RSS/Atom feeds of changes - Centralized discussions: the mailing-list is nice to have exchanges, but after some time it becomes gruesome to aggregate all that was said on a particular extension. - Possible web-based edition for quick edits to the documents, with instant visual feedback. Ultimately I agree with you. This is not needed as the current system is working fine. The time investment required to tidy up things with the wiki to make it transparent to the user, to use wiki templates instead of the xml ones, to fix formatting errors with inner links and tables, would be costly. But now that we have the discussion going. Those points that you mentioned are interesting (RSS/Atom would be amazing). (Please don’t think that I’m overly negative, I’m just writing down some quick thoughts) Not at all. I'm not invested in this. This is the result of me fooling around for a week.
[Standards] wiki for xeps
Hello everyone, I found myself thinking about the process of proposing and editing xeps that the xsf implements. I think there are some improvements that could be made in order to make the process easier. These are the main areas which I think are problematic with the current way of doing things: * git has a learning curve. * xml has a learning curve. * scripts are necessary to convert the xml to readable documents. * Everything is tracked in the same project (scripts, templates, xeps). XEPs aren't tracked as individual projects. These problems aren't that grave, after all, most of us are developers. However, I think it is worthwhile to explore alternatives to make it easier. I spent the past week working on a wiki page that closely resembles the look and feel of xeps. Here is the url: http://jlagrange.net/wiki/index.php/Category:Extensions In that week, I learned that by developing your own mediawiki skins and extensions, you are able to make a wiki look however you like. Because wikis were made specifically to track and modify documents, they have some features could be useful for us. * view and compare history of changes online * user management to allow authors to modify their xeps and nothing more. Or to allow trusted users to help Peter. * rss of changes * easier to edit and see the results without using scripts to convert it to html * discussion pages for each xep At the very least it could serve a way to make quick prototypes and get feedback. -- Jefry Lagrange
Re: [Standards] Proposed XMPP Extension: File Information Sharing
On 11/28/2012 05:00 AM, Sergey Dobrov wrote: I still think that reusing of service discovery to discover shared files as it was in the XEP-135 is better. jabber:iq:search can be reused there too. Well, like I said, I've been trying to do this extension for a year now. In that time there have been plenty of drafts that use XEP-135. Every draft I made have tradeoffs, so there is no perfect solution. I really don't mind what shape the extension has, as long as the functionality remains the same. If this one gets rejected, I will make another one using XEP-135. Also, it seems that there is a mistake in the regex here: validate xmlns='http://jabber.org/protocol/xdata-validate' datatype='xs:string' regex*.png/regex /validate Yea, I used glob because I didn't have time to learn the one that the xep uses. Do could you provide me with the right regex? On 11/28/2012 12:57 AM, XMPP Extensions Editor wrote: The XMPP Extensions Editor has received a proposal for a new XEP. Title: File Information Sharing Abstract: This document specifies a simple extension to existing protocols that allows an entity to request information about files. URL: http://xmpp.org/extensions/inbox/fis.html The XMPP Council will decide in the next two weeks whether to accept this proposal as an official XEP.
Re: [Standards] File sharing xep proposal
This is certainly easier. If this extension gets rejected, I will definitely try to use xep-0135 instead. On 11/12/2012 04:14 AM, Sergey Dobrov wrote: So I suggest to keep XEP-0135 as is and just add ability to retrieve files via Jingle and to perform a search through ad-hoc commands. On 11/09/2012 03:26 PM, Sergey Dobrov wrote: On 11/09/2012 03:17 AM, Matthew Wild wrote: On 8 November 2012 18:41, Sergey Dobrov bin...@jrudevels.org wrote: On 11/09/2012 12:51 AM, Jefry Lagrange wrote: I have tried many different ways of making the protocol. The first time around, I modified xep0135 to include search capabilities. Someone told me, I think it was Matthew, that xep0135 abuses service discovery by using it too much. Also another thing that I noticed is that, data-forms make the xep0135 stanzas become really big, as they waste too much space. I actually don't understand hot is it possible to abuse service discovery. Maybe someone can explain? I believe the thread Jefry references is this one: http://mail.jabber.org/pipermail/standards/2011-November/025456.html ah, it seems that Jefrey was confused with the overload and abuse terms. So I still think that it would be better to keep files directories in the service discovery, just we need another way to do search queries. I believe that the same thing Matthew actually said in those thread. Regards, Matthew
Re: [Standards] File sharing xep proposal
Hi On 11/08/2012 03:20 AM, Sergey Dobrov wrote: Hello Jefry, Thank you for the HTML version you provided. That wasn't me, thank Matthew for converting it and hosting it in his website. The problem is very actual so thank you for your interest too. The first thing is that I think that the XEP-0135 (http://xmpp.org/extensions/xep-0135.html) way of browsing files is nicer because it reuses service discovery in very natural way. :) But the search compatibilities you provided in your XEP are nice too, but I think that queries should be improved though. I have tried many different ways of making the protocol. The first time around, I modified xep0135 to include search capabilities. Someone told me, I think it was Matthew, that xep0135 abuses service discovery by using it too much. Also another thing that I noticed is that, data-forms make the xep0135 stanzas become really big, as they waste too much space. For those reasons, I decided to move away from xep0135 and instead of using dataforms, use the well known file stanzas as defined in xep096 and xep0234. In what way do you think that the queries could be improved? The other thing I noticed is that you saying the the client MUST respond with the list of files if it was queried to do so. I don't think it's right because there are many conditions when client might decide not to do so: permissions control, bandwith control, etc. Maybe it wasn't clear in the xep, I will check the writing. Basically if you don't have or want to share any files you just reply with a: match offer directory/ /offer /match Thanks for your feedback. On 11/06/2012 08:19 AM, Jefry Lagrange wrote: Hello, I've been working for quite some time in a protocol extension to allow file sharing. I have a (some-what) working prototype of its implementation. The prototype is available at Gajim's plugin repository: http://hg.gajim.org/gajim-plugins It doesn't do a lot, but it can share files between computers. Although the code is in alpha stage everyone who is brave enough is welcomed to try it. I have attached a draft of the protocol extension. I would like some feedback on it. If it doesn't looks too bad, I would like to present it to the council by the end of the week. Cheers,
Re: [Standards] File sharing xep proposal
Sorry, I don't know how to do that. Apparently there is a script for it gen.py, but I don't know how to use it and it finds some syntax errors that I don't know how to fix. On 11/07/2012 02:59 AM, Sergey Dobrov wrote: Could you attach rendered html version please? On 11/06/2012 08:19 AM, Jefry Lagrange wrote: Hello, I've been working for quite some time in a protocol extension to allow file sharing. I have a (some-what) working prototype of its implementation. The prototype is available at Gajim's plugin repository: http://hg.gajim.org/gajim-plugins It doesn't do a lot, but it can share files between computers. Although the code is in alpha stage everyone who is brave enough is welcomed to try it. I have attached a draft of the protocol extension. I would like some feedback on it. If it doesn't looks too bad, I would like to present it to the council by the end of the week. Cheers,
Re: [Standards] File sharing xep proposal
Hello Matthew, thanks for the feedback. On 11/05/2012 11:06 PM, Matthew Wild wrote: Hi Jefry, On 6 November 2012 01:19, Jefry Lagrangejefry.re...@gmail.com wrote: Hello, I've been working for quite some time in a protocol extension to allow file sharing. I have a (some-what) working prototype of its implementation. Great to hear! I can think of a million-and-one uses for such a protocol, and I'm sure I'm not alone. The prototype is available at Gajim's plugin repository: http://hg.gajim.org/gajim-plugins and running code is always good :) I read very quickly through (it's 4am after all...), and it seems pretty sensible. Any comments I have are minor, and I think it's perfectly fine to submit it as it is. A few of my comments... It seems like it might be possible to make the directory listing XML somewhat more compact than it currently is. For example just listing files with no metadata could simply be: file name=foo.txt/ file name=bar.txt/ which seems a lot simpler than the nested XML elements in the example. Metadata can be child elements offile/ because although every file has a name, metadata may vary and will almost certainly want to be extended in some cases. Yeah, about that... I wanted to reuse current extensions whenever possible. This format for file is the same in xep096 and xep234. I see what you are saying, it is indeed more compact and saving bandwidth in a situation where big stanzas are sent, is very important. At first I considered the dataforms as they are used in xep0135, but they were too big to be used efficiently. I definitely favor making this as compact as possible. I'm not sure how necessary it is to require the leading '/' prefix, it seems natural to omit it if I just have a flat list of files. And then 'foo/bar.txt' is perfectly fine when I don't, so it seems like it could be dropped without much problem. Does it look weird? The only reason why I made it a requirement to use a leading '/' was to make it possible to distinguish between asking for more information on a file or directory, and searching for a file. When searching files there should not be a / prefix. I don't know of any other way of doing this. Dataforms would be very good for searching, since they support regular expressions. However, as I said before, they can get pretty big. I would like to listen to some ideas about how search could be accomplished in a nice way. The security note about access control being implemented on servers doesn't make complete sense. Files are shared by a specific resource, so each resource can enforce its own access controls... and different resources won't necessarily be sharing the same sets of files. A logged in client generally already has the roster, which is probably the main access control advantage that the server could offer anyway. Ah... I didn't make it clear. What I was thinking about was not about individual file permissions, it was about preventing an user to see any files at all. Blocking an user from file sharing, would require agreement across the different resources so they can agree about no sharing files with that particular user. I know realize that this is more of an implementation issue than a security issue, I should move it to implementation notes, because it is up to the implementers if they want to add a feature to block an user from all file sharing. Anyway, good work... and I look forward to it being approved soon, and perhaps implementing it myself :) Regards, Matthew Thanks.
[Standards] File sharing xep proposal
Hello, I've been working for quite some time in a protocol extension to allow file sharing. I have a (some-what) working prototype of its implementation. The prototype is available at Gajim's plugin repository: http://hg.gajim.org/gajim-plugins It doesn't do a lot, but it can share files between computers. Although the code is in alpha stage everyone who is brave enough is welcomed to try it. I have attached a draft of the protocol extension. I would like some feedback on it. If it doesn't looks too bad, I would like to present it to the council by the end of the week. Cheers, ?xml version='1.0' encoding='UTF-8'? !DOCTYPE xep SYSTEM 'xep.dtd' [ !ENTITY % ents SYSTEM 'xep.ent' %ents; ] ?xml-stylesheet type='text/xsl' href='xep.xsl'? xep header titleFile Information Sharing/title abstractThis document specifies a simple extension to existing protocols that allows an entity to request information about files./abstract LEGALNOTICE; number/number statusProtoXEP/status typeStandards Track/type sigStandards/sig approverCouncil/approver dependencies specXMPP Core/spec specXEP-0234/spec specXEP-0059/spec specXEP-0265/spec /dependencies supersedes specXEP-0135/spec /supersedes shortnamefis/shortname author firstnameJefry/firstname surnameLagrange/surname emailjefry.re...@gmail.com/email jidj.lagra...@jabber.org/jid jidj.lagra...@gajim.org/jid /author revision version0.0.1/version date2012-10-14/date initialsjl/initials remarkFirst draft/remark /revision /header section1 topic='Introduction' anchor='intro' pXMPP extensions provide ways of transferring files between peers (such as xep0234; and xep0096;). However, file transfer is currently limited to needing that the transfer be initiated by the hosting entity. The xep0234; extension, provides for a way to request files, but it requires the requesting entity to have information about the file being requested, so that it can be uniquely identified. /p pThis documents defines an extension which allows the request of information of files being offered by a hosting entity so they can later be requested in a file transfer; If the requesting entity is interested in the file./p pIRC users have been able to bypass the limitations of the protocol by using bots that provide information of files and initiate transfer on command. A major downside of this method is that not every user is capable of sharing its files. The aim of this document is to provide a similar functionality while making it easier for users to offer and request information about files. /p pMicrosoft's MSN proprietary IM client, used to provide similar functionality using Sharing Folders, but this was replaced by Windows Live SkyDrive /p /section1 section1 topic='Requirements' anchor='reqs' ol liEnable a requesting entity to traverse the shared directory of an offering entity (REQUIRED)/li liEnable a requesting entity to get detailed information about files. (REQUIRED)/li liEnable a requesting entity to search files hosted by an offering entity.(OPTIONAL)/li liAllow the use of MUCs to share information about files among the users.(OPTIONAL)/li !--liAllow the use of Pub/Sub service to search files (OPTIONAL)/li -- /ol /section1 section1 topic='Assumptions' anchor='assump' !-- Maybe make this into a list -- p This protocol assumes the existence of a shared directory (either virtual or physical), whose root can be accessed by /. The hosting entity must not advertise empty directories. The hosting entity is responsible of maintaining the structure of that directory (such as not allowing two files with the same name and preventing cycles within directories). The hosting entity is in no way required to present the same shared directory to different requesters./p /section1 section1 topic='Getting Information About Files ' anchor='disco' section2 topic='Traversing Files' anchor='traver' pIf a requesting entity wishes to traverse the shared folder of an offering entity. It can do so by querying the root directory as it is shown in the following example:/p example caption='Requester queries the root of the shared folder' ![CDATA[ iq type='get' from='jul...@capulet.com/chamber' to='ro...@montague.net/home' id='1234' match request directory/ /request /match /iq ]] /example p If the offering entity has files to share, it MUST respond with the root-level files of its shared folder. Files and directories at the root level MUST not be the child of any directory tag. In order to save bandwidth, the offering entity MAY omit all the children of the file tag except the name which is required and MUST always be present. /p pThe file tag has the same attributes as defined in xep0096;. The name attribute is required and must be included in every response. The size attribute is only required when
Re: [Standards] LAST CALL: XEP-0297 (Stanza Forwarding)
I think is specification is needed. It serves a purpose and it might be used by other xeps to propagate message (or request) through a network of friends. I don't think the use case with message is enough. It would be more clear if it had an use case with an IQ. It is not clear how one should respond to a forwarded IQ. Another question would be: Is it possible to bypass the middle man, once you get the forwarded stanza (in case you need to reply)? On Wed, Aug 22, 2012 at 7:04 PM, XMPP Extensions Editor edi...@xmpp.org wrote: This message constitutes notice of a Last Call for comments on XEP-0297 (Stanza Forwarding). Abstract: This document defines a protocol to forward a stanza from one entity to another. URL: http://xmpp.org/extensions/xep-0297.html This Last Call begins today and shall end at the close of business on 2012-09-07. Please consider the following questions during this Last Call and send your feedback to the standards@xmpp.org discussion list: 1. Is this specification needed to fill gaps in the XMPP protocol stack or to clarify an existing protocol? 2. Does the specification solve the problem stated in the introduction and requirements? 3. Do you plan to implement this specification in your code? If not, why not? 4. Do you have any security concerns related to this specification? 5. Is the specification accurate and clearly written? Your feedback is appreciated! -- Jefry Lagrange
Re: [Standards] File hosting XEP?
Sure, but how client will upload it's attachments? I can do it in a client-specific way, but then others will not be able to implement the same thing. If I understand correctly, what you want to do is to attach files to a microblog post. Since you can use Out of Band Data for that, you can upload the file out of band without the need of xmpp e.g. uploading it to an FTP. True, the client has to be modified to support this, but since it isn't covered in any XEP afaik, you would have to modify it anyway. Therefore, your client will be able to upload files while other clients will not until their devs deem it a necessary feature. On Tue, Aug 14, 2012 at 9:04 AM, Sergey Dobrov bin...@jrudevels.org wrote: On 08/14/2012 07:47 PM, Todd Herman wrote: -Original Message- From: standards-boun...@xmpp.org [mailto:standards-boun...@xmpp.org] On Behalf Of Sergey Dobrov Sent: Tuesday, August 14, 2012 4:44 AM To: standards@xmpp.org Subject: [Standards] File hosting XEP? Hello all, hope you are all good. Me and Jaussoin Timothée faced with a problem to attach files to microblog posts. The easier way to do that is to serve files somewhere and link to them from the posts. But what is the appropriate way to do that? Having files somewhere and linking to them sounds a lot like Out-Of-Band Data (XEP-0066). It covers the linking part (mainly used in SI File Transfers and Message stanzas) and leaves the transferring of the files and how they are stored on the server to you. Sure, but how client will upload it's attachments? I can do it in a client-specific way, but then others will not be able to implement the same thing. -- With best regards, Sergey Dobrov, XMPP Developer and JRuDevels.org founder. -- Jefry Lagrange
Re: [Standards] [XEP-0234] File checking in JingleFT
I was thinking that maybe we propose a separate XEP for file checking. Still, some changes should have to be done to JingleFT such as allowing range transfer reuse the session and add a limit attribute in the range file transfer. So that it can send chunks of data (with offset and limit). On Thu, Dec 8, 2011 at 3:10 PM, Waqas Hussain waqa...@gmail.com wrote: On Thu, Dec 8, 2011 at 6:48 AM, Jefry Lagrange jefry.re...@gmail.com wrote: So far what I've read in the XEP-0234 doesn't say anything about file checking. I suppose, it is implied that once the comparing of the hashes takes place, the program has to inform the user of a possible file corruption and then restart the file transfer manually. This is very inefficient, and awkward (for the user). Instead of resending the whole file, we should be able to detect in which part of the file there is corruption and resend that part using ranged transfers (as defined in http://xmpp.org/extensions/xep-0234.html#range). There are various issues with this. In the current implementation of jingleFT, range file transfer have to initiate a new session in order to transfer the file from a given offset. Ideally, we would want to use the current session. Example: * initiator sends file * responder receives it * initiator communicates the hash * responder checks, and notices that the file is corrupt. Finds where the corruption took place (using techniques explained later) * responder ask for ranged file transfer over the same session (note: we haven't terminated the session). * initiator resends the requested part of the file I'm not sure that only providing an offset for as a range can accomplish this, it would be probably be a better idea to provide a range as a range. I might be wrong about this, I'm not sure. Now, the question is, how do we know which part of the file is corrupted? One way to go about this is using hash-lists. Which is the way bittorrent makes sure there isn't corruption in the files being transfered. Basically, what we do with this is that we send the file in parts. For each part the initiator will send an IQ with the hash of that file part. The responder will respond with a result IQ or an error. In case there is an error from part of the responder, the initiator will resend that part. Easy right? The problem with this algorithm is that it is too expensive. It makes sense using it in p2p programs, because a file may be share to other peers before it is completed. It doesn't make very much sense using this in XMPP, where there is usually just one sender and one receiver. But this technique is very illustrative and easy to understand. Another alternative technique that we can use is binary-hashing (I made the name up, I don't know what this is called). Using this technique we will wait until the file is transfered as usual, then when we check the hashes if there is a mismatch we will splice the file in two parts, take the hash of both parts and ask the initiator for confirmation on those hashes, if one mismatches, that should tell us that the corrupt part is, and we continue repeating this process until the parts are small enough for them to be reasonable resent. If the file is small enough to being with, then we can skip this. The cost (in terms of computation) of this technique might be greater than hash-list. But the overall cost is many times smaller than hash-list, because we only need to use this if we find that the file is corrupt, which is an unlikely event. I would love to know what you guys think of this. -- Jefry Lagrange I have always been interested in ranged transfers, and verification. I'm not sure how much interest there is in general XMPP community though. The way bittorrent and other file sharing protocols have been doing this for ages is quite interesting, and efficient. Here's is the specification of data contained in a bittorrent file: http://www.bittorrent.org/beps/bep_0003.html#metainfo-files-are-bencoded-dictionaries-with-the-following-keys And what you called 'binary-hashing' are Merkel trees, or Hash trees. http://en.wikipedia.org/wiki/Hash_tree A bittorrent extension for merkel trees was defined two years ago: http://www.bittorrent.org/beps/bep_0030.html In bittorrent, the size of a block of data is defined in the torrent file, specific to that torrent. A common size is 512KB. For a 1GB transfer, this would be 2048 blocks of 512KB each. Given SHA-1 as hash, this would be 20*2048 = 40KB of hashes. 53.3KB with base-64 encoding. Your suggested hash tree may be twice that size, at 106.67KB, or it can be just 53.3KB by keeping hashes in all levels of the tree, not just leaves. See how BEP-0030 does it. A 1TB file though would have 53.3MB of hashes.. and just think how many IQs it would take if you are requesting subsection hashes, and how much CPU and I/O time it would take if you are calculating hashes of parts independently
Re: [Standards] UPDATED: XEP-0300 (Use of Cryptographic Hash Functions in XMPP)
Well, I think the problem that xep-300 pretends to fix, can easily be fixed by changing the XEPs that make use of hashes. If a XEP is using a hash algorithm that has been deprecated or is insecure, that XEP should be updated to mandate the discontinuation of such algorithm in clients. Hash algorithms don't deprecated as often, and using XEP-0300 would lead to a slower adoption of newer hash algo. For example: Client A sends file to Client B Client B is updated to use the most secure hash available Client A has a deprecated version of the hash Client B negotiates using XEP-300 and ends up using the less secure hash File transfer is successful This case would lead to a slower adoption of more secure technology. A negotiation makes sense in a lot of cases, for IBB or Socks5. But negotiating for hashes makes no sense because the implementation is as easy as changing a couple of lines of code. That's why it should be mandated to use the most secure and updated algorithm, and as I said, hashes don't deprecate frequently so that approach would be less inconvenient than xep-300. Now, I understand that there might be some ambiguity on which hashes are being use. I personally think that the attribute hash should never be used, instead md5 or sha1 should be used. P.S: I realized half way through that XEP-0300 is not negotiating, but instead is discovering support of hashes of the clients. Still, my point stands. It doesn't change anything. On Mon, Dec 5, 2011 at 5:32 PM, Peter Saint-Andre stpe...@stpeter.im wrote: On 12/5/11 3:16 PM, XMPP Extensions Editor wrote: Version 0.2 of XEP-0300 (Use of Cryptographic Hash Functions in XMPP) has been released. Abstract: This document provides recommendations for the use of cryptographic hash functions in XMPP protocol extensions. Changelog: Updated to reflect initial analysis of existing XMPP protocol extensions. (psa) Diff: http://xmpp.org/extensions/diff/api/xep/0300/diff/0.1/vs/0.2 URL: http://xmpp.org/extensions/xep-0300.html Folks, I started to look at XEP-0300 in relation to existing extensions. Please review my work so far, and do your own thinking about how useful (or not useful) XEP-0300 is. Peter -- Peter Saint-Andre https://stpeter.im/ -- Jefry Lagrange
Re: [Standards] Proposed changes to XEP-0135
Thanks for your response. ;-) I'm not sure it's a good idea to overload the disco request this way. Perhaps having the filter in a separate namespace would be ok, but I think perhaps it would be better to use another protocol for this. Yeah, I wasn't sure about it either. But this way is at least consistent with what the XEP says about asking for file information. The XEP currently uses disco requests extensively in section 4 and 5. If disco overuse is a problem, wouldn't it be better to stop using it in those sections as well? Alternatively we could use a new stanza, like match /match, with a new namespace. But it wouldn't be consistent. We would want to keep requesting file list and requesting file information, within the same namespace as finding specific files. The filter I propose has no new elements. Everything is reused from XEP-122. A new extension for it would be small, and it wouldn't contain new information. Cheers, On Mon, Nov 28, 2011 at 9:41 AM, Matthew Wild mwi...@gmail.com wrote: On 27 November 2011 03:50, Jefry Lagrange jefry.re...@gmail.com wrote: Hi, I been working on some changes to XEP-0135. * Replacing SI file transfer with jingle FT Good :) * Replacing section 6, with a link pointing to section 5 of XEP-0234, which already covers the same function. Makes sense. * Adding support for pubsub, only for finding files using the method I introduce bellow. It doesn't make much sense to traverse the directory of every user subscribed to a pubsub, but it will make a lot of sense searching for specific files. (XEP-0137 does not suffice for this) Agreed. 5.5 Finding Specific Files Finding files by asking for a file list is not very practical if there are too many files being shared. It is very resource intensive and it is understood that the user may not be interested in all of the files, but rather he or she would be interested in finding one specific file or one specific kind of file (text, image or videos). In order to do this, the identity stanza is used to match files by one or more fields i.e. 'name', 'date', 'size', etc... Example XX. Finding Specific Files iq type='get' from='ha...@shakespeare.lit/pda' to='darkc...@shakespeare.lit' id='find45' query xmlns='http://jabber.org/protocol/disco#info' node='files' identity category='filesys' type='file' name='file1' / /query /iq I'm not sure it's a good idea to overload the disco request this way. Perhaps having the filter in a separate namespace would be ok, but I think perhaps it would be better to use another protocol for this. Example XX. Finding files using Regular Expressions Another useful feature, but I even more strongly feel this shouldn't be done over disco. Any feedback would be greatly appreciated, I just want to know if I am on the right track here. Absolutely, I'd love to see this spec revived. I look forward to seeing a new XEP draft :) Regards, Matthew -- Jefry Lagrange
[Standards] Proposed changes to XEP-0135
Hi, I been working on some changes to XEP-0135.
* Replacing SI file transfer with jingle FT
* Replacing section 6, with a link pointing to section 5 of XEP-0234,
which already covers the same function.
* Adding support for pubsub, only for finding files using the method I
introduce bellow. It doesn't make much sense to traverse the directory
of every user subscribed to a pubsub, but it will make a lot of sense
searching for specific files. (XEP-0137 does not suffice for this)
For example:
A user is subscribed to the books pubsub channel. It sends a query,
looking for a book Romeo and Juliet - By Shakespeare. The
subscribers reply if they get a match with information about their
files. The initiator requests the file from whoever has what he wants
and file transfer starts.
* A new section should be added to cover finding files by providing a
criteria, instead of just asking for all the files.
For example:
5.5 Finding Specific Files
Finding files by asking for a file list is not very practical if there
are too many files being shared. It is very resource intensive and it
is understood that the user may not be interested in all of the files,
but rather he or she would be interested in finding one specific file
or one specific kind of file (text, image or videos).
In order to do this, the identity stanza is used to match files by one
or more fields i.e. 'name', 'date', 'size', etc...
Example XX. Finding Specific Files
iq type='get'
from='ha...@shakespeare.lit/pda'
to='darkc...@shakespeare.lit'
id='find45'
query xmlns='http://jabber.org/protocol/disco#info'
node='files'
identity category='filesys' type='file' name='file1' /
/query
/iq
The fields in the identity stanza, are optional, but at least one
field MUST be provided. In this example, the responders will match its
files looking for the file names that contain 'file1' and are of the
size 1024..
Example XX. Returning with Matched Files
iq type='result'
from='darkc...@macbeth.shakespeare.lit'
to='ha...@shakespeare.lit/pda'
id='find45'
query xmlns='http://jabber.org/protocol/disco#info'
node='files/somefile'
identity category='filesys' type='file' name='file1'
hash='552da749930852c69ae5d2141d3766b1'/
/query
/iq
A responding entity MUST include the name and the hash of the file.
Since more than one responder may respond with the same file, it is
strongly suggested that the initiator makes use of ranged file
transfers (as defined in XEP-0234), to speed up the file transfer.
Example XX. Finding files using Regular Expressions
Regular expressions may be use to find files that match the
expression. One or more fields can be used. The label attribute is
optional.
iq type='get'
from='ha...@shakespeare.lit/pda'
to='darkc...@shakespeare.lit'
id='find46'
query xmlns='http://jabber.org/protocol/disco#info'
node='files'
identity category='filesys' type='file' name='file1'/
x xmlns='jabber:x:data' type='get'
field var='ssn' type='text-single' label='Social Security Number'
regex([0-9]{3})-([0-9]{2})-([0-9]{4})/regex
/field
/query
/iq
The XML character data of this element is the pattern to apply. The
syntax of this content MUST be that defined for POSIX extended regular
expressions, including support for Unicode.
The regex/ element MUST contain character data only (i.e., not
contain any child elements) and MUST NOT possess any attributes.
Any feedback would be greatly appreciated, I just want to know if I am
on the right track here.
--
Jefry Lagrange
Re: [Standards] XEP-198 when to start counting?
Thanks, I can't believe I missed that. It is clear now. On Fri, Jun 17, 2011 at 2:00 PM, standards-requ...@xmpp.org wrote: Send Standards mailing list submissions to standards@xmpp.org To subscribe or unsubscribe via the World Wide Web, visit http://mail.jabber.org/mailman/listinfo/standards or, via email, send a message with subject or body 'help' to standards-requ...@xmpp.org You can reach the person managing the list at standards-ow...@xmpp.org When replying, please edit your Subject line so it is more specific than Re: Contents of Standards digest... Today's Topics: 1. XEP-198 when to start counting? (Jefry Lagrange) 2. Re: XEP-198 when to start counting? (Matthew Wild) 3. Re: XEP-198 when to start counting? (Peter Saint-Andre) 4. Re: XEP-198 when to start counting? (Matthew Wild) 5. Re: XEP-198 when to start counting? (Peter Saint-Andre) -- Message: 1 Date: Fri, 17 Jun 2011 13:25:18 -0400 From: Jefry Lagrange jefry.re...@gmail.com Subject: [Standards] XEP-198 when to start counting? To: standards@xmpp.org Message-ID: BANLkTi=rc_ey+kcnwxrt14yho-pllmj...@mail.gmail.com Content-Type: text/plain; charset=ISO-8859-1 When I test stream management with a prosody server, this happens: !-- Out -- enable xmlns=urn:xmpp:sm:2 resume=true / !-- Out -- iq type=set id=2 session xmlns=urn:ietf:params:xml:ns:xmpp-session / /iq !-- In -- enabled id='a8110118-42ec-485c-b386-85b2d12a02a1' resume='true' xmlns='urn:xmpp:sm:2'/ !-- Out -- r xmlns=urn:xmpp:sm:2 / !-- In -- iq id='2' type='result' to='jt...@thiessen.im/919aece2-21f2-4fc4-9e02-ac522509_'/ r xmlns='urn:xmpp:sm:2'/ !-- Out -- a xmlns=urn:xmpp:sm:2 h=1 / The client sent an IQ stanza before it received confirmation for SM negotiation enabled /. When should I start counting? When I send enable / or when I receive enabled /? Also in the XEP document, section 11.3 Stream Features. It says that The XMPP Registrar includes 'urn:xmpp:sm:3' in its registry of stream features at http://xmpp.org/registrar/stream-features.html. But at that URL you can't find 'urn:xmpp:sm:3', only 'urn:xmpp:sm:2' is there. -- Jefry Lagrange -- Message: 2 Date: Fri, 17 Jun 2011 18:37:15 +0100 From: Matthew Wild mwi...@gmail.com Subject: Re: [Standards] XEP-198 when to start counting? To: XMPP Standards standards@xmpp.org Message-ID: BANLkTi=jj4wwxm9cn_i2zfwlvth2utb...@mail.gmail.com Content-Type: text/plain; charset=UTF-8 On 17 June 2011 18:25, Jefry Lagrange jefry.re...@gmail.com wrote: When I test stream management with a prosody server, this happens: !-- Out -- enable xmlns=urn:xmpp:sm:2 resume=true / !-- Out -- iq type=set id=2 session xmlns=urn:ietf:params:xml:ns:xmpp-session / /iq !-- In -- enabled id='a8110118-42ec-485c-b386-85b2d12a02a1' resume='true' xmlns='urn:xmpp:sm:2'/ !-- Out -- r xmlns=urn:xmpp:sm:2 / !-- In -- iq id='2' type='result' to='jt...@thiessen.im/919aece2-21f2-4fc4-9e02-ac522509_'/ r xmlns='urn:xmpp:sm:2'/ !-- Out -- a xmlns=urn:xmpp:sm:2 h=1 / The client sent an IQ stanza before it received confirmation for SM negotiation enabled /. When should I start counting? When I send enable / or when I receive enabled /? When you send enable/, if you're going to send stanzas before you receive enabled/. The XEP says this: The value of 'h' starts at zero at the point stream management is enabled or requested to be enabled, is incremented to one for the first stanza handled, and is incremented by one again with each subsequent stanza handled. ...which isn't too decisive, to say the least :) You should definitely start counting from when you send enable/, because it is from receiving that that the server starts counting on its end and because it has no idea at what time you, the client, receive the enabled/. Also in the XEP document, section 11.3 Stream Features. It says that The XMPP Registrar includes 'urn:xmpp:sm:3' in its registry of stream features at http://xmpp.org/registrar/stream-features.html. But at that URL you can't find 'urn:xmpp:sm:3', only 'urn:xmpp:sm:2' is there. Probably a communications failure between the XEP Editor and the XMPP Registrar :) Regards, Matthew -- Message: 3 Date: Fri, 17 Jun 2011 11:40:41 -0600 From: Peter Saint-Andre stpe...@stpeter.im Subject: Re: [Standards] XEP-198 when to start counting? To: XMPP Standards standards@xmpp.org Message-ID: 4dfb9199.7050...@stpeter.im Content-Type: text/plain; charset=utf-8 On 6/17/11 11:37 AM, Matthew Wild wrote: On 17 June 2011 18:25, Jefry Lagrange jefry.re...@gmail.com wrote: When I test stream management with a prosody server, this happens: !-- Out -- enable xmlns=urn:xmpp:sm:2 resume=true / !-- Out -- iq type=set id=2 session xmlns=urn:ietf:params:xml:ns:xmpp
[Standards] XEP-198 when to start counting?
When I test stream management with a prosody server, this happens: !-- Out -- enable xmlns=urn:xmpp:sm:2 resume=true / !-- Out -- iq type=set id=2 session xmlns=urn:ietf:params:xml:ns:xmpp-session / /iq !-- In -- enabled id='a8110118-42ec-485c-b386-85b2d12a02a1' resume='true' xmlns='urn:xmpp:sm:2'/ !-- Out -- r xmlns=urn:xmpp:sm:2 / !-- In -- iq id='2' type='result' to='jt...@thiessen.im/919aece2-21f2-4fc4-9e02-ac522509_'/ r xmlns='urn:xmpp:sm:2'/ !-- Out -- a xmlns=urn:xmpp:sm:2 h=1 / The client sent an IQ stanza before it received confirmation for SM negotiation enabled /. When should I start counting? When I send enable / or when I receive enabled /? Also in the XEP document, section 11.3 Stream Features. It says that The XMPP Registrar includes 'urn:xmpp:sm:3' in its registry of stream features at http://xmpp.org/registrar/stream-features.html. But at that URL you can't find 'urn:xmpp:sm:3', only 'urn:xmpp:sm:2' is there. -- Jefry Lagrange
[Standards] Resuming after or before binding?
Hi! I'm implementing XEP-198, and it says that negotiation is done only after authentication and binding, which is fine. But at the time of resumption I don't know if I should start it after authentication and binding or only after authentication. The document at http://xmpp.org/extensions/xep-0198.html is not very explicit about it. Thanks. -- Jefry Lagrange -- Jefry Lagrange
