Re: [Standards] Re-escalating update of XEP-0313: MAM
On Oct 13, 2013, at 5:55 PM, Matthew Wild mwi...@gmail.com wrote: On 13 October 2013 16:46, Valérian Saliou valer...@valeriansaliou.name wrote: On Oct 11, 2013, at 1:13 PM, Matthew Wild mwi...@gmail.com wrote: Our server implementation (courtesy of Marco Cirillo aka Maranda) is also working well in production environment, with more than 300 unique users simultaneously using the module (and much more over a larger timespan). How is memory usage? If I recall you were storing each user's archive entirely in memory. Memory usage is pretty good. We don't see any difference (just a tiny one). We also configured the Lua GC to be triggered more often: AFAIK it's now being triggered when memory is multiplied by 1.25 (instead of the default x2 coming from Prosody, which was not efficient enough and caused allocated memory, which was not used at most, to grow rapidly). More frequent GC cycles cause higher latency and higher CPU usage. In my experience the optimal GC settings vary between different deployments, according to the modules loaded, traffic patterns, and the goal of the system (low resource usage or low latency). Anyway, this isn't related to MAM so much :) Yep, this is what we wanted to avoid while making the GC more aggressive. It turned out to work very well on Jappix.com (Debian 7 32bits w/ 512MB RAM, handling up to 800 simultaneous XMPP users connected w/ about 180MB daemon RAM usage). No latency noticed, mean CPU usage increased 4% from 7% to 11%. Pretty good results on our side :) Having had time to think about it now, and considering the general consensus forming in this thread, I'm not sure deletion is going to get in. A new XEP perhaps if you want to attempt it :) All right. A new XEP might be a better option yes. Excellent, thanks for your understanding. Regards, Matthew Ping me once you reviewed my changes and added yours, and we'll be ready to submit XEP update :) -- Valérian Saliou Jappix FrenchTouch Web Agency founder. Waaave Network co-founder. Uno IM product lead. More about me on my personal page. smime.p7s Description: S/MIME cryptographic signature
Re: [Standards] Re-escalating update of XEP-0313: MAM
On Oct 10, 2013, at 3:27 PM, Matthew Wild mwi...@gmail.com wrote: Hi Valérian, On 10 October 2013 05:56, Valérian Saliou valer...@valeriansaliou.name wrote: Hello everyone, I'm re-escalating my request for XEP-0313 update, which purpose is to bring a way to change the user's message archiving preferences (something very basic, so that we keep things simple and respect the philosophy of MAM which is to remove the burden of previous message archiving XEPs). Sorry about the delay in processing your contribution. I've not had much time to work on XEP stuff the past couple of months. No problem to that ;) You can find the Web browser-viewable version of the updated specification there: https://demo.frenchtouch.pro/valerian.saliou/xmpp/extensions/xep-0313.html Also, my previous update request from August on this mailing list: http://mail.jabber.org/pipermail/standards/2013-August/027873.html I had some questions about it from the other day which I can't recall right now (and I'm going out). I was going to email you at the weekend to see how your implementation of your changes was working out. Well since we discussed about MAM this summer, nothing more has been added. Our implementation in Jappix (client) and Metronome (server) is pretty stable and works great. Changes have only been bug fixes and performance improvements (mostly on server-side) rather than feature additions. I can tell you from a Web-client point of view that MAM (in its updated version) is just okay, and should be ready for implementation in more clients. Since we're Web only with Jappix, we have no archive synchronization between a local and remote database, though we could have such a feature with localStorage or more advanced IndexedDB. Our server implementation (courtesy of Marco Cirillo aka Maranda) is also working well in production environment, with more than 300 unique users simultaneously using the module (and much more over a larger timespan). Our server module has been coded from scratch, we considered the Prosody MAM module as too performance scratching for production and large-scale environments (as it serializes the whole message stanza before storing it as an XML/Lua object in DB and then unserialize it to send it back on retrieval - where on our side we only store the body and some critical information). Also, would be nice if someone could code an ejabberd and an Openfire implementation, this would really push clients forward to implementing it quickly. Oh, I think one of my questions was that it might make more sense to be able to delete by id (individual message), or even range of ids. This is a bit awkward to implement, as it doesn't really make sense to use RSM for deletion IMHO - but that's how access-by-id is implemented. Thoughts? You mean we'd better use RSM for purges also, instead of using the 'id' attribute on the purge / element? (as seen in: https://demo.frenchtouch.pro/valerian.saliou/xmpp/extensions/xep-0313.html#purge-single). Well my purge specification aims at extreme simplicity. But I understand we should be stricter to standards and rather use RSM, would make it more understandable for everyone. Also in my todo queue are updates related to message hints. After deletion and hints are in, I think we're ready to push for draft! That's great, did not think about those hints, might be useful for desktop clients syncing the archives from an already-existing local database! May I give you an access to the private Git repo hosting the XEP file? You can still retrieve the updated XEP XML source there: https://demo.frenchtouch.pro/valerian.saliou/xmpp/extensions/xep-0313.xml (it's kept up to date with my Git pushes in real-time) Regards, Matthew Cheers, -- Valérian Saliou Jappix FrenchTouch Web Agency founder. Uno IM product lead. More about me on my personal page. smime.p7s Description: S/MIME cryptographic signature
[Standards] Re-escalating update of XEP-0313: MAM
Hello everyone,I'm re-escalating my request for XEP-0313 update, which purpose is to bring a way to change the user's message archiving preferences (something very basic, so that we keep things simple and respect the philosophy of MAM which is to remove the burden of previous message archiving XEPs).Plus, we have a working implementation of the updated version of MAM in Jappix. We really needed to proceed the XEP update to make it fully usable by clients, and we'd be glad to see its support being pushed in major XMPP desktop mobile clients.You can find the Web browser-viewable version of the updated specification there:https://demo.frenchtouch.pro/valerian.saliou/xmpp/extensions/xep-0313.htmlAlso, my previous update request from August on this mailing list:http://mail.jabber.org/pipermail/standards/2013-August/027873.htmlHope we can discuss those new changes actively. For your convenience, I attached the diff file for my updated version.Cheers, --Valérian SaliouJappixFrenchTouch Web Agencyfounder.Uno IMproduct lead.More about me onmy personal page. 19a20 specXEP-0086/spec 32a34,41 urihttp://matthewwild.co.uk//uri /author author firstnameValérian/firstname surnameSaliou/surname emailvaler...@valeriansaliou.name/email jidvaler...@jappix.com/jid urihttp://valeriansaliou.name//uri 34a44,49 version0.3/version date2013-08-05/date initialsvs/initials remarkpIntroduce the ability to purge part or all of archives. Preferences more flexible: retrieval and push made possible. Starting namespace versioning./p/remark /revision revision 152c167 query xmlns='urn:xmpp:mam:tmp' queryid='f27' / --- query xmlns='urn:xmpp:mam:0' queryid='f27' / 173c188 query xmlns='urn:xmpp:mam:tmp' --- query xmlns='urn:xmpp:mam:0' 189,190c204,205 pIf omitted, the server SHOULD assume the value of lt;start/gt; to be equal to the date/time of the earliest message stored in the archive./p --- pIf omitted, the server SHOULD assume the value of lt;end/gt; to be equal to the date/time of the latest message stored in the archive./p 198c213 query xmlns='urn:xmpp:mam:tmp' --- query xmlns='urn:xmpp:mam:0' 206c221 query xmlns='urn:xmpp:mam:tmp' --- query xmlns='urn:xmpp:mam:0' 220c235 query xmlns='urn:xmpp:mam:tmp' --- query xmlns='urn:xmpp:mam:0' 234c249 iq type='error' id='q29302' --- iq type='error' id='q29302' to='ro...@montague.lit/orchard' 250,251c265,266 iq type='result' id='q29302' query xmlns='urn:xmpp:mam:tmp' --- iq type='result' id='q29302' to='jul...@capulet.lit/balcony' query xmlns='urn:xmpp:mam:0' 276c291 query xmlns='urn:xmpp:mam:tmp' --- query xmlns='urn:xmpp:mam:0' 286a302,317 section3 topic='Get latest messages' anchor='query-latest' pIn the case where the client does not build a local cache by fetching the whole archive in a chronological order, a reverse order logic is made possible by allowing the retrieval of latest archived messages. It achieves this by including a lt;setgt; element within its request, containing an empty lt;before/gt;. After the reply to this request is received, it can page through previous results normally (as defined in xep0059;)./p example caption='Querying the archive for latest messages'![CDATA[ iq type='get' id='q29304' query xmlns='urn:xmpp:mam:0' withjul...@capulet.lit/with set xmlns='http://jabber.org/protocol/rsm' max10/max before/ /set /query /iq ]]/example pNote: There is no concept of an open query, and servers MUST be prepared to receive arbitrary page requests at any time./p /section3 304c335 result xmlns='urn:xmpp:mam:tmp' queryid='f27' id='28482-98726-73623' --- result xmlns='urn:xmpp:mam:0' queryid='f27' id='28482-98726-73623' 317c348 result xmlns='urn:xmpp:mam:tmp' queryid='f27' id='5d398-28273-f7382' --- result xmlns='urn:xmpp:mam:0' queryid='f27' id='5d398-28273-f7382' 331a363,425 section1 topic='Purging the archive' anchor='purge' pWhen it comes that an user needs to do some cleanup in his archived conversations and remove all if not part of them, its client SHOULD provide him a way to remove archived messages./p pIn the case archiving is enforced by the server's policy, archive removal MAY not be allowed, resulting in the server replying with an error reply to such purge requests./p pA purge query consists of an lt;iq/gt; stanza addressed to the account or server entity hosting the archive, with a 'purge' payload. On receiving the query, the server purges the archived messages that match the client's given criteria, and finally returns the lt;iq/gt; result./p example caption='Purging the whole archive'![CDATA[ iq type='set' id='purge1' purge xmlns='urn:xmpp:mam:0' / /iq iq type='result' id='purge1'/ ]]/example pThe server MUST reply with a result lt;iq/gt; (purge successful) or an
Re: [Standards] Vcard4 implementation in Movim
Hey Tim, Deferred does not mean Deprecated, it simply means that an active/draft XEP has not been updated for a long time (more than 6 months I believe). By the way, I think the best thing to do is definitely not deprecating it, but rather switching it to Obsolete. We really have an opportunity to leverage the adoption of this XEP and broaden this. A good starting point would be to have a production-stable mod_vcard4 for Prosody and its forks (if not already done), and then moving fast on client implementations. Cheers, -- Valérian Saliou Jappix FrenchTouch Web Agency founder. Uno IM product lead. More about me on my personal page. On Aug 22, 2013, at 9:28 AM, Jaussoin Timothée edhe...@gmail.com wrote: Le 13/08/2013 20:17, Valérian Saliou a écrit : Hey, The transition to vCard4 could be made possible by having one single vCard module that would handle both the legacy vcard-temp and the newest vCard4. The module would put an abstraction layer over the legacy vCard thing so that it can handle/serve the set/get for vcard-temp also, while storing and managing database vCard data as if it was all-vCard4. Then, the clients still storing vcard-temp (that said, all the clients available as of today - except OneSocialWeb and probably BuddyCloud), would not have to be updated so that the vCard4 compatible clients can retrieve the proper/new vCard4 of an user only storing vcard-temp. We are really looking forward to implementing vCard4 XEP in Jappix, too - as we work closely with the Movim team! We can also look into making such an hybrid Prosody/Metronome module to handle that idea. Cheers, -- Valérian Saliou Jappix FrenchTouch Web Agency founder. Uno IM product lead. More about me on my personal page. On Aug 13, 2013, at 7:01 PM, Peter Saint-Andre stpe...@stpeter.im wrote: On 8/13/13 8:01 AM, Jaussoin Timothée wrote: Hi ! I see that the XEP-0292 has been deffered but I think that XMPP really need a descent and modenr vCard support I completely agree, which is why I authored XEP-0292. IMHO it needs only a few small fixes (the XSLT is incomplete). I will commit to updating it by the end of August. so I will implement the XEP in Movim using the PEP specification. That's great news! Please send any implementation feedback to this list. We really need to forget vcard-temp now and go ahead. +1 :-) If you are an XMPP developers I invite you to do the same :) I don't think that there's any change to do server-side ? It's just a simple PEP node for me. Can you confirm this (or not) ? For the IQ-based storage, some server-side changes would be needed. However, as I understand it many server implementations don't use vcard-temp as the native data storage format anyway (or, if they did, they could run vcard-temp data through a corrected and complete XSLT to serve up vCard4 data). Maybe I need to work on this for Prosody... ;-) Peter -- Peter Saint-Andre https://stpeter.im/ Hi ! Maybe we can change the vcard-temp XEP status to deffered, update the vCard4 XEP to only keep the PEP part (I see that in this XEP there's two way to set the vCard, a simple PEP node and a weird IQ/vcard-temp like way). We can also try to clean all the existents avatar/vcards XEP to keep a simple unique way to implement all theses things. Tim smime.p7s Description: S/MIME cryptographic signature
Re: [Standards] Vcard4 implementation in Movim
Hey, The transition to vCard4 could be made possible by having one single vCard module that would handle both the legacy vcard-temp and the newest vCard4. The module would put an abstraction layer over the legacy vCard thing so that it can handle/serve the set/get for vcard-temp also, while storing and managing database vCard data as if it was all-vCard4. Then, the clients still storing vcard-temp (that said, all the clients available as of today - except OneSocialWeb and probably BuddyCloud), would not have to be updated so that the vCard4 compatible clients can retrieve the proper/new vCard4 of an user only storing vcard-temp. We are really looking forward to implementing vCard4 XEP in Jappix, too - as we work closely with the Movim team! We can also look into making such an hybrid Prosody/Metronome module to handle that idea. Cheers, -- Valérian Saliou Jappix FrenchTouch Web Agency founder. Uno IM product lead. More about me on my personal page. On Aug 13, 2013, at 7:01 PM, Peter Saint-Andre stpe...@stpeter.im wrote: On 8/13/13 8:01 AM, Jaussoin Timothée wrote: Hi ! I see that the XEP-0292 has been deffered but I think that XMPP really need a descent and modenr vCard support I completely agree, which is why I authored XEP-0292. IMHO it needs only a few small fixes (the XSLT is incomplete). I will commit to updating it by the end of August. so I will implement the XEP in Movim using the PEP specification. That's great news! Please send any implementation feedback to this list. We really need to forget vcard-temp now and go ahead. +1 :-) If you are an XMPP developers I invite you to do the same :) I don't think that there's any change to do server-side ? It's just a simple PEP node for me. Can you confirm this (or not) ? For the IQ-based storage, some server-side changes would be needed. However, as I understand it many server implementations don't use vcard-temp as the native data storage format anyway (or, if they did, they could run vcard-temp data through a corrected and complete XSLT to serve up vCard4 data). Maybe I need to work on this for Prosody... ;-) Peter -- Peter Saint-Andre https://stpeter.im/ smime.p7s Description: S/MIME cryptographic signature
[Standards] [UPDATED] XEP-0313: Message Archive Management
Hello everyone, After a chat yesterday with Matthew Wild (MattJ) and Kim Alvefur (Zash) on the Prosody chatroom (pros...@conference.prosody.im) about the state of the MAM XEP and its Prosody implementation; I decided to update the XEP draft with what I/we thought was missing. I released a local v0.3 of the XEP, featuring: - Ability to retrieve preferences - Preferences push to all connected resources when updated by a client (as done in XEP-0016: Privacy Lists) - Ability to purge all or some archived messages (given a filter similar to the one used for retrieval) The updated, browser viewable XEP doc is available there: https://demo.frenchtouch.pro/valerian.saliou/xmpp/extensions/xep-0313.html Before submitting the updated XEP XML doc, I'd like to get some inputs of what's the point of view of the community on those updates. After (and if) these changes go mainstream, we will rework the Prosody's mod_mam, and finally implement it in Jappix (after dropping support for the old, flawed and too complex XEP-0136: Message Archiving). Cheers, -- Valérian Saliou Jappix FrenchTouch Web Agency founder. Uno IM product lead. More about me on my personal page. smime.p7s Description: S/MIME cryptographic signature
Re: [Standards] Proposed XMPP Extension: Notification Inbox
Hi, That was after seeing a ProcessOne conference video about this that I got the idea to write a XEP (http://www.process-one.net/en/blogs/article/sea_beyond_2011_talk_2_christophe_romain_karim_gemayel_on_pubsub_and_distri). ProcessOne defined a way to manage such notifications entirely server-side, as you think it is better. And I agree. But actually, there are no way to do it server-side with Pubsub: no XEP exists for that, and no home-made server patches exists for that. We need something which works with the actual standards, and it actually works (implemented in Jappix dev. version for instance). That's why we would better work on a clean way to store and retrieve notifications first, and then, standardize a way to do it entirely server-side. Cheers, Valérian Saliou Jappix founder --- On Wednesday, June 01, 2011 10:37:07 AM XMPP Extensions Editor wrote: The XMPP Extensions Editor has received a proposal for a new XEP. Title: Notification Inbox Abstract: This document defines a protocol to manage a notification inbox for pending events. URL: http://www.xmpp.org/extensions/inbox/notification-inbox.html The XMPP Council will decide at its next meeting whether to accept this proposal as an official XEP. Good to see people are thinking about this. However, I somewhat disagree with the approach of remote services directly writing into a user's inbox. My plan (which, admittedly, is not fully realized anywhere yet) is that entities may monitor a remote pubsub node via normal subscriptions, and one of these entities could be some sort of tracking service that monitors the node while the user is offline (we can call this inbox, but I prefer tracking service in order to stay a little more generic). Additionally, for cases like tagging a user whose tracking service is not yet subscribed to the node, the remote service can send an unsolicited notification. The above approach keeps remote services simple. They don't need to care what may count as an inbox notification and what may not. They simply emit regular pubsub notifications. Interpretation of these pubsub notifications by the tracker into an inbox-like facility and beyond (e.g. sending emails, SMS, etc to the user) remains a separate issue. So, maybe this XEP could be a definition of how a tracker offers inbox management to a user. But it would not necessarily define how the tracker consumes remote pubsub notifications and converts them. Justin
Re: [Standards] UPDATED: XEP-0277 (Microblogging over XMPP)
Hello, I will try to reply to your answers/suggestions ;) 1. In the microblogging XEP we wanted to define a simple way to share some files. But I never heard of the XEP-0135, will read it as soon as possible. Anyway, by looking at it quickly, it gives me some ideas about how we could manage photo albums/file albums to extend XMPP social features! 2.1. It is actually clear: there is an example after this text. 2.2. Mhh, maybe I did not understand you when you suggested us this. Were you talking of the parent microblog item of a comment node, in case of re-using the comment node somewhere else? Or to identify the original comment node parent if the parent was repeated by some users in their own microblog? 2.3. I don't really see where the security leak is. The microblog owner publishes an entry linking to a comment node, and the client retrieves it if there is a link to it. Or maybe you mean, avoid re-using a comment node URI in another microblog post by a nasty microblog item publisher? 3. I really think it makes things much simpler and clearer by separating the content itself from the informations, so that the owner can purge its microblog node without any meta-infos lost. 4. The notifications, as defined in the XEP, can be sent to the node's owner (you can retrieve who the owner is with a pubsub query), and all the comment publishers. By the way, the parent microblog item publisher might be also notified. 5. I studied this case before. There is a solution with the set xmlns='http://jabber.org/protocol/rsm' element, to say the server Hey, I just want the items published after the item with the ID=XX (see the Pubsub XEP for that), but its usage is not clearly described and it is not implemented in XMPP servers like ejabberd. 6. Related to Pubsub XEP, I remember some mails were exchanged about this Pubsub item count during the last month, right? Best regards, Valérian Saliou, Jappix founder - On 06/02/2011 12:36 AM, XMPP Extensions Editor wrote: Version 0.4 of XEP-0277 (Microblogging over XMPP) has been released. Abstract: This specification defines a method for microblogging over XMPP. Changelog: Added microblog informations feature, ID innacurracy fixed, urn:xmpp:inbox support added, new commenting namespaces, first comment marker, security considerations added. (vs) Diff: http://xmpp.org/extensions/diff/api/xep/0277/diff/0.3/vs/0.4 URL: http://xmpp.org/extensions/xep-0277.html The revision of document has disappointed me... 1. The 2.7 Attaching files to a post point. I don't really understand how client should to serve the files. I think that we should to use XEP-135 to do that. I don't understand why to use rel=self instead of rel=related for a link to a file (rel=self identifies a resource equivalent to the containing element which is obviously false. Please correct me if I wrong) 2. The 3. Comments point. 2.1. Romeo's client MIGHT add a element to the PubSub item. It's not clear what element might Romeo's client to add. 2.2. If the comment to publish is the first item of the node, the client MAY add a link element, with the rel=start attribute. Why only for first? How to determine if it first? What if the comment will be retracted? The spec doesn't clarify what href should be added with the link. I use rel=start as identify of the post to what whole replies node is related. I mean to the post which have link rel=replies to this pubsub node with comments. In this way client can always determine a thread to what comment is related even if it receive an xmpp-link to some comment and not for an original post. 2.3. There are one possible attack to a comments node. If some entity will publish post with link rel=replies to a comments node for another post then users can be confused with actual content of the post to which they replying. I assume to add cross references links to both post and replies node metadata and oblige client to check if links correspond to one another. 3. The 5. Microblog informations point. Why to make more and more new entities? I mean why we need the new node urn:xmpp:microblog:0:informations? Why can't we use the urn:xmpp:microblog:0 node itself? How to store metadata for replies nodes this way? 4. The 6. Notifications point is the most strange for me. Why to send notification when commenting on node? I mean we already have replies node to which anyone can subscribe and receive usual notifications. I understand that it's impossible to do that when replies node is not used. But how can I determine to which users I should send my notifications? If I constantly attached to the network I can see all thread and all comments but if I use different resources and often switch between them then all this schema will fail. Some things that not related to new revision: 5. Retrieve offline items. I don't know how it possible to retrieve items that was sent while my client was in offline or the priority of my client was not highest. I think
