Re: [Standards] Re-escalating update of XEP-0313: MAM
On 13 October 2013 16:46, Valérian Saliou valer...@valeriansaliou.name wrote: On Oct 11, 2013, at 1:13 PM, Matthew Wild mwi...@gmail.com wrote: Our server implementation (courtesy of Marco Cirillo aka Maranda) is also working well in production environment, with more than 300 unique users simultaneously using the module (and much more over a larger timespan). How is memory usage? If I recall you were storing each user's archive entirely in memory. Memory usage is pretty good. We don't see any difference (just a tiny one). We also configured the Lua GC to be triggered more often: AFAIK it's now being triggered when memory is multiplied by 1.25 (instead of the default x2 coming from Prosody, which was not efficient enough and caused allocated memory, which was not used at most, to grow rapidly). More frequent GC cycles cause higher latency and higher CPU usage. In my experience the optimal GC settings vary between different deployments, according to the modules loaded, traffic patterns, and the goal of the system (low resource usage or low latency). Anyway, this isn't related to MAM so much :) Having had time to think about it now, and considering the general consensus forming in this thread, I'm not sure deletion is going to get in. A new XEP perhaps if you want to attempt it :) All right. A new XEP might be a better option yes. Excellent, thanks for your understanding. Regards, Matthew
Re: [Standards] Re-escalating update of XEP-0313: MAM
On Oct 13, 2013, at 5:55 PM, Matthew Wild mwi...@gmail.com wrote: On 13 October 2013 16:46, Valérian Saliou valer...@valeriansaliou.name wrote: On Oct 11, 2013, at 1:13 PM, Matthew Wild mwi...@gmail.com wrote: Our server implementation (courtesy of Marco Cirillo aka Maranda) is also working well in production environment, with more than 300 unique users simultaneously using the module (and much more over a larger timespan). How is memory usage? If I recall you were storing each user's archive entirely in memory. Memory usage is pretty good. We don't see any difference (just a tiny one). We also configured the Lua GC to be triggered more often: AFAIK it's now being triggered when memory is multiplied by 1.25 (instead of the default x2 coming from Prosody, which was not efficient enough and caused allocated memory, which was not used at most, to grow rapidly). More frequent GC cycles cause higher latency and higher CPU usage. In my experience the optimal GC settings vary between different deployments, according to the modules loaded, traffic patterns, and the goal of the system (low resource usage or low latency). Anyway, this isn't related to MAM so much :) Yep, this is what we wanted to avoid while making the GC more aggressive. It turned out to work very well on Jappix.com (Debian 7 32bits w/ 512MB RAM, handling up to 800 simultaneous XMPP users connected w/ about 180MB daemon RAM usage). No latency noticed, mean CPU usage increased 4% from 7% to 11%. Pretty good results on our side :) Having had time to think about it now, and considering the general consensus forming in this thread, I'm not sure deletion is going to get in. A new XEP perhaps if you want to attempt it :) All right. A new XEP might be a better option yes. Excellent, thanks for your understanding. Regards, Matthew Ping me once you reviewed my changes and added yours, and we'll be ready to submit XEP update :) -- Valérian Saliou Jappix FrenchTouch Web Agency founder. Waaave Network co-founder. Uno IM product lead. More about me on my personal page. smime.p7s Description: S/MIME cryptographic signature
Re: [Standards] Re-escalating update of XEP-0313: MAM
On Oct 10, 2013, at 3:27 PM, Matthew Wild mwi...@gmail.com wrote: Hi Valérian, On 10 October 2013 05:56, Valérian Saliou valer...@valeriansaliou.name wrote: Hello everyone, I'm re-escalating my request for XEP-0313 update, which purpose is to bring a way to change the user's message archiving preferences (something very basic, so that we keep things simple and respect the philosophy of MAM which is to remove the burden of previous message archiving XEPs). Sorry about the delay in processing your contribution. I've not had much time to work on XEP stuff the past couple of months. No problem to that ;) You can find the Web browser-viewable version of the updated specification there: https://demo.frenchtouch.pro/valerian.saliou/xmpp/extensions/xep-0313.html Also, my previous update request from August on this mailing list: http://mail.jabber.org/pipermail/standards/2013-August/027873.html I had some questions about it from the other day which I can't recall right now (and I'm going out). I was going to email you at the weekend to see how your implementation of your changes was working out. Well since we discussed about MAM this summer, nothing more has been added. Our implementation in Jappix (client) and Metronome (server) is pretty stable and works great. Changes have only been bug fixes and performance improvements (mostly on server-side) rather than feature additions. I can tell you from a Web-client point of view that MAM (in its updated version) is just okay, and should be ready for implementation in more clients. Since we're Web only with Jappix, we have no archive synchronization between a local and remote database, though we could have such a feature with localStorage or more advanced IndexedDB. Our server implementation (courtesy of Marco Cirillo aka Maranda) is also working well in production environment, with more than 300 unique users simultaneously using the module (and much more over a larger timespan). Our server module has been coded from scratch, we considered the Prosody MAM module as too performance scratching for production and large-scale environments (as it serializes the whole message stanza before storing it as an XML/Lua object in DB and then unserialize it to send it back on retrieval - where on our side we only store the body and some critical information). Also, would be nice if someone could code an ejabberd and an Openfire implementation, this would really push clients forward to implementing it quickly. Oh, I think one of my questions was that it might make more sense to be able to delete by id (individual message), or even range of ids. This is a bit awkward to implement, as it doesn't really make sense to use RSM for deletion IMHO - but that's how access-by-id is implemented. Thoughts? You mean we'd better use RSM for purges also, instead of using the 'id' attribute on the purge / element? (as seen in: https://demo.frenchtouch.pro/valerian.saliou/xmpp/extensions/xep-0313.html#purge-single). Well my purge specification aims at extreme simplicity. But I understand we should be stricter to standards and rather use RSM, would make it more understandable for everyone. Also in my todo queue are updates related to message hints. After deletion and hints are in, I think we're ready to push for draft! That's great, did not think about those hints, might be useful for desktop clients syncing the archives from an already-existing local database! May I give you an access to the private Git repo hosting the XEP file? You can still retrieve the updated XEP XML source there: https://demo.frenchtouch.pro/valerian.saliou/xmpp/extensions/xep-0313.xml (it's kept up to date with my Git pushes in real-time) Regards, Matthew Cheers, -- Valérian Saliou Jappix FrenchTouch Web Agency founder. Uno IM product lead. More about me on my personal page. smime.p7s Description: S/MIME cryptographic signature
Re: [Standards] Re-escalating update of XEP-0313: MAM
On 11 October 2013 11:28, Valérian Saliou valer...@valeriansaliou.name wrote: Our server implementation (courtesy of Marco Cirillo aka Maranda) is also working well in production environment, with more than 300 unique users simultaneously using the module (and much more over a larger timespan). How is memory usage? If I recall you were storing each user's archive entirely in memory. Our server module has been coded from scratch, we considered the Prosody MAM module as too performance scratching for production and large-scale environments (as it serializes the whole message stanza before storing it as an XML/Lua object in DB and then unserialize it to send it back on retrieval - where on our side we only store the body and some critical information). MAM explicitly allows you to do this is you want. I'll defend Prosody not only storing the body by default, as I think people would like to preserve various extensions (off the top of my head: XHTML-IM, chat markers, security labels, signatures...). I don't think either you nor anyone on the Prosody team has done any benchmarks on our stanza storage API yet, so judging performance just because you think it will be slow isn't a valid technical argument :) In any case, rest assured that we'll be doing benchmarks before the 0.10 release that this is targeted at, to be sure we do have the most efficient storage solution for MAM archives. Also in my todo queue are updates related to message hints. After deletion and hints are in, I think we're ready to push for draft! Having had time to think about it now, and considering the general consensus forming in this thread, I'm not sure deletion is going to get in. A new XEP perhaps if you want to attempt it :) Regards, Matthew
Re: [Standards] Re-escalating update of XEP-0313: MAM
On Oct 11, 2013, at 12:28 PM, Valérian Saliou valer...@valeriansaliou.name wrote: Also, would be nice if someone could code an ejabberd and an Openfire implementation, this would really push clients forward to implementing it quickly. There is a pull request from Michael Uvarov https://github.com/esl/ejabberd/pull/114 with MAM implementation for MongooseIM (fork of ejabberd). We also have tests written for this functionality compatible with current XEP-0313 specification. We are going to merge this pull request as soon as there is a decision whether proposed XEP update should be considered or not. -- Best regards Michał Piotrowski michal.piotrow...@erlang-solutions.com
Re: [Standards] Re-escalating update of XEP-0313: MAM
Hi Michał, On 11 October 2013 12:17, Michał Piotrowski michal.piotrow...@erlang-solutions.com wrote: On Oct 11, 2013, at 12:28 PM, Valérian Saliou valer...@valeriansaliou.name wrote: Also, would be nice if someone could code an ejabberd and an Openfire implementation, this would really push clients forward to implementing it quickly. There is a pull request from Michael Uvarov https://github.com/esl/ejabberd/pull/114 with MAM implementation for MongooseIM (fork of ejabberd). We also have tests written for this functionality compatible with current XEP-0313 specification. We are going to merge this pull request as soon as there is a decision whether proposed XEP update should be considered or not. That's excellent news! Thanks for the info. Regards, Matthew
Re: [Standards] Re-escalating update of XEP-0313: MAM
On 11.10.2013 20:28, Valérian Saliou wrote: Also, would be nice if someone could code an ejabberd and an Openfire implementation, this would really push clients forward to implementing it quickly. There is an implementation in commercial version of ejabberd. I wrote it, the issues with the XEP I found I reported here: http://mail.jabber.org/pipermail/standards/2013-July/027761.html (mostly cosmetic bugs).
Re: [Standards] Re-escalating update of XEP-0313: MAM
Hi Valérian, On 10 October 2013 05:56, Valérian Saliou valer...@valeriansaliou.name wrote: Hello everyone, I'm re-escalating my request for XEP-0313 update, which purpose is to bring a way to change the user's message archiving preferences (something very basic, so that we keep things simple and respect the philosophy of MAM which is to remove the burden of previous message archiving XEPs). Sorry about the delay in processing your contribution. I've not had much time to work on XEP stuff the past couple of months. You can find the Web browser-viewable version of the updated specification there: https://demo.frenchtouch.pro/valerian.saliou/xmpp/extensions/xep-0313.html Also, my previous update request from August on this mailing list: http://mail.jabber.org/pipermail/standards/2013-August/027873.html I had some questions about it from the other day which I can't recall right now (and I'm going out). I was going to email you at the weekend to see how your implementation of your changes was working out. Oh, I think one of my questions was that it might make more sense to be able to delete by id (individual message), or even range of ids. This is a bit awkward to implement, as it doesn't really make sense to use RSM for deletion IMHO - but that's how access-by-id is implemented. Thoughts? Also in my todo queue are updates related to message hints. After deletion and hints are in, I think we're ready to push for draft! Regards, Matthew
Re: [Standards] Re-escalating update of XEP-0313: MAM
The Purge API is something that I have been looking for, but the spec doesn't appear to have any way of finding out what messages have been purged from the archive, which means the local cache cannot be synced up with archive without fetching it all. Spencer On Thu, Oct 10, 2013 at 2:27 PM, Matthew Wild mwi...@gmail.com wrote: Hi Valérian, On 10 October 2013 05:56, Valérian Saliou valer...@valeriansaliou.name wrote: Hello everyone, I'm re-escalating my request for XEP-0313 update, which purpose is to bring a way to change the user's message archiving preferences (something very basic, so that we keep things simple and respect the philosophy of MAM which is to remove the burden of previous message archiving XEPs). Sorry about the delay in processing your contribution. I've not had much time to work on XEP stuff the past couple of months. You can find the Web browser-viewable version of the updated specification there: https://demo.frenchtouch.pro/valerian.saliou/xmpp/extensions/xep-0313.html Also, my previous update request from August on this mailing list: http://mail.jabber.org/pipermail/standards/2013-August/027873.html I had some questions about it from the other day which I can't recall right now (and I'm going out). I was going to email you at the weekend to see how your implementation of your changes was working out. Oh, I think one of my questions was that it might make more sense to be able to delete by id (individual message), or even range of ids. This is a bit awkward to implement, as it doesn't really make sense to use RSM for deletion IMHO - but that's how access-by-id is implemented. Thoughts? Also in my todo queue are updates related to message hints. After deletion and hints are in, I think we're ready to push for draft! Regards, Matthew
Re: [Standards] Re-escalating update of XEP-0313: MAM
On 10 October 2013 16:22, Spencer MacDonald spencer.macdonald.ot...@gmail.com wrote: The Purge API is something that I have been looking for, but the spec doesn't appear to have any way of finding out what messages have been purged from the archive, which means the local cache cannot be synced up with archive without fetching it all. A very good point. This is why feature creep == complexity creep == evil :) I'll have to give this some thought... Regards, Matthew
Re: [Standards] Re-escalating update of XEP-0313: MAM
On 10 Oct 2013 11:23, Spencer MacDonald spencer.macdonald.ot...@gmail.com wrote: The Purge API is something that I have been looking for, but the spec doesn't appear to have any way of finding out what messages have been purged from the archive, which means the local cache cannot be synced up with archive without fetching it all. Astonishingly close to 5 years ago, I suggested that if we wanted to really have a generic archival sync protocol, we'd be best off just using IMAP. This is why. So removing random messages and then being able to later figure out which have been removed without storing buckets of state on the server is a hard problem. IMAP solves this by using eternally-unique ids that are numeric (and so collapse into ranges nicely, and are issued predictably), and deploying some fearsome logic, and allowing certain deletions to be re-transmitted. But it's hard - I think there's a good argument for moving any purging and resync to a different spec at least, and keeping MAM simple in scope. Dave.
Re: [Standards] Re-escalating update of XEP-0313: MAM
On Thu, Oct 10, 2013 at 4:52 PM, Dave Cridland d...@cridland.net wrote: But it's hard - I think there's a good argument for moving any purging and resync to a different spec at least, and keeping MAM simple in scope. Oh please oh please oh please, yes. /K
Re: [Standards] Re-escalating update of XEP-0313: MAM
On 10 October 2013 16:52, Dave Cridland d...@cridland.net wrote: On 10 Oct 2013 11:23, Spencer MacDonald spencer.macdonald.ot...@gmail.com wrote: The Purge API is something that I have been looking for, but the spec doesn't appear to have any way of finding out what messages have been purged from the archive, which means the local cache cannot be synced up with archive without fetching it all. Astonishingly close to 5 years ago, I suggested that if we wanted to really have a generic archival sync protocol, we'd be best off just using IMAP. This is why. So removing random messages and then being able to later figure out which have been removed without storing buckets of state on the server is a hard problem. IMAP solves this by using eternally-unique ids that are numeric (and so collapse into ranges nicely, and are issued predictably), and deploying some fearsome logic, and allowing certain deletions to be re-transmitted. Reminds me of: https://github.com/pushcx/sup/blob/master/lib/sup/imap.rb#L12 (warning: strong language ahead!) But it's hard - I think there's a good argument for moving any purging and resync to a different spec at least, and keeping MAM simple in scope. Naturally I'd be in favour of that! Since Valérian proposed this change I've mostly been thinking about the server-side, for which implementing this operation is easy (and already done now by Kim in Prosody I think). But it does change the fundamental principles of how a MAM archive is expected to behave. Also, expanding Spencer's point, other clients connected to the account would also need to be notified of the deletion - and they might be offline at the time. Potential issues if clients aren't aware of deletions is most visible if the tail of the archive is removed, clients won't be able to fetch from the last message they saw. In fact requesting an id that doesn't exist should probably fetch the whole archive (with the old message has gone, and the whole remainder of the archive is new to the client). Regards, Matthew
Re: [Standards] Re-escalating update of XEP-0313: MAM
Also, expanding Spencer's point, other clients connected to the account would also need to be notified of the deletion - and they might be offline at the time. What about exposing the archive as a pub-sub node? That would take care of notifications and resyncing etc. S. -- Simon Tennant | buddycloud.com | +49 17 8545 0880 | office hours: goo.gl/tQgxP
Re: [Standards] Re-escalating update of XEP-0313: MAM
On 10 October 2013 17:17, Simon Tennant si...@buddycloud.com wrote: Also, expanding Spencer's point, other clients connected to the account would also need to be notified of the deletion - and they might be offline at the time. What about exposing the archive as a pub-sub node? That would take care of notifications and resyncing etc. and using what protocol for sync? Regards, Matthew
Re: [Standards] Re-escalating update of XEP-0313: MAM
On Thu, Oct 10, 2013 at 5:44 PM, Matthew Wild mwi...@gmail.com wrote: On 10 October 2013 17:17, Simon Tennant si...@buddycloud.com wrote: Also, expanding Spencer's point, other clients connected to the account would also need to be notified of the deletion - and they might be offline at the time. What about exposing the archive as a pub-sub node? That would take care of notifications and resyncing etc. and using what protocol for sync? I thought you could use MAM for that. /K
Re: [Standards] Re-escalating update of XEP-0313: MAM
On 2013-10-10 17:52, Dave Cridland wrote: But it's hard - I think there's a good argument for moving any purging and resync to a different spec at least, and keeping MAM simple in scope. Yes, we like simple. If you want resync and deletion there's always XEP-0136 :) signature.asc Description: OpenPGP digital signature
Re: [Standards] Re-escalating update of XEP-0313: MAM
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 10/10/2013 02:33 PM, Kim Alvefur wrote: On 2013-10-10 17:52, Dave Cridland wrote: But it's hard - I think there's a good argument for moving any purging and resync to a different spec at least, and keeping MAM simple in scope. Yes, we like simple. If you want resync and deletion there's always XEP-0136 :) XEP-0136 got way too complex. Let's not do that again. Peter -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.12 (GNU/Linux) Comment: GPGTools - http://gpgtools.org Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQIcBAEBAgAGBQJSV3BFAAoJEOoGpJErxa2pbBMP/A70dNBBNIjOjEysJ3us4Ccj s6TsaflaLpR3sZEP21gy3zx+tgy3veFHmsRPTV1xnVPPqIS7K5t3Oe9UGEHnh480 rjTaKdAhKYE62ZGh827UmeSwt1IvZ64sd9iEY6WvhtEk5ZgUx+ll++V3WEjRM70A tAOPwcKJ7YZCJ442vnNO3lV73letmGniQtZjGlmprD3qfYAljGJZAMa5uYd6BY1q 5v1F76g4Kyd7xK6dAm1Gl8pgQy2Oe6QESLJ7FcdLdc/VaxWWReATj43jOvK4DIaq MEV4d94LnwjikB9I8vLxkw7+MLBsaKVM/3MGF3ne4CahjQN1GnFhlUzC/ttbOHAF 3JnDSXPWnaW+YjBAIwa9IXfDfhc03oovtDx9wokhtYU6pZ4kmGesWzZu6/U2et9e hDXAt00p9Wpe2X8xxWxNYECs+MvoRm3TmbWlIVnDb12mxV3f/kNDRUGyyrllpxbI +a5FhztVoy3hA+H5QDJ4bageC7aJRm5enmBwN5Lt3+OG1n70IhMMLqwoXRunMtb8 t1fKq5F8tfSEU2S9yg2YIgamu/5jOyZOlM5HhEBbd9Zewp4bzBv00WzD8DIAYfFv xLGdQ5Aja05JvVQXB42xiBJF2B396j7CVokbcp0e156s/c2cK0oMBeAbpfbngyQJ 47z1QVp+EdsGBinIiSuA =dRlh -END PGP SIGNATURE-
[Standards] Re-escalating update of XEP-0313: MAM
Hello everyone,I'm re-escalating my request for XEP-0313 update, which purpose is to bring a way to change the user's message archiving preferences (something very basic, so that we keep things simple and respect the philosophy of MAM which is to remove the burden of previous message archiving XEPs).Plus, we have a working implementation of the updated version of MAM in Jappix. We really needed to proceed the XEP update to make it fully usable by clients, and we'd be glad to see its support being pushed in major XMPP desktop mobile clients.You can find the Web browser-viewable version of the updated specification there:https://demo.frenchtouch.pro/valerian.saliou/xmpp/extensions/xep-0313.htmlAlso, my previous update request from August on this mailing list:http://mail.jabber.org/pipermail/standards/2013-August/027873.htmlHope we can discuss those new changes actively. For your convenience, I attached the diff file for my updated version.Cheers, --Valérian SaliouJappixFrenchTouch Web Agencyfounder.Uno IMproduct lead.More about me onmy personal page. 19a20 specXEP-0086/spec 32a34,41 urihttp://matthewwild.co.uk//uri /author author firstnameValérian/firstname surnameSaliou/surname emailvaler...@valeriansaliou.name/email jidvaler...@jappix.com/jid urihttp://valeriansaliou.name//uri 34a44,49 version0.3/version date2013-08-05/date initialsvs/initials remarkpIntroduce the ability to purge part or all of archives. Preferences more flexible: retrieval and push made possible. Starting namespace versioning./p/remark /revision revision 152c167 query xmlns='urn:xmpp:mam:tmp' queryid='f27' / --- query xmlns='urn:xmpp:mam:0' queryid='f27' / 173c188 query xmlns='urn:xmpp:mam:tmp' --- query xmlns='urn:xmpp:mam:0' 189,190c204,205 pIf omitted, the server SHOULD assume the value of lt;start/gt; to be equal to the date/time of the earliest message stored in the archive./p --- pIf omitted, the server SHOULD assume the value of lt;end/gt; to be equal to the date/time of the latest message stored in the archive./p 198c213 query xmlns='urn:xmpp:mam:tmp' --- query xmlns='urn:xmpp:mam:0' 206c221 query xmlns='urn:xmpp:mam:tmp' --- query xmlns='urn:xmpp:mam:0' 220c235 query xmlns='urn:xmpp:mam:tmp' --- query xmlns='urn:xmpp:mam:0' 234c249 iq type='error' id='q29302' --- iq type='error' id='q29302' to='ro...@montague.lit/orchard' 250,251c265,266 iq type='result' id='q29302' query xmlns='urn:xmpp:mam:tmp' --- iq type='result' id='q29302' to='jul...@capulet.lit/balcony' query xmlns='urn:xmpp:mam:0' 276c291 query xmlns='urn:xmpp:mam:tmp' --- query xmlns='urn:xmpp:mam:0' 286a302,317 section3 topic='Get latest messages' anchor='query-latest' pIn the case where the client does not build a local cache by fetching the whole archive in a chronological order, a reverse order logic is made possible by allowing the retrieval of latest archived messages. It achieves this by including a lt;setgt; element within its request, containing an empty lt;before/gt;. After the reply to this request is received, it can page through previous results normally (as defined in xep0059;)./p example caption='Querying the archive for latest messages'![CDATA[ iq type='get' id='q29304' query xmlns='urn:xmpp:mam:0' withjul...@capulet.lit/with set xmlns='http://jabber.org/protocol/rsm' max10/max before/ /set /query /iq ]]/example pNote: There is no concept of an open query, and servers MUST be prepared to receive arbitrary page requests at any time./p /section3 304c335 result xmlns='urn:xmpp:mam:tmp' queryid='f27' id='28482-98726-73623' --- result xmlns='urn:xmpp:mam:0' queryid='f27' id='28482-98726-73623' 317c348 result xmlns='urn:xmpp:mam:tmp' queryid='f27' id='5d398-28273-f7382' --- result xmlns='urn:xmpp:mam:0' queryid='f27' id='5d398-28273-f7382' 331a363,425 section1 topic='Purging the archive' anchor='purge' pWhen it comes that an user needs to do some cleanup in his archived conversations and remove all if not part of them, its client SHOULD provide him a way to remove archived messages./p pIn the case archiving is enforced by the server's policy, archive removal MAY not be allowed, resulting in the server replying with an error reply to such purge requests./p pA purge query consists of an lt;iq/gt; stanza addressed to the account or server entity hosting the archive, with a 'purge' payload. On receiving the query, the server purges the archived messages that match the client's given criteria, and finally returns the lt;iq/gt; result./p example caption='Purging the whole archive'![CDATA[ iq type='set' id='purge1' purge xmlns='urn:xmpp:mam:0' / /iq iq type='result' id='purge1'/ ]]/example pThe server MUST reply with a result lt;iq/gt; (purge successful) or an
