Re: [Standards] Proposed XMPP Extension: HTTP File Upload
Hello, We had a try at changing a bit our approach for HTTP-based file transfer to support use cases, pointed out by Sam and Daniel. The result is here: https://github.com/processone/ejabberd-saas-docs/blob/master/xmpp-specs/http-filetransfer/http-filetransfer.md What we changed is: - Ability to have permanent download URLs that do not require the client to know about the protocol. When URL is permanent (or before the download URL expires) client is not forced to request a download URL from the file id. - Added ability for the server to pass headers along with upload URL. The client may need to use those HTTP headers to be allowed to upload the file. Please, let me know if this is helpful. We have something that we have left pending, waiting for your feedback: the hash for now is a binary md5 as this is the one supported by Amazon S3. However, other services may use other type of hash or no hash at all. What bothered me is that User avatar use SHA-1 as id. Some service may use the same hash for consistency. Some service may not require any hash at all or just use file size as a basic check. We may need a data form to accommodate that flexibility. Do you think it is worth it ? -- Mickaël Rémond http://www.process-one.net
Re: [Standards] Proposed XMPP Extension: HTTP File Upload
Hello Philipp, On 30 Jul 2015, at 20:12, Philipp Hancke wrote: HTTP Upload is just another transport as far as jingle is concerned. So you would send your session-initiate with the XEP-0234 description (instead of reinventing this in example 5) along with an empty transport/ qualified by urn:xmpp:jingle:http-transport-something-something. In the session-accept you would get a transport element telling you where to put this. This would basically be the put url. After the transfer is done, the receiver sends a session-info telling you the location where to get the file (this is the only design change compared to the current approach but I think it allows the service to process things and decouple things) I think this wouldn't change the semantics of the proposal much, just how things are done on the wire. In my mind, Jingle set of protocols are for live streams and they have the following characteristics: - You negotiate either an infinite stream (for voice / video calls) or more limited streams. - Negotiation happens between sending and receiving party. It means they need to be both online at the same time. - They can agree on any transport they like, including HTTP. We have developed a streaming HTTP proxy to allow upload and download at the same time before the file is fully received. However, the semantic in that case is still live transfer. The protocol proposed by Daniel is asynchronous by nature. It is more about file sharing than transfer actually. I truly think both approach have their use and they do not overlap: - Jingle for live negotiation if binary stream transfer. Use is mostly for live conferencing systems for example. - HTTP file slot approach for asynchronous sharing. Use is mostly for mobile client that have, by nature, very intermittent connection. I hope clearly specialising both XEP will make developers of both types of apps happy. -- Mickaël Rémond http://www.process-one.net
Re: [Standards] Proposed XMPP Extension: HTTP File Upload
Hello Sam, On 1 Aug 2015, at 21:51, Sam Whited wrote: If you're going to try and implement a version of your spec with headers and test it against a particular service, I recommend AWS S3 (I've been using it as an example as it's probably the most common large file storage solution [citation needed]). Actually, the first spec we published is already used for an Amazon S3 backend in production. -- Mickaël Rémond http://www.process-one.net
Re: [Standards] Proposed XMPP Extension: HTTP File Upload
Sat, 1 Aug 2015 14:51:19 -0500 Sam Whited s...@samwhited.com wrote: If Content-MD5 were to be supported in particular it would probably have to be something that clients supported, eg. header name=Content-MD5 / Another issue is how to pass this data to the file recipient. While it looks obvious to pass an url, passing headers looks not that simple and XEP doesn't describe that as it stands for Upload only. I think there is no much sense to describe headers in this XEP without suggesting rules on passing this data to the file recipients.
Re: [Standards] Proposed XMPP Extension: HTTP File Upload
On Sat, Aug 1, 2015 at 5:22 AM, Mickaël Rémond mrem...@process-one.net wrote: ok, I think I missed you point there, then. Do you mean the information passed in the slot request on XMPP should pass generic extra values that can be use in header by XMPP server component, like Content-MD5 header ? That's what I was thinking indeed; of course, Content-MD5 in particular would require that clients had some sort of support for it, so it's sort of a separate issue. A better example might be something like an oauth token, so the server could generate a PUT URL and a one time use oauth token, and pass down: header name=AuthorizationBearer sometoken/header and the client would construct the PUT request with the header: Authorization: Bearer sometoken If Content-MD5 were to be supported in particular it would probably have to be something that clients supported, eg. header name=Content-MD5 / without a text node might trigger clients which support it to send an appropriate header. Of course, this would mean that clients have to support a list of such headers (I can't think of any others, but I'm sure there are other headers like this which require that the client generate the actual header content), and that feels poor to me. If you're going to try and implement a version of your spec with headers and test it against a particular service, I recommend AWS S3 (I've been using it as an example as it's probably the most common large file storage solution [citation needed]). I have a branch of Daniel's reference component with S3 support here: https://github.com/SamWhited/HttpUploadComponent/tree/s3_support though w/o generic header support it's a bit brittle and requires changes to Conversations (https://github.com/SamWhited/Conversations/tree/http_upload_s3_support) to work properly. —Sam -- Sam Whited pub 4096R/54083AE104EA7AD3 https://blog.samwhited.com
Re: [Standards] Proposed XMPP Extension: HTTP File Upload
Hello Sam, On 31 Jul 2015, at 22:17, Sam Whited wrote: On Fri, Jul 31, 2015 at 1:25 AM, Mickaël Rémond mrem...@process-one.net wrote: I don't see any mention of headers in your document, except for Content-MD5, (though it does cover the case where you can request a file's GET url from the server). ok, I think I missed you point there, then. Do you mean the information passed in the slot request on XMPP should pass generic extra values that can be use in header by XMPP server component, like Content-MD5 header ? For example, slot request could be: iq type='set' id='upload1' to='domain.com' query xmlns='p1:http-file-transfer' header name='Content-MD5' value='qN48INKsmQyPMZkHs0DWog==' / /query /iq Is that what you mean ? I'd like that, but I'm not sure if Daniel would :) I know he really wants to keep it simple. Yes, I understand that. That's why I am asking to not give the impression I am too pushy in another direction other that the original need :) Maybe I will just update my doc and try to adapt our implementation to another backend (Dropbox ?) and check if it adapts well. I see generic headers as something that are almost a requirement to make this useable though. Otherwise it only works for the most simple cases (eg. small XMPP servers that want to run their own simple storage). I agree, we had the same needs to support HTTP storage backends. As long as you want to support large user base, the file storage is almost all the time a separate service. -- Mickaël Rémond http://www.process-one.net
Re: [Standards] Proposed XMPP Extension: HTTP File Upload
On Fri, Jul 31, 2015 at 1:25 AM, Mickaël Rémond mrem...@process-one.net wrote: I think what we had implemented covers all this: https://github.com/processone/ejabberd-saas-docs/blob/master/http-filetransfer/http-filetransfer.md Here is what you have in mind ? Anything else missing ? I don't see any mention of headers in your document, except for Content-MD5, (though it does cover the case where you can request a file's GET url from the server). If you want I can try to propose a patch on HTTP file upload proto XEP based on some of these ideas, but I would like to be sure this is helpful. I'd like that, but I'm not sure if Daniel would :) I know he really wants to keep it simple. I see generic headers as something that are almost a requirement to make this useable though. Otherwise it only works for the most simple cases (eg. small XMPP servers that want to run their own simple storage). —Sam -- Sam Whited pub 4096R/54083AE104EA7AD3 https://blog.samwhited.com
Re: [Standards] Proposed XMPP Extension: HTTP File Upload
Hello Sam, Le 30 Jul 2015 à 20:21, Sam Whited s...@samwhited.com a écrit : My position is that we should adopt HTTP File Upload with the provision that it add support for specifying headers that the client should send back, and possibly an IQ to fetch the GET URL later to support file upload systems where the final file name is not known up front (though this is less of a priority for me). I think what we had implemented covers all this: https://github.com/processone/ejabberd-saas-docs/blob/master/http-filetransfer/http-filetransfer.md Here is what you have in mind ? Anything else missing ? If you want I can try to propose a patch on HTTP file upload proto XEP based on some of these ideas, but I would like to be sure this is helpful. -- Mickaël Rémond http://www.process-one.net
Re: [Standards] Proposed XMPP Extension: HTTP File Upload
On Thu, Jul 30, 2015 at 5:06 AM, Goffi go...@goffi.org wrote: jid auth validation is a clear big advantage over random url. This is one of the reasons why we need some method of conveying headers which the clients should PUT or GET with, IMO (so that we can do some form of authentication, eg one time use oauth tokens, if necessary). —Sam -- Sam Whited pub 4096R/54083AE104EA7AD3 https://blog.samwhited.com
Re: [Standards] Proposed XMPP Extension: HTTP File Upload
On Thu, Jul 30, 2015 at 1:12 PM, Philipp Hancke fi...@goodadvice.pages.de wrote: 2015-07-30 11:07 GMT+02:00 Goffi go...@goffi.org: On 30/07/2015 02:29, Sam Whited wrote: HTTP give no advantage over Socks5, and doesn't do NAT traversal as Jingle can do, and it's an other whole different server to maintain. For the record, my name is a copy/paste error or typo in the quoted message there: I did not write that statement. Goffi did :) My position is that we should adopt HTTP File Upload with the provision that it add support for specifying headers that the client should send back, and possibly an IQ to fetch the GET URL later to support file upload systems where the final file name is not known up front (though this is less of a priority for me). —Sam -- Sam Whited pub 4096R/54083AE104EA7AD3 https://blog.samwhited.com
Re: [Standards] Proposed XMPP Extension: HTTP File Upload
2015-07-30 11:07 GMT+02:00 Goffi go...@goffi.org: On 30/07/2015 02:29, Sam Whited wrote: HTTP give no advantage over Socks5, and doesn't do NAT traversal as Jingle can do, and it's an other whole different server to maintain. When uploading a file to a server you don't need NAT traversal. NAT traversal is a feature of xep-0176, not 0166. Thinking out loud: HTTP Upload is just another transport as far as jingle is concerned. So you would send your session-initiate with the XEP-0234 description (instead of reinventing this in example 5) along with an empty transport/ qualified by urn:xmpp:jingle:http-transport-something-something. In the session-accept you would get a transport element telling you where to put this. This would basically be the put url. After the transfer is done, the receiver sends a session-info telling you the location where to get the file (this is the only design change compared to the current approach but I think it allows the service to process things and decouple things) I think this wouldn't change the semantics of the proposal much, just how things are done on the wire.
Re: [Standards] Proposed XMPP Extension: HTTP File Upload
I do like the idea of an additional jingle transport, that's very pluggable and does not exclude using of other transports. On 30/07/2015 20:21, Sam Whited wrote: On Thu, Jul 30, 2015 at 1:12 PM, Philipp Hancke fi...@goodadvice.pages.de wrote: 2015-07-30 11:07 GMT+02:00 Goffi go...@goffi.org: On 30/07/2015 02:29, Sam Whited wrote: HTTP give no advantage over Socks5, and doesn't do NAT traversal as Jingle can do, and it's an other whole different server to maintain. For the record, my name is a copy/paste error or typo in the quoted message there: I did not write that statement. Goffi did :) My position is that we should adopt HTTP File Upload with the provision that it add support for specifying headers that the client should send back, and possibly an IQ to fetch the GET URL later to support file upload systems where the final file name is not known up front (though this is less of a priority for me). —Sam
Re: [Standards] Proposed XMPP Extension: HTTP File Upload
Thu, 30 Jul 2015 12:06:18 +0200 Goffi go...@goffi.org wrote: why wouldn't file transfer work offline or with MUC ? We just need a way to request a file with an uri. We definitely need to transfer files with MUCs (think of lolcats in chats). Regarding offline: sometimes I want to send a lolcat to my offline friend and go sleep. That's a pretty common usage. This is a situation when we cannot negotiate. jid auth validation is a clear big advantage over random url. It depends. jingle is adopted in several clients as far as I know: https://en.wikipedia.org/wiki/Jingle_(protocol) Most implementations are buggy as hell and almost unusable. And I'm not saying about poor user interface, but that's a whole different story.
Re: [Standards] Proposed XMPP Extension: HTTP File Upload
On 30/07/2015 11:35, Evgeny Khramtsov wrote: Thu, 30 Jul 2015 11:07:28 +0200 Goffi go...@goffi.org wrote: What is the point in implementing file transfer protocol which will not work in all cases (MUC, offline, etc)? Why a developer would need proxy65 if it's not MUC friendly? I really see no point. why wouldn't file transfer work offline or with MUC ? We just need a way to request a file with an uri. Beside the risk of losing the url that I have already mentioned, HTTP give no advantage over Socks5, and doesn't do NAT traversal as Jingle can do, and it's an other whole different server to maintain. There is a clear advantage over Socks5: MUC and offline support. And there is no NAT traversal problem for HTTP as far as I'm concerned. jid auth validation is a clear big advantage over random url. Jingle adoption? It has been 10 years left. Where is it? Should we wait 10 more years? jingle is adopted in several clients as far as I know: https://en.wikipedia.org/wiki/Jingle_(protocol) Not sure if FT is available everywhere though. Anyway, I agree that a first good step would be to deprecate XEP-0096, and I have already expressed my opinion, if HTTP upload is validated I'll implement it. Goffi
Re: [Standards] Proposed XMPP Extension: HTTP File Upload
Hi folks, This thread shows that XMPP really need some roadmap of what it should look like and what is its target on the market. Because otherwise it's not obvious, how to decide on the topics like this. Of course, jingle does not mean we can't serve offline transfers, it can be solved by adding another infrastructure for hosting them, but given the current level of implementation jingle, are we ready to add more and more XEPs again? While I'm very interested in an infrastructure that's fully XMPP-driven, I'm not sure that we can obtain it in a reasonable time, because take a look on Pubsub. But again, I don't want to look like an advocate of utilizing HTTP, I really think it would be much greater to have XMPP infrastructure, but how can we motivate someone to implement that? I know that XSF is not commercial and probably does not want to solve such a question, delegating that to the companies that use XMPP instead, but can it actually be the thing because of which many companies refused to use XMPP recently? They can be just not sure how to use it, yes, they could contribute to the community and solve this but somehow they don't want? Sorry, for some offtopic, but I'm really sensible to this question, as I can't decide for myself also. Thanks. On 30/07/2015 12:06, Goffi wrote: On 30/07/2015 11:35, Evgeny Khramtsov wrote: Thu, 30 Jul 2015 11:07:28 +0200 Goffi go...@goffi.org wrote: What is the point in implementing file transfer protocol which will not work in all cases (MUC, offline, etc)? Why a developer would need proxy65 if it's not MUC friendly? I really see no point. why wouldn't file transfer work offline or with MUC ? We just need a way to request a file with an uri. Beside the risk of losing the url that I have already mentioned, HTTP give no advantage over Socks5, and doesn't do NAT traversal as Jingle can do, and it's an other whole different server to maintain. There is a clear advantage over Socks5: MUC and offline support. And there is no NAT traversal problem for HTTP as far as I'm concerned. jid auth validation is a clear big advantage over random url. Jingle adoption? It has been 10 years left. Where is it? Should we wait 10 more years? jingle is adopted in several clients as far as I know: https://en.wikipedia.org/wiki/Jingle_(protocol) Not sure if FT is available everywhere though. Anyway, I agree that a first good step would be to deprecate XEP-0096, and I have already expressed my opinion, if HTTP upload is validated I'll implement it. Goffi
Re: [Standards] Proposed XMPP Extension: HTTP File Upload
On Thu, Jul 30, 2015 at 12:19 PM, Sergey Dobrov bin...@jrudevels.org wrote: While I'm very interested in an infrastructure that's fully XMPP-driven, I'm But again, I don't want to look like an advocate of utilizing HTTP, I really think it would be much greater to have XMPP infrastructure, but how can we motivate someone to implement that? I can understand how administrators (and companies in general) will not choose XMPP if it's an almighty protocol handling all of their requirements. An optimal infrastructure is made up of many protocols working together. In many cases, HTTP may and can be useful IMHO. But I don't want to repeat other people's arguments about that. -- Daniele
Re: [Standards] Proposed XMPP Extension: HTTP File Upload
On 30/07/2015 02:29, Sam Whited wrote: HTTP Upload should be used when you want a file to be made available somewhere to one or more people (maybe publically, maybe with authentication), it's a sharing action, whereas JFT should be used when you want a (potentially encrypted) file transfer to a single recepient that should never be stored on an intermediary machine, eg. the sending action. My point is that HTTP transfer is nothing more than a P2P transfer between server and requester(s) (and why would you not store the file ?), and while trying to make life easier it will result in either developers implementing both (so more complications at the end), either they'll implement only one (say HTTP), resulting in a feature loss. Beside the risk of losing the url that I have already mentioned, HTTP give no advantage over Socks5, and doesn't do NAT traversal as Jingle can do, and it's an other whole different server to maintain. Anyway, if the HTTP XEP is published I'll implement it anyway, I just hope it will not have bad side effects on jingle adoption, neither it will prevent to have new XEP for requesting a file with jingle (which is a more elegant option in my opinion). If indeed both can live together, well, why not. Goffi
Re: [Standards] Proposed XMPP Extension: HTTP File Upload
Thu, 30 Jul 2015 11:07:28 +0200 Goffi go...@goffi.org wrote: My point is that HTTP transfer is nothing more than a P2P transfer between server and requester(s) (and why would you not store the file ?), and while trying to make life easier it will result in either developers implementing both (so more complications at the end), either they'll implement only one (say HTTP), resulting in a feature loss. What is the point in implementing file transfer protocol which will not work in all cases (MUC, offline, etc)? Why a developer would need proxy65 if it's not MUC friendly? I really see no point. Beside the risk of losing the url that I have already mentioned, HTTP give no advantage over Socks5, and doesn't do NAT traversal as Jingle can do, and it's an other whole different server to maintain. There is a clear advantage over Socks5: MUC and offline support. And there is no NAT traversal problem for HTTP as far as I'm concerned. Anyway, if the HTTP XEP is published I'll implement it anyway, I just hope it will not have bad side effects on jingle adoption, neither it will prevent to have new XEP for requesting a file with jingle (which is a more elegant option in my opinion). If indeed both can live together, well, why not. Jingle adoption? It has been 10 years left. Where is it? Should we wait 10 more years?
Re: [Standards] Proposed XMPP Extension: HTTP File Upload
2015-07-30 11:07 GMT+02:00 Goffi go...@goffi.org: On 30/07/2015 02:29, Sam Whited wrote: HTTP give no advantage over Socks5, and doesn't do NAT traversal as Jingle can do, and it's an other whole different server to maintain. When uploading a file to a server you don't need NAT traversal. Anyway, if the HTTP XEP is published I'll implement it anyway, I just hope it will not have bad side effects on jingle adoption I get the feeling that this is what a lot of people are afraid of. I said this before and I can't stress this enough. HTTP File Upload is not meant to replace Jingle FT (and _especially_not_ Jingle since Jingle is so much more than Jingle FT). I spend a lof of time in the last couple of days reading Jingle related XEPs. It is in fact a great stack. And we will definitely need it Jingle in the near future in regards to Voip. (And once you got a proper jingle stack for voice. doing jingle ft is not a big step) One thing that would benefit Jingle FT much more than 'preventing' this XEP from getting accepted is to finally deprecate SI file transfer. (See the parallel thread started by Sam) cheers Daniel
Re: [Standards] Proposed XMPP Extension: HTTP File Upload
2015-07-29 17:06 GMT+02:00 Goffi go...@goffi.org: I don't see any problem in using SI or Jingle File Transfer on the server side, and you can use the requesting jid to check auth, which is far better than using a random URL which can be lost in the nature and setting a full HTTP server just for that. The thing is that jingle is a session negotiation protocol. It works very well when two p2p entities that don't know each other wont to work out the best available method to transfer a file between them. When uploading a file to a server you don't need to negotiate anything. If you were to write a component that would just offer a place to upload files all negotiations would end up with the same result: Upload file directly via socks5 to that component since this is the most efficient one. but nothing prevent XMPP libraries to implement file transfer only that library developer are also lazy... Don't get me wrong. There is definitely a place for Jingle and Jingle FT. I don't want to deprecate Jingle at all. But Jingle is not always the right tool for the job. cheers Daniel
Re: [Standards] Proposed XMPP Extension: HTTP File Upload
On 29 July 2015 at 16:36, Daniel Gultsch dan...@gultsch.de wrote: Don't get me wrong. There is definitely a place for Jingle and Jingle FT. I don't want to deprecate Jingle at all. But Jingle is not always the right tool for the job. I see your point, but I'd prefer to see client developers only have to implement one thing instead of several (and have to select which file transfer mechanism to use in different circumstances). Dave.
Re: [Standards] Proposed XMPP Extension: HTTP File Upload
On Wed, Jul 29, 2015 at 11:19 AM, Dave Cridland d...@cridland.net wrote: I see your point, but I'd prefer to see client developers only have to implement one thing instead of several (and have to select which file transfer mechanism to use in different circumstances). Client specs should pick the best spec for the job they're trying to do, and the audience of the client; while I'm normally all about having a single way to do something to reduce confusion, JFT and HTTP Upload are completely orthogonal proposals, and the fact that one already exists should not block work on the other. HTTP Upload should be used when you want a file to be made available somewhere to one or more people (maybe publically, maybe with authentication), it's a sharing action, whereas JFT should be used when you want a (potentially encrypted) file transfer to a single recepient that should never be stored on an intermediary machine, eg. the sending action. —Sam P.S. Aside: That being said, were I to write a client I'd almost certainly prefer to use HTTP Upload exclusively. While Jingle is very powerful, and is actually quite a nifty protocol, it spans many specs and is way to complicated for a lazy developer like me to want to implement, whereas HTTP file upload is almost trivial to implement. -- Sam Whited pub 4096R/54083AE104EA7AD3 https://blog.samwhited.com
Re: [Standards] Proposed XMPP Extension: HTTP File Upload
G'day On 27/07/2015 16:37, XMPP Extensions Editor wrote: Title: HTTP File Upload Abstract: This specification defines a protocol to request permissions from another entity to upload a file to a specific path on an HTTP server and at the same time receive a URL from which that file can later be downloaded again. due to their very nature of being peer-to-peer they don't work very well in scenarios where it is requeried to send a file to multiple receipients or multiple resources of the same receipient at once. They also don’t work alongside offline storage, MUC history and Message Archive Management (XEP-0313) [3]. I don't see any problem in using SI or Jingle File Transfer on the server side, and you can use the requesting jid to check auth, which is far better than using a random URL which can be lost in the nature and setting a full HTTP server just for that. If a public share is needed, outside of XMPP, nothing prevent a server or component to have an HTTP gateway. But I think XMPP software should stick to XMPP. The easy to implement requirement is based on the fact that HTTP libraries are available in most languages, but nothing prevent XMPP libraries to implement file transfer, making developers life better. So to sum up: I rather see an xmpp: uri than an http: one to share files with XMPP. Goffi
Re: [Standards] Proposed XMPP Extension: HTTP File Upload
Hello, As a follow up, as promised, here is what we have implemented and tested with Amazon S3: https://github.com/processone/ejabberd-saas-docs/blob/master/http-filetransfer/http-filetransfer.md I hope you will find some ideas could be valuable for Daniel's proposal. -- Mickaël Rémond http://www.process-one.net On 27 Jul 2015, at 19:03, Mickaël Rémond wrote: The general workflow is as follow: 1. Sender request an upload URL. XMPP server return an upload URL and a file ID. This is the step to ensure that the master key is not shared, as client will only receive a temporary URL, only valid for a given file (based on its MD5). The requirement for the MD5 could be discussed of course. At some point we needed it for S3 API upload delegation. 2. Sender upload the file and can share the file ID. Download URLs are typically not permanent. A storage backend could purge them or change them. The main reason of that HTTP upload / download protocol is to be asynchronous. We do not know when the receiver will receive the file notification (when back online) so we needed to cope with possibly changing file URL. Moreover, with a service like S3, the download URL is not permanent. Having a file ID allow us to decouple the file from the actual download URL. 3. The receiver ask the XMPP server and URL based on the file ID. It has two benefits: - The XMPP server can use the master key to generate a temporary URL. - If the server wants to do some XMPP level credentials checks it is possible (but not mandatory). - It protect against leaked URL having the file permanently available to any third-party. I think this should address a few of the concerned expressed here, by Sam, Rick and Evgeny. We may need to open to more straightforward cases, where the final download URL is constant, but we are willing to help.
Re: [Standards] Proposed XMPP Extension: HTTP File Upload
On 27.07.2015 17:03, Rick van Rein wrote: Hello, The HTTP upload mechanism hinges on the secrecy of the upload URL. Any client who receives it and accidentally publishes it can convey the document. I would like to point you to an alternative, namely MSRP. I am willing to look for the time to write this up in a new XEP- -- but only if this is considered sufficiently interesting as an alternative to this list. Happy to see MSRP XEPed. But I don't think that a XEP using MSRP will ever be an *alternative* to the HTTP File Upload XEP, which strives to be a simple and straightforward XEP that solves a problem with the most primitive but robust possible approach. Implementing a HTTP upload is also pretty easy: Most frameworks and platforms ship with a standard HTTP library. - Florian signature.asc Description: OpenPGP digital signature
Re: [Standards] Proposed XMPP Extension: HTTP File Upload
Hello, The HTTP upload mechanism hinges on the secrecy of the upload URL. Any client who receives it and accidentally publishes it can convey the document. I would like to point you to an alternative, namely MSRP. It is a straightforward protocol, similar in nature to HTTP but with a few desirable facilities extra: - mention both source and destination address, which may be of u...@domain.tld form - block-by-block uploads enable multiplexing streams - checksums on each block - the protocol is symmetric; either side can initiate a transfer - TLS may be used for encryption and server authentication and, possibly, client authentication Although I agree that HTTP is useful for resource sharing, it lacks the security facilities to separate independent downloaders which weakens the security model of this proposal in ways that MSRP does not. MSRP, as you may know, is the SIP answer to file sharing. I am willing to look for the time to write this up in a new XEP- -- but only if this is considered sufficiently interesting as an alternative to this list. -Rick
Re: [Standards] Proposed XMPP Extension: HTTP File Upload
Mon, 27 Jul 2015 17:03:25 +0200 Rick van Rein r...@openfortress.nl wrote: I would like to point you to an alternative, namely MSRP MSRP is scary as an atomic war :)
Re: [Standards] Proposed XMPP Extension: HTTP File Upload
Hello, I would like to point you to an alternative, namely MSRP MSRP is scary as an atomic war :) For my understanding, could you rephrase that in technical terms please? -Rick
Re: [Standards] Proposed XMPP Extension: HTTP File Upload
As mentioned off list, I have two concerns with the protocl as it stands: While the simplicity is laudable, the proposals use is extremely limited by the fact that headers can not be defined. I would like to see a methohd for passing down arbitrary headers which should be sent with the PUT request. This would allow a few different use cases: 1) Authenticated requests (eg. generate a 1 time use oauth token, tell the client to use an `Authentication: Bearer token` header) 2) Easily match signed requests; I did an example implementation where the server delegated storage to Amazon S3. The server would generate a pre-signed upload URL with a limited TTL, then the client would make a PUT to this URL. However, the headers did not match, so I had to modify the client to add a Content-Type header which was required by Amazon. Other headers might be required as well, and this would mean implementaitons which use services like this would only be compatible with certain clients 3) It would allow (in the simplest case) for sending down a nonce over XMPP which the client would send back up, thereby validating that the client is actually the one doing an upload (and the URL wasn't just guessed by an attacker). I don't think this would add a great deal of complexity to the protocol, but it does make it much more powerful. My second (and less pressing) concern is that the GET url must be known up front. One can imagine the case of a blob storage service which stores files by hash and does not want to maintain infrastructure to map hashes to filenames. They would not know the final GET url (eg. myservice.tld/somefilehash) until after the file had been uploaded. To fix this an optional to support IQ should be added. If the file name is not available, a random token or other means of looking up the file would be sent down instead of the get request (or it could always be sent down if your server supports it), then, after the client performs the PUT it could send an IQ requesting the GET url with that token as an identifier. This also does not add a lot of complexity to the XEP, but would require that XMPP server implementors (not storage providers) maintain a mapping of that random token to GET url's (these could also have a timeout so that they don't need to be stored very long). If this is optional and a server DOES always know the GET URL up front, they could ignore it and return a not-implemented error if the client sends this IQ. Best, Sam On Mon, Jul 27, 2015 at 9:37 AM, XMPP Extensions Editor edi...@xmpp.org wrote: The XMPP Extensions Editor has received a proposal for a new XEP. Title: HTTP File Upload Abstract: This specification defines a protocol to request permissions from another entity to upload a file to a specific path on an HTTP server and at the same time receive a URL from which that file can later be downloaded again. URL: http://xmpp.org/extensions/inbox/http-upload.html The XMPP Council will decide in the next two weeks whether to accept this proposal as an official XEP. -- Sam Whited pub 4096R/54083AE104EA7AD3 https://blog.samwhited.com
Re: [Standards] Proposed XMPP Extension: HTTP File Upload
Mon, 27 Jul 2015 17:20:16 +0200 Rick van Rein r...@openfortress.nl wrote: For my understanding, could you rephrase that in technical terms please? Technically, you can count letters in the MSRP RFC and in HTTP-Upload XEP :) And to be serious, how do you see MSRP implemented in XMPP clients? You need to get full MSRP stack working, where do you get that from? For python, lua, erlang, etc? Do you think I will write MSRP for ejabberd in Erlang? No thanks :)
Re: [Standards] Proposed XMPP Extension: HTTP File Upload
Mon, 27 Jul 2015 11:01:58 -0500 Sam Whited s...@samwhited.com wrote: 2) Easily match signed requests; I did an example implementation where the server delegated storage to Amazon S3. The server would generate a pre-signed upload URL with a limited TTL, then the client would make a PUT to this URL. However, the headers did not match, so I had to modify the client to add a Content-Type header which was required by Amazon. Other headers might be required as well, and this would mean implementaitons which use services like this would only be compatible with certain clients Do you have any ideas how HTTP headers can be incorporated into the XEP? Because, for example, Content-MD5 header (used in Amazon S3) is also required for a receiver to be known, but there is no interaction between a sender and a receiver defined in the XEP.
Re: [Standards] Proposed XMPP Extension: HTTP File Upload
On 27 July 2015 at 17:01, Sam Whited s...@samwhited.com wrote: As mentioned off list, I have two concerns with the protocl as it stands: While the simplicity is laudable, the proposals use is extremely limited by the fact that headers can not be defined. I would like to see a methohd for passing down arbitrary headers which should be sent with the PUT request. This would allow a few different use cases: 1) Authenticated requests (eg. generate a 1 time use oauth token, tell the client to use an `Authentication: Bearer token` header) 2) Easily match signed requests; I did an example implementation where the server delegated storage to Amazon S3. The server would generate a pre-signed upload URL with a limited TTL, then the client would make a PUT to this URL. However, the headers did not match, so I had to modify the client to add a Content-Type header which was required by Amazon. Other headers might be required as well, and this would mean implementaitons which use services like this would only be compatible with certain clients 3) It would allow (in the simplest case) for sending down a nonce over XMPP which the client would send back up, thereby validating that the client is actually the one doing an upload (and the URL wasn't just guessed by an attacker). I'll just quickly note that I don't see any security advantage to a token in a header (if we're always over HTTPS, which I assume we are if we care about this). The attacker guessing an unpredictable URL is no different to an attacker guessing an unpredictable auth token. Regards, Matthew
Re: [Standards] Proposed XMPP Extension: HTTP File Upload
On Mon, Jul 27, 2015 at 11:28 AM, Matthew Wild mwi...@gmail.com wrote: I'll just quickly note that I don't see any security advantage to a token in a header (if we're always over HTTPS, which I assume we are if we care about this). The attacker guessing an unpredictable URL is no different to an attacker guessing an unpredictable auth token. In my case this would be about delegating to external services that require auth; if it's a local file upload service which we control, then yes, a random URL is enough. —Sam -- Sam Whited pub 4096R/54083AE104EA7AD3 https://blog.samwhited.com
Re: [Standards] Proposed XMPP Extension: HTTP File Upload
Hello, Interesting talk, I would like to contribute a bit to the discussion. A bit of context: we (at ProcessOne) have been working on an HTTP transfert protocol that is in production for some of our ejabberd SaaS customers. The goal is to allow to use a third party backend for storage (like S3) and to let the XMPP server be just a helper to make that happen. That said, I am not saying it is perfect, but some of the pitfall we had to address may be useful to build upon Daniel's proposal if this makes sense. The issue we wanted to solve is generally that the server owner will be paying for the storage, so for obvious reasons the master key that grant access to the storage need to be on XMPP server. The general workflow is as follow: 1. Sender request an upload URL. XMPP server return an upload URL and a file ID. This is the step to ensure that the master key is not shared, as client will only receive a temporary URL, only valid for a given file (based on its MD5). The requirement for the MD5 could be discussed of course. At some point we needed it for S3 API upload delegation. 2. Sender upload the file and can share the file ID. Download URLs are typically not permanent. A storage backend could purge them or change them. The main reason of that HTTP upload / download protocol is to be asynchronous. We do not know when the receiver will receive the file notification (when back online) so we needed to cope with possibly changing file URL. Moreover, with a service like S3, the download URL is not permanent. Having a file ID allow us to decouple the file from the actual download URL. 3. The receiver ask the XMPP server and URL based on the file ID. It has two benefits: - The XMPP server can use the master key to generate a temporary URL. - If the server wants to do some XMPP level credentials checks it is possible (but not mandatory). - It protect against leaked URL having the file permanently available to any third-party. I think this should address a few of the concerned expressed here, by Sam, Rick and Evgeny. We may need to open to more straightforward cases, where the final download URL is constant, but we are willing to help. I do not think three steps is that complex, but, tell me :) The plan is that we publish the doc on what we have tomorrow on Github (We need to convert to Markdown). That will makes things clearer and maybe we can pick a few ideas from there for the XEP. I hope this helps, -- Mickaël Rémond http://www.process-one.net/ On Mon, Jul 27, 2015 at 9:37 AM, XMPP Extensions Editor edi...@xmpp.org wrote: The XMPP Extensions Editor has received a proposal for a new XEP. Title: HTTP File Upload Abstract: This specification defines a protocol to request permissions from another entity to upload a file to a specific path on an HTTP server and at the same time receive a URL from which that file can later be downloaded again. URL: http://xmpp.org/extensions/inbox/http-upload.html The XMPP Council will decide in the next two weeks whether to accept this proposal as an official XEP.
Re: [Standards] Proposed XMPP Extension: HTTP File Upload
On Mon, Jul 27, 2015 at 11:23 AM, Evgeny Khramtsov xramt...@gmail.com wrote: Do you have any ideas how HTTP headers can be incorporated into the XEP? Because, for example, Content-MD5 header (used in Amazon S3) is also required for a receiver to be known, but there is no interaction between a sender and a receiver defined in the XEP. For the sake of reduced complexity, I was thinking of something as simple as modifying the put/get elements like so (I'm just making the XML up off the top of my head, the actual schema might look different, I have no idea what best practices are): ``` put urlhttps://someuploadservice.tld/somefilename.ext/url headers header name=content-typeapplication/some-mime/header header name=x-some-services-auth-headersometoken/header /headers /put ``` and similarly for the GET. You're right, this would not solve all the problems (eg. no Content-MD5), but I think it will hit the majority of use cases without adding additional complexity. (slightly OT: But Amazon S3 uploads are working great for me after I added the content-type header to Conversations... I'm using it on my personal account right now to great effect) —Sam -- Sam Whited pub 4096R/54083AE104EA7AD3 https://blog.samwhited.com
