It's not a size thing, it's an "existence" thing. In the case of client-to-client, if I'm running a Jabber server at Exodus say, I pay for pretty much 0 bandwidth for that transfer since that cost is paid by the clients ISP's (which may or may not be me, most likely not). If the server is involved, I pay for 2x the size of the file + whatever chunking or retransmission overhead + temporary (how long) storage and resource load. Now, *MAYBE* an answer could be that instead of HTTP as our "random" protocol, we actually use a simple Jabber server on the client for client-to-client, and that therefore if it happens to be hosted by the "real" Jabber server, everything just works the same. Maybe this is what others have suggested? But even then, I'd say we want a "parallel-band" method rather than in-band, and I don't think it solves the H.323 and other protocol problems...
-----Original Message----- From: Michael F Lin [mailto:[EMAIL PROTECTED]] Sent: Friday, June 07, 2002 1:46 PM To: [EMAIL PROTECTED] Subject: RE: [JDEV] File transfers Max, I understand the argument about bandwidth costs if we were talking about encoding all binary data in base64, increasing its size by 33%. However, this does not need to be the case, as XIB and SOAP Attachments (well, sorta) show. The model we envision for deployment of Jabber is just like sendmail, with servers being deployed at the site level. This is the model on which sendmail defeated propriety e-mail providers (online services), and I propose it is the model upon which Jabber can defeat proprietary IM providers. In this case there isn't much additional bandwidth usage that you should have to pay for. Presumably, you pay for your backbone connection, and the same amount of data is going to the external Internet as would be the case in a P2P transfer to some external Internet node. I'll grant there is some argument about increasing bandwidth usage internally within your NOC (maybe over a culmulative 20 feet of cable), but if you're not running at saturation this shouldn't cost you anything, and if it does it's a one-time cost. I am saying that file transfers and binary data of unextreme size or bitrate belong in band. I don't have a concrete definition of "unextreme", but I think we can come up with some examples: Unextreme (in-band | parallel-band): Office-type documents Source files PDFs Pictures Streaming audio (maybe) Largish XML data islands which the server should not have to parse, load into a DOM, etc. Extreme for the forseeable future: CD images DivX Extreme for the near future, but probably not for terribly long: MP3s Streaming video The point is that it should not be necessary to jump through the absurd hoops of negotiating firewalls with these stateful proxy servers and talking and/or understanding a foreign protocol in order to transfer a slightly largish chunk of data which does may or may not happen to be XML. I am not saying that nothing should be done out of band. I am saying it is feasible and very worthwhile to do certain things (which are more than what we are doing now, and which will grow as time goes on) in-band. I will agree that current Jabber technology is not suitable for this. We would like to see a simple and ruthlessly efficient binary wire protocol which uses XML exactly where XML is advantageous in JNG, and finally acknowledges that the advantages of using XML as a wire protocol are not the same as the advantages of using XML as a document format. Whether you agree with all the large payload stuff or not, JNG needs a better wire protocol, and I think everybody knows it. The point is that once you adopt such a protocol (which allows the possibility of kernel-mode caching, zero-copy streaming, page-aligned buffers, and other performance tricks), not only do we get huge performance and scalability benefits, but transporting large binary payloads becomes completely feasible - and it's all almost as easy as what we're doing now. But more on that later. -Mike |---------+----------------------------> | | Max Metral | | | <Max.Metral@peopl| | | epchq.com> | | | Sent by: | | | jdev-admin@jabber| | | .org | | | | | | | | | 06/07/2002 12:17 | | | PM | | | Please respond to| | | jdev | | | | |---------+----------------------------> >--------------------------------------------------------------------------- ---------------------------------------------------| | | | To: "'[EMAIL PROTECTED]'" <[EMAIL PROTECTED]> | | cc: | | Subject: RE: [JDEV] File transfers | | | | | >--------------------------------------------------------------------------- ---------------------------------------------------| Have you been looking at the fact that no service provider is going to be able to afford a system that does in-band binary transfer exclusively? Are you saying all file transfer and binary data (H.323) belong in band? Just trying to make sure I understand your suggestion. -----Original Message----- From: Michael F Lin [mailto:[EMAIL PROTECTED]] Sent: Friday, June 07, 2002 11:38 AM To: [EMAIL PROTECTED] Subject: Re: [JDEV] File transfers I would also add that encoding concerns could be alleviated by using something like Jer's XIB or SOAP attachments, which are extensions to XML streams that allows inline binary payloads. There are a few things we would do differently, but that's the general idea. Again, the key advantage is that all packets are still JID-routed, so we get through all firewalls and NATs without having to set up complicated stateful proxy servers in addition to what we already have. Really, the way to do this is to define a maximum _packet_ size, for all Jabber packets. This makes everything manageable from the server side. If you need to transfer a packet larger than that maximum size, either drop out of band and deal with firewall problems yourself, or chunk the message. Personally, I think the latter is preferable, because all these problems of negotiating through firewalls and finding PASS servers just go away. Unfortunately, it is not easy to define a maximum packet size because we don't have length-prefixed framing, so you never know the length of a packet until you've received the whole thing - by which time you (almost) might as well send it along anyway! This is why we need something like JEP-0017 or XATP. Guys, frankly, we've been looking at this problem in this lab for the last _year_ (some of you may remember me talking about it at JabberCon, over most of the same objections). We've done the research and the prototypes for high-performance Jabber-like messaging systems. I know the disadvantages seem obvious and the advantages seem hard to accept - but we are positive that high-performance, scalable in-band or parallel-band transfer of large payloads is both practically advantageous and technically feasible. We are preparing a detailed proposal on the matter, which we hope will be a major concrete step towards JNG. I hope we can soon put this silly argument to rest for now. -Mike |---------+------------------------------> | | "Michael Rothwell" | | | <rothwell@holly-spr| | | ings.nc.us> | | | Sent by: | | | [EMAIL PROTECTED]| | | rg | | | | | | | | | 06/06/2002 04:23 PM| | | Please respond to | | | jdev | | | | |---------+------------------------------> > --------------------------------------------------------------------------- ---------------------------------------------------| | | | To: <[EMAIL PROTECTED]> | | cc: | | Subject: Re: [JDEV] File transfers | | | | | > --------------------------------------------------------------------------- ---------------------------------------------------| The jabber server could simply have a config option for "max transfer size," which ,when set to zero, disables all file transfers. Clients are notified of the file-transfer capabilities of the server by the server. You could even take it one step further and allow/disallow file-transfter use per account ID and/or groups. Other config options which would be useful are "cache time" and "max cache size" -- if cache time and/or size are zero, then if the server cannot stream the data directly to the recipient(s), it informs the client that the recipient isn't accepting transfers. You could even limit the number of transfers/person/day, etc. Leave it up to the clients to B64 encode/decode the transfer. The server doesn't have to care, which will reduce processing load. The "max size" parameters will be for the encoded versions, which is what server operators care about, because it's what they pay for. Server-supported transfers would be a nice config option, esp. for behind-the-firewall and small-number-of-users servers. ----- Original Message ----- From: Gallo, Felix S. To: '[EMAIL PROTECTED]' Sent: Thursday, June 06, 2002 3:54 PM Subject: RE: [JDEV] File transfers Mike writes: > Firstly, there is no inherent problem with sending moderately > large files through a software server. Sendmail does it all > day, every day, on a massive scale, without relying on > client-to-client connections. However, most mail is accessed through POP, IMAP, or Exchange, which are definitely client-to-client connections -- for the simple reason that sendmail doesn't scale very well. For every byte send to a sendmail server, two bytes traverse the network. Most unfortunately, those two bytes always involve just the one sendmail server and its attached network. What positives do you get for having the sendmail server involved in a large file transaction between two parties? You get a guarantee of delivery and no need for continued storage on the sending party's side (since the file 'moves' to the sendmail server). If you're sending the file to multiple users at once, you get less traffic on your local network (the server takes the load). You also get an opportunity to mediate the file somehow (e.g., virus checking it as a service, converting it from aac into mp3, storing it for later delivery to other users..) What positives do you get for not having the sendmail server involved? The network on the sendmail server sees 0X load rather than 2X load; the latency is lower; the sendmail server has no storage requirement; and you have arguably fewer points of failure. Pragmatically, taking the load off the server is more valuable in the normal case than replicating HTTP/FTP/SMTP/FXP yet again. The fact that 'SMTP does it' is not a great rationale for forcing all the jabber servers to pay 2X bandwidth costs for file transfers between their users. F. ********************************************************************** E-mail sent through the Internet is not secure. Western Asset therefore recommends that you do not send any confidential or sensitive information to us via electronic mail, including social security numbers, account numbers, or personal identification numbers. Delivery, and or timely delivery of Internet mail is not guaranteed. Western Asset therefore recommends that you do not send time sensitive or action-oriented messages to us via electronic mail. ********************************************************************** _______________________________________________ jdev mailing list [EMAIL PROTECTED] http://mailman.jabber.org/listinfo/jdev _______________________________________________ jdev mailing list [EMAIL PROTECTED] http://mailman.jabber.org/listinfo/jdev _______________________________________________ jdev mailing list [EMAIL PROTECTED] http://mailman.jabber.org/listinfo/jdev _______________________________________________ jdev mailing list [EMAIL PROTECTED] http://mailman.jabber.org/listinfo/jdev
