I don't know of rsync, but I implemented http range support in my http servers and it's great for http clients that support i. (like streaming html5 movies). They request the chunks they want. Still it's good to use .pipe() once the data is being sent to the client. If you must send large amounts of data from node to a remote client, use .pipe(). If there are other solutions (like serving from a cdn, torrent, rsync, etc..) consider that as well.
On Fri, Feb 24, 2012 at 12:21 PM, Dan North <tasta...@gmail.com> wrote: > ** > Firstly, top marks for being able to make any sense of my phone sending a > reply in mid-sentence! > > I was starting to say... for sending large amounts of data it's more > common to pull rather than push, in other words for the source of the data > to simply send an "it's available" message (for instance in a RESTful > system you might use a POST with a Location header to say where the data > is) and then the reading end decides what it wants and when. > > The receiving end will know if it has sufficient disk space etc. Also > there are established protocols for ensuring large amounts of data safely > end up in the right place, from simple RETR using a length to pick up an > interrupted transfer, to things like rsync that figure out the minimum data > segments to transfer in large, similar binary files. > > I would use these existing tools (checksums for partial transfers etc.) > rather than try to reinvent anything. > > Which brings me to ask: Is there an rsync implementation for node? (or > could the OP actually use rsync in this case?) > > Cheers, > Dan (on a BlackBerry with an itchy send button) > ------------------------------ > *From: * Tim Caswell <t...@creationix.com> > *Sender: * nodejs@googlegroups.com > *Date: *Fri, 24 Feb 2012 10:03:40 -0600 > *To: *<nodejs@googlegroups.com> > *ReplyTo: * nodejs@googlegroups.com > *Subject: *Re: [nodejs] Best practice for pushing lots of data over TCP > to a distant server > > Dan, push or pull only matters for who initiates the connection. The > reading too fast and buffering issue is 100% local. It's between the code > reading from the local disk and the code writing to the local kernel > buffer. But by monitoring when the local kernel buffer is full or not, you > can make sure to not send data faster than it's able to leave. > > On Thu, Feb 23, 2012 at 3:50 PM, Dan North <tasta...@gmail.com> wrote: > >> ** >> The "best practice" here is to prefer pull over push. The remote daemon >> will always knbow best r >> ------------------------------ >> *From: * Tim Caswell <t...@creationix.com> >> *Sender: * nodejs@googlegroups.com >> *Date: *Thu, 23 Feb 2012 09:00:30 -0600 >> *To: *<nodejs@googlegroups.com> >> *ReplyTo: * nodejs@googlegroups.com >> *Subject: *Re: [nodejs] Best practice for pushing lots of data over TCP >> to a distant server >> >> Tcp itself handles a lot of the connection issues, so it's fairly >> reliable. You can still get disconnects at this level and you'll need to >> write reconnect logic if you want to be that robust. As far as bandwidth >> throttling, use Stream.prototype.pipe which implements "backpressure". >> That is, it stops reading from the source when the destination is full and >> not writable. When the dest is ready for more data, it resumes the stream. >> This way you won't end up buffering all the data in memory in the likely >> case that the source can read data faster than the remote dest can receive >> it. >> >> On Wed, Feb 22, 2012 at 9:30 PM, Jules <ju...@jules.com.au> wrote: >> >>> Daemon A needs to push lots of data to Daemon B, which is located on >>> another continent. The bandwidth available is usually quite high, but >>> it's highly variable and might drop out entirely at times. >>> >>> Is there an accepted "best strategy" for implementing this in Node? >>> >>> Does A just write as fast as it can to the socket and hope for the >>> best? Should it detect errors and re-connect, resuming from the last >>> offset that B successfully received? >>> >>> Does a socket ever become non-writeable? i.e. Does daemon A need to >>> catch "socket not writeable" exceptions and throttle itself to suit >>> the available bandwidth? >>> >>> I imagine some articles have been written on this topic, but I haven't >>> found them yet. >>> >>> -- >>> Job Board: http://jobs.nodejs.org/ >>> Posting guidelines: >>> https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines >>> You received this message because you are subscribed to the Google >>> Groups "nodejs" group. >>> To post to this group, send email to nodejs@googlegroups.com >>> To unsubscribe from this group, send email to >>> nodejs+unsubscr...@googlegroups.com >>> For more options, visit this group at >>> http://groups.google.com/group/nodejs?hl=en?hl=en >>> >> >> -- >> Job Board: http://jobs.nodejs.org/ >> Posting guidelines: >> https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines >> You received this message because you are subscribed to the Google >> Groups "nodejs" group. >> To post to this group, send email to nodejs@googlegroups.com >> To unsubscribe from this group, send email to >> nodejs+unsubscr...@googlegroups.com >> For more options, visit this group at >> http://groups.google.com/group/nodejs?hl=en?hl=en >> >> -- >> Job Board: http://jobs.nodejs.org/ >> Posting guidelines: >> https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines >> You received this message because you are subscribed to the Google >> Groups "nodejs" group. >> To post to this group, send email to nodejs@googlegroups.com >> To unsubscribe from this group, send email to >> nodejs+unsubscr...@googlegroups.com >> For more options, visit this group at >> http://groups.google.com/group/nodejs?hl=en?hl=en >> > > -- > Job Board: http://jobs.nodejs.org/ > Posting guidelines: > https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines > You received this message because you are subscribed to the Google > Groups "nodejs" group. > To post to this group, send email to nodejs@googlegroups.com > To unsubscribe from this group, send email to > nodejs+unsubscr...@googlegroups.com > For more options, visit this group at > http://groups.google.com/group/nodejs?hl=en?hl=en > > -- > Job Board: http://jobs.nodejs.org/ > Posting guidelines: > https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines > You received this message because you are subscribed to the Google > Groups "nodejs" group. > To post to this group, send email to nodejs@googlegroups.com > To unsubscribe from this group, send email to > nodejs+unsubscr...@googlegroups.com > For more options, visit this group at > http://groups.google.com/group/nodejs?hl=en?hl=en > -- Job Board: http://jobs.nodejs.org/ Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines You received this message because you are subscribed to the Google Groups "nodejs" group. To post to this group, send email to nodejs@googlegroups.com To unsubscribe from this group, send email to nodejs+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/nodejs?hl=en?hl=en
