On Mon, Oct 30, 2017 at 09:23:51PM +0200, Mihai Popescu wrote: > Hi, > > I am trying to setup a solution on an OpenBSD computer, where i want > to upload and then download large volume of data. I was using ftpd > daemon to do this, but I wonder if there is another way to do this, > regarding speed of transfer. >
Back in a former life, I often had to transfer terabytes of information between hosts in my network. The goal was to reduce the overhead of the transfer so that more of the data would get transferred. I had been using netcat/nc and pushing data through the connection as fast as I could. This worked great, provided I could verify that things were being transferred without errors. Any connection problems and I'd be screwed. Eventually, I found this web page which details another guy's research into the same problem: http://intermediatesql.com/linux/scrap-the-scp-how-to-copy-data-fast-using-pigz-and-nc/ Essentially, he's compressing files and then sending them over to fit more data down the pipe. I also learned of this other program called bbcp which was developed specifically for our purposes. It's mentioned in the above article, and that author wasn't big on it. I like it though; it's what I ended up using because to me it is a more elegant solution and worked well for my needs. http://www.slac.stanford.edu/~abh/bbcp/ Check those out, and I'm very curious if anyone else on the list has more OpenBSD-centric techniques! -- Put your Nose to the Grindstone! -- Amalgamated Plastic Surgeons and Toolmakers, Ltd.
