Re: which one is faster?
Am Mittwoch, 8. August 2012 schrieb Johannes Wiedersich: On 08/08/12 09:14, lina wrote: It's a bit big data to transfer, around 1.1 T, from one server to another server. I checked that rsync is faster than scp, but in my situations rsync has elapsed for 1 hour, I guess the network is also a problem, Here I wish to know are there some tools (better default) can use for fast transferring, regardless the security reason, my data is just some data, no need special security care. IIUC, the question is not just, which is the fastest tool. If you have network problems (ie. intermittent connections) or fear thereof, you need a fast *and* a reliable tool. Yes. It likely would also be a good idea to fix these network problems ;) I suggest you stick with rsync. IMHO it is the best tool for your task. With the -c option, eg. you could check, whether all files transferred correctly, without much demand on the network. I only use rsync for these kinds of stuff. Its just reliable. With BTRFS I will investigate btrfs send/receive, but in the first time I will – as recommended – make sure I run rsync -c after it. -- Martin 'Helios' Steigerwald - http://www.Lichtvoll.de GPG: 03B0 0D6C 0040 0710 4AFA B82F 991B EAAC A599 84C7 -- To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/201208101103.19086.mar...@lichtvoll.de
Re: which one is faster?
Am Mittwoch, 8. August 2012 schrieb Darac Marjal: On Wed, Aug 08, 2012 at 03:14:50PM +0800, lina wrote: Hi, It's a bit big data to transfer, around 1.1 T, from one server to another server. I checked that rsync is faster than scp, but in my situations rsync has elapsed for 1 hour, I guess the network is also a problem, Here I wish to know are there some tools (better default) can use for fast transferring, regardless the security reason, my data is just some data, no need special security care. In addition to the suggestions mentioned by other people, consider the compressability of your data. I don't believe it's possible to rsync -z and possibly --compress-level might some in handy as well. It should be more efficient than using compression on SSH level. -- Martin 'Helios' Steigerwald - http://www.Lichtvoll.de GPG: 03B0 0D6C 0040 0710 4AFA B82F 991B EAAC A599 84C7 -- To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/201208101105.26784.mar...@lichtvoll.de
Re: which one is faster?
On 08/10/2012 12:52 AM, Kelly Clowers wrote: On Thu, Aug 9, 2012 at 9:00 AM, linalina.lastn...@gmail.com wrote: On 9 Aug, 2012, at 23:05, Chris Bannistercbannis...@slingshot.co.nz wrote: On Wed, Aug 08, 2012 at 09:00:18PM +0800, lina wrote: I don't know the reliable of the connection between the two servers, I guess it's okay. But from my side, the wireless is not stable. I don't know how to let it stable. I mean, not login every 10~15 minutes. (btw, Is big wind affects the wireless signal, kinda of silly to ask, I prefer the fresh air outside, so move laptop outside) Wind won't affect the signal, but if you are outside you may be in a spot where the signal is weak. Can you see what the signal strength is? Wind won't affect the signal?! Thanks. Nope. The air itself absorbs some of the electromagnetic spectrum (which is why gamma ray and x ray telescopes are *all* in space). Movement of the air does not really affect it much though. Density changes in the air (from pressure or thermal differences) can affect EM radiation, which is one reason the big optical telescopes have computer controlled micro-adjustments or are in space (Hubble). However, that is visible light, which has a much shorter wavelength than radio. Being longer, radio is much less affected by density differences, and on the scale of wifi it is not worth thinking about. And wind is basically a non-effect even for light (unless it carries dust or snow or something, but that is a different matter). I will say that in a heavy Montana blizzard, satellite TV signal can fade. But that comes down to the amount of water (frozen) in the air, as water tends to be a pretty good absorber of EM radiation in general. Cheers, Kelly Clowers All of that is quite correct, but it neglects one parameter: if the RF connection depends on directional antennas, which it will for any reasonable distance--say 1/2 mile or more--then wind may become a significant effect if it causes the antennas to jiggle, or to point off-target sometimes. --doug, WA2SAY, retired RF engineer -- Blessed are the peacekeepers...for they shall be shot at from both sides. --A.M. Greeley -- To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/502531fc.3080...@optonline.net
Re: which one is faster?
On 8/10/2012 11:08 AM, Doug wrote: All of that is quite correct, but it neglects one parameter: if the RF connection depends on directional antennas, which it will for any reasonable distance--say 1/2 mile or more--then wind may become a significant effect if it causes the antennas to jiggle, or to point off-target sometimes. This isn't an issue of the technology, but of the knowledge/skill of the installer, the quality and rigidity of the mounting hardware, and the chosen mounting location. Mount the antennae to the top of 25ft tube steel poles of 4 diameter and they will move quite a bit in windy conditions, likely causing signal issues. Mount them to 25ft treated utility grade 10 diameter posts and you'll likely never have wind issues. Both cases assume the pole is properly sunk 6-10ft with 4-6 of concrete fill. If you mount to the side or top of a building structure wind will never be an issue. Assuming proper installation, modern wifi directional antenna designs are immune to wind. -- Stan -- To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/50253a08.1070...@hardwarefreak.com
Re: which one is faster?
On Wed, Aug 08, 2012 at 09:00:18PM +0800, lina wrote: I don't know the reliable of the connection between the two servers, I guess it's okay. But from my side, the wireless is not stable. I don't know how to let it stable. I mean, not login every 10~15 minutes. (btw, Is big wind affects the wireless signal, kinda of silly to ask, I prefer the fresh air outside, so move laptop outside) Wind won't affect the signal, but if you are outside you may be in a spot where the signal is weak. Can you see what the signal strength is? -- If you're not careful, the newspapers will have you hating the people who are being oppressed, and loving the people who are doing the oppressing. --- Malcolm X -- To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20120809150547.GK2687@tal
Re: which one is faster?
On 9 Aug, 2012, at 23:05, Chris Bannister cbannis...@slingshot.co.nz wrote: On Wed, Aug 08, 2012 at 09:00:18PM +0800, lina wrote: I don't know the reliable of the connection between the two servers, I guess it's okay. But from my side, the wireless is not stable. I don't know how to let it stable. I mean, not login every 10~15 minutes. (btw, Is big wind affects the wireless signal, kinda of silly to ask, I prefer the fresh air outside, so move laptop outside) Wind won't affect the signal, but if you are outside you may be in a spot where the signal is weak. Can you see what the signal strength is? Wind won't affect the signal?! Thanks. I don't know how to check the signal strength except seeing the icon of the network manager. -- If you're not careful, the newspapers will have you hating the people who are being oppressed, and loving the people who are doing the oppressing. --- Malcolm X -- To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20120809150547.GK2687@tal -- To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/d9eaa49a-3915-4dea-8565-8ef3850fd...@gmail.com
Re: which one is faster?
On Fri, Aug 10, 2012 at 12:00:12AM +0800, lina wrote: I don't know how to check the signal strength except seeing the icon of the network manager. Sorry, I don't use network manager, but you could check by going inside to test. The reason the transfer speeds could be slow, is because of a flaky connection where it has to do retransmits every so often. Compare speeds of large file transfer, if same inside as outside, then at least you can sit outside and try the other suggestions from this thread. :) -- If you're not careful, the newspapers will have you hating the people who are being oppressed, and loving the people who are doing the oppressing. --- Malcolm X -- To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20120809175158.GA5072@tal
Re: which one is faster?
On Thu, Aug 9, 2012 at 9:00 AM, lina lina.lastn...@gmail.com wrote: On 9 Aug, 2012, at 23:05, Chris Bannister cbannis...@slingshot.co.nz wrote: On Wed, Aug 08, 2012 at 09:00:18PM +0800, lina wrote: I don't know the reliable of the connection between the two servers, I guess it's okay. But from my side, the wireless is not stable. I don't know how to let it stable. I mean, not login every 10~15 minutes. (btw, Is big wind affects the wireless signal, kinda of silly to ask, I prefer the fresh air outside, so move laptop outside) Wind won't affect the signal, but if you are outside you may be in a spot where the signal is weak. Can you see what the signal strength is? Wind won't affect the signal?! Thanks. Nope. The air itself absorbs some of the electromagnetic spectrum (which is why gamma ray and x ray telescopes are *all* in space). Movement of the air does not really affect it much though. Density changes in the air (from pressure or thermal differences) can affect EM radiation, which is one reason the big optical telescopes have computer controlled micro-adjustments or are in space (Hubble). However, that is visible light, which has a much shorter wavelength than radio. Being longer, radio is much less affected by density differences, and on the scale of wifi it is not worth thinking about. And wind is basically a non-effect even for light (unless it carries dust or snow or something, but that is a different matter). I will say that in a heavy Montana blizzard, satellite TV signal can fade. But that comes down to the amount of water (frozen) in the air, as water tends to be a pretty good absorber of EM radiation in general. Cheers, Kelly Clowers -- To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/CAFoWM=94wkdle9nebo9zke4n2t5y5ujj17pfn460vmpvg9u...@mail.gmail.com
Re: which one is faster?
2012/8/8 lina lina.lastn...@gmail.com Hi, It's a bit big data to transfer, around 1.1 T, from one server to another server. I checked that rsync is faster than scp, but in my situations rsync has elapsed for 1 hour, I guess the network is also a problem split (man split) your data into little chuncks then and/or use ftp -r -- *L'unica speranza di catarsi, ammesso che ne esista una, resta affidata all'istinto di ribellione, alla rivolta non isterilita in progetti, alla protesta violenta e viscerale. (V. Evangelisti) *
Re: which one is faster?
lina: It's a bit big data to transfer, around 1.1 T, from one server to another server. Either use rsync without encryption (= not tunneled over SSH), or pipe tar through netcat. The latter does not support resuming. If you can tell us a bit more, we might be able to help better. Do you have many small files or just a fewer big files? How fast and reliable is the connection between the two computers? J. -- When driving at night I find the headlights of oncoming vehicles very attractive. [Agree] [Disagree] http://www.slowlydownward.com/NODATA/data_enter2.html signature.asc Description: Digital signature
Re: which one is faster?
On Wed, Aug 8, 2012 at 12:14 AM, lina lina.lastn...@gmail.com wrote: Hi, It's a bit big data to transfer, around 1.1 T, from one server to another server. I checked that rsync is faster than scp, Well, sometimes. Certainly if you have some of the data in both places and need to sync it, rsync will just do the delta. But if the destination does not have any of the data yet, *at best* it will be similar. And rsync, depending on version and settings, will do things like make a full file list and/or checksums up front. That can be good, but it can also take along time on large data. You should carefully check out what options you might want or *not* want: --append, --inplace, --size-only, -c, -z, etc And since rsync typically goes over ssh, the encryption adds to the overhead as well. Or just use scp or ftp or something in those cases. Cheers, Kelly Clowers -- To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/CAFoWM=8nvwdbx1rzfabfnt3pooa0vtp79wi+tyd-i2ewnev...@mail.gmail.com
Re: which one is faster?
Hallo Lina, It's a bit big data to transfer, around 1.1 T, from one server to another server. You can also use netcat (man nc - see under examples). It is probably the fastest method. I have used that in the past in combination with rsync: copy it using netcat and check the result with rsync. Regards Johann -- Johann SpiesTelefoon: 021-808 4699 Databestuurder / Data manager Sentrum vir Navorsing oor Evaluasie, Wetenskap en Tegnologie Centre for Research on Evaluation, Science and Technology Universiteit Stellenbosch. Ye lust, and have not: ye kill, and desire to have, and cannot obtain: ye fight and war, yet ye have not, because ye ask not.James 4:2 E-pos vrywaringsklousule Hierdie e-pos mag vertroulike inligting bevat en mag regtens geprivilegeerd wees en is slegs bedoel vir die persoon aan wie dit geadresseer is. Indien u nie die bedoelde ontvanger is nie, word u hiermee in kennis gestel dat u hierdie dokument geensins mag gebruik, versprei of kopieer nie. Stel ook asseblief die sender onmiddellik per telefoon in kennis en vee die e-pos uit. Die Universiteit aanvaar nie aanspreeklikheid vir enige skade, verlies of uitgawe wat voortspruit uit hierdie e-pos en/of die oopmaak van enige lês aangeheg by hierdie e-pos nie. E-mail disclaimer This e-mail may contain confidential information and may be legally privileged and is intended only for the person to whom it is addressed. If you are not the intended recipient, you are notified that you may not use, distribute or copy this document in any manner whatsoever. Kindly also notify the sender immediately by telephone, and delete the e-mail. The University does not accept liability for any damage, loss or expense arising from this e-mail and/or accessing any files attached to this e-mail. -- To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20120808075852.ga21...@sun.ac.za
RE: which one is faster?
I user rsync for everything. rsync -vaP /location/ user@remote-host:/location/ I've found it best sofar. -Original Message- From: Johann Spies [mailto:jsp...@sun.ac.za] Sent: Wednesday, August 08, 2012 9:59 AM To: debian-user@lists.debian.org Subject: Re: which one is faster? Hallo Lina, It's a bit big data to transfer, around 1.1 T, from one server to another server. You can also use netcat (man nc - see under examples). It is probably the fastest method. I have used that in the past in combination with rsync: copy it using netcat and check the result with rsync. Regards Johann -- Johann SpiesTelefoon: 021-808 4699 Databestuurder / Data manager Sentrum vir Navorsing oor Evaluasie, Wetenskap en Tegnologie Centre for Research on Evaluation, Science and Technology Universiteit Stellenbosch. Ye lust, and have not: ye kill, and desire to have, and cannot obtain: ye fight and war, yet ye have not, because ye ask not.James 4:2 E-pos vrywaringsklousule Hierdie e-pos mag vertroulike inligting bevat en mag regtens geprivilegeerd wees en is slegs bedoel vir die persoon aan wie dit geadresseer is. Indien u nie die bedoelde ontvanger is nie, word u hiermee in kennis gestel dat u hierdie dokument geensins mag gebruik, versprei of kopieer nie. Stel ook asseblief die sender onmiddellik per telefoon in kennis en vee die e-pos uit. Die Universiteit aanvaar nie aanspreeklikheid vir enige skade, verlies of uitgawe wat voortspruit uit hierdie e-pos en/of die oopmaak van enige l??s aangeheg by hierdie e-pos nie. E-mail disclaimer This e-mail may contain confidential information and may be legally privileged and is intended only for the person to whom it is addressed. If you are not the intended recipient, you are notified that you may not use, distribute or copy this document in any manner whatsoever. Kindly also notify the sender immediately by telephone, and delete the e-mail. The University does not accept liability for any damage, loss or expense arising from this e-mail and/or accessing any files attached to this e-mail. -- To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20120808075852.ga21...@sun.ac.za Ezen üzenet és annak bármely csatolt anyaga bizalmas, jogi védelem alatt áll, a nyilvános közléstől védett. Az üzenetet kizárólag a címzett, illetve az általa meghatalmazottak használhatják fel. Ha Ön nem az üzenet címzettje, úgy kérjük, hogy telefonon, vagy e-mail-ben értesítse erről az üzenet küldőjét és törölje az üzenetet, valamint annak összes csatolt mellékletét a rendszeréből. Ha Ön nem az üzenet címzettje, abban az esetben tilos az üzenetet vagy annak bármely csatolt mellékletét lemásolnia, elmentenie, az üzenet tartalmát bárkivel közölnie vagy azzal visszaélnie. This message and any attachment are confidential and are legally privileged. It is intended solely for the use of the individual or entity to whom it is addressed and others authorised to receive it. If you are not the intended recipient, please phone or email the sender and delete this message and any attachment of it from your system. Please note that any dissemination, distribution, copying or use of or reliance upon the information contained in and transmitted by this e-mail or to anyone other than the recipient designated above by the sender is unauthorised and strictly prohibited. -- To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/7af9f7955a605547ab5bcd2b303f4b7d01024248d...@budakzpmbx02.mak.allamkincstar.gov.hu
Re: which one is faster?
On 08/08/12 09:14, lina wrote: It's a bit big data to transfer, around 1.1 T, from one server to another server. I checked that rsync is faster than scp, but in my situations rsync has elapsed for 1 hour, I guess the network is also a problem, Here I wish to know are there some tools (better default) can use for fast transferring, regardless the security reason, my data is just some data, no need special security care. IIUC, the question is not just, which is the fastest tool. If you have network problems (ie. intermittent connections) or fear thereof, you need a fast *and* a reliable tool. I suggest you stick with rsync. IMHO it is the best tool for your task. With the -c option, eg. you could check, whether all files transferred correctly, without much demand on the network. Cheers, Johannes -- To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/502226a7.9030...@aktendiener.de
Re: which one is faster?
On Wed, Aug 08, 2012 at 03:14:50PM +0800, lina wrote: Hi, It's a bit big data to transfer, around 1.1 T, from one server to another server. I checked that rsync is faster than scp, but in my situations rsync has elapsed for 1 hour, I guess the network is also a problem, Here I wish to know are there some tools (better default) can use for fast transferring, regardless the security reason, my data is just some data, no need special security care. In addition to the suggestions mentioned by other people, consider the compressability of your data. I don't believe it's possible to definitely predict the trade-offs here (it depends on how well the data compresses, basically, but also on how your CPUs compare to the network bandwidth, but you may find that spending some time compressing the data reduces the overall time. In that case try, the following. On the receiver: $ nc -l -p 12345 | $COMP -d | pv outfile and on the sender: $ pv infile | $COMP | nc receiver 12345 where $COMP is your preferred streaming compressor (gzip, bzip2 and xz should all work nicely here). By the way, pv (package: pv) is a useful pipeline-viewer and will show you progress (on the sending side) as well as throughput levels. signature.asc Description: Digital signature
Re: which one is faster?
On Wed, Aug 8, 2012 at 3:35 PM, Jochen Spieker m...@well-adjusted.de wrote: lina: It's a bit big data to transfer, around 1.1 T, from one server to another server. Either use rsync without encryption (= not tunneled over SSH), or pipe tar through netcat. The latter does not support resuming. Thanks. I failed to find the mute the encryption options in rsync. BTW, How to set the port for netcat? The remote one has the following ports open: Not shown: 987 closed ports PORT STATE SERVICE 21/tcpopen ftp 22/tcpopen ssh 80/tcpopen http 111/tcp open rpcbind 711/tcp open cisco-tdp 1027/tcp open IIS 1029/tcp open ms-lsa 3001/tcp open nessus 3389/tcp open ms-wbt-server 8649/tcp open unknown 8651/tcp open unknown 8652/tcp open unknown 15004/tcp open unknown $ nc badapple.net 21 220 (vsFTPd 2.0.5) it chocked there. another one has: Not shown: 995 filtered ports PORT STATE SERVICE 80/tcp open http 443/tcp open https 3128/tcp open squid-http 8080/tcp open http-proxy /tcp open sun-answerbook If you can tell us a bit more, we might be able to help better. Do you have many small files or just a fewer big files? How fast and reliable is the connection between the two computers? It's 8000 files, half 400M, half 5M. I don't know the reliable of the connection between the two servers, I guess it's okay. But from my side, the wireless is not stable. I don't know how to let it stable. I mean, not login every 10~15 minutes. (btw, Is big wind affects the wireless signal, kinda of silly to ask, I prefer the fresh air outside, so move laptop outside) J. -- When driving at night I find the headlights of oncoming vehicles very attractive. [Agree] [Disagree] http://www.slowlydownward.com/NODATA/data_enter2.html -- To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/cag9cjmnajxhb9mcpbdums5oftynhyowiqxphujhkhq1ptmo...@mail.gmail.com
Re: which one is faster?
On Wed, Aug 8, 2012 at 3:58 PM, Johann Spies jsp...@sun.ac.za wrote: Hallo Lina, It's a bit big data to transfer, around 1.1 T, from one server to another server. You can also use netcat (man nc - see under examples). It is probably the fastest method. I have used that in the past in combination with rsync: copy it using netcat and check the result with rsync. Thanks, this is smart. but I still have not figured it out how to use nc. I will try, Best regards, Regards Johann -- Johann SpiesTelefoon: 021-808 4699 Databestuurder / Data manager Sentrum vir Navorsing oor Evaluasie, Wetenskap en Tegnologie Centre for Research on Evaluation, Science and Technology Universiteit Stellenbosch. Ye lust, and have not: ye kill, and desire to have, and cannot obtain: ye fight and war, yet ye have not, because ye ask not.James 4:2 E-pos vrywaringsklousule Hierdie e-pos mag vertroulike inligting bevat en mag regtens geprivilegeerd wees en is slegs bedoel vir die persoon aan wie dit geadresseer is. Indien u nie die bedoelde ontvanger is nie, word u hiermee in kennis gestel dat u hierdie dokument geensins mag gebruik, versprei of kopieer nie. Stel ook asseblief die sender onmiddellik per telefoon in kennis en vee die e-pos uit. Die Universiteit aanvaar nie aanspreeklikheid vir enige skade, verlies of uitgawe wat voortspruit uit hierdie e-pos en/of die oopmaak van enige lês aangeheg by hierdie e-pos nie. E-mail disclaimer This e-mail may contain confidential information and may be legally privileged and is intended only for the person to whom it is addressed. If you are not the intended recipient, you are notified that you may not use, distribute or copy this document in any manner whatsoever. Kindly also notify the sender immediately by telephone, and delete the e-mail. The University does not accept liability for any damage, loss or expense arising from this e-mail and/or accessing any files attached to this e-mail. -- To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20120808075852.ga21...@sun.ac.za -- To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/cag9cjmkoe5gtf98wpjkg60gaasa5wfblj4k9xukb2qpw_we...@mail.gmail.com
Re: which one is faster?
On Wed, Aug 8, 2012 at 4:43 PM, Johannes Wiedersich deb...@aktendiener.de wrote: On 08/08/12 09:14, lina wrote: It's a bit big data to transfer, around 1.1 T, from one server to another server. I checked that rsync is faster than scp, but in my situations rsync has elapsed for 1 hour, I guess the network is also a problem, Here I wish to know are there some tools (better default) can use for fast transferring, regardless the security reason, my data is just some data, no need special security care. IIUC, the question is not just, which is the fastest tool. If you have network problems (ie. intermittent connections) or fear thereof, you need a fast *and* a reliable tool. I suggest you stick with rsync. IMHO it is the best tool for your task. With the -c option, eg. you could check, whether all files transferred correctly, without much demand on the network. Thanks, at present an email had been sent to the administrator to hopefully get 2TB space for data handling. /dev/gpfs1117T 43T 74T 37% /scratch /dev/gpfs3 30T 74G 30T 1% /userbackup Seems lots of free space, hope won't be refused. But here I still wish to hear the suggestions, very nice, at least I started to know the nc now, Best regards, Cheers, Johannes -- To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/502226a7.9030...@aktendiener.de -- To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/CAG9cJmk63_YLsjXZWyZMnrQ=V8VOV4OaHn6=zna9ruzfgaf...@mail.gmail.com
Re: which one is faster?
On Wed, Aug 8, 2012 at 8:10 PM, Darac Marjal mailingl...@darac.org.uk wrote: On Wed, Aug 08, 2012 at 03:14:50PM +0800, lina wrote: Hi, It's a bit big data to transfer, around 1.1 T, from one server to another server. I checked that rsync is faster than scp, but in my situations rsync has elapsed for 1 hour, I guess the network is also a problem, Here I wish to know are there some tools (better default) can use for fast transferring, regardless the security reason, my data is just some data, no need special security care. In addition to the suggestions mentioned by other people, consider the compressability of your data. I don't believe it's possible to definitely predict the trade-offs here (it depends on how well the data compresses, basically, but also on how your CPUs compare to the network bandwidth, but you may find that spending some time compressing the data reduces the overall time. In that case try, the following. On the receiver: $ nc -l -p 12345 | $COMP -d | pv outfile and on the sender: $ pv infile | $COMP | nc receiver 12345 where $COMP is your preferred streaming compressor (gzip, bzip2 and xz should all work nicely here). By the way, pv (package: pv) is a useful pipeline-viewer and will show you progress (on the sending side) as well as throughput levels. Thank you, neither receiver nor sender has the pv installed. I will spend sometime to figure the nc out first. Best regards, -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (GNU/Linux) iQIcBAEBCgAGBQJQIlc9AAoJEKB7YbRsd8TGRkkP/iQLhUKprzjOhVAPv8A4rRoI QYktoGPfRedswG4ivs0EaUVySUuK4sjmFuSj84vLh3WRLmsQUtDpiFbOUnlJS1Uj BrtKGEe73uMZQHrO6l/HtnOvUj/X5hxKPLI5ZFId3AvDIiiDOtZxVbLgjVEfrFnV LeV5YKVwks0O9uGxTOwUItXjSF2LUG2W96gKDqVAOvRLSKweG6mKtWALjDs4QYTw PPfpuvXxdEr0CscqUehph2iY/27MwPW6yNbC91o0uxRAj6/fcqnnJr3bR9+Vyxxk HtP6G+niroSoF8EFV4ETtZQXubIZuMkp7FhaxRjynLkVFbskJ4pRIXgL789qOX38 LJxHhPDNpn3F009rYaawz7t1ySv+sobicbI38hEQynCv9hZ9BiX0nIriTB2DXJ/A 8RLkpKGSlUXbbCA8UJ35wBwVBdaVEhLGACOXsE3zLbjcjsEDRtiHJIwdeHfpJbK4 xV1seAxkgKljPvTf+k+GEuRyh60CIduYx9yLG5icUV/HFVz/21azpqELBH2kWdpq WKREKD5ymmzofFLQtS8QI8I5UH6ZerHfptj/r5E5ff4BcFNmQ7ys7q52vkV3HPeI QBJ3GMuey/iNruQJXs7xU6P+BdXq+TyG9XK0LF4/4nk6mPSMot/Vqc5ShWsmeYkW tHddx04MGvI0OHc04vOo =a8qg -END PGP SIGNATURE- -- To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/CAG9cJm=cpg0uxfu38atmhkt7ku-5rfkvfwbgvb4wsj0h45z...@mail.gmail.com
Re: which one is faster?
On Wed, Aug 08, 2012 at 09:00:18PM +0800, lina wrote: BTW, How to set the port for netcat? The remote one has the following ports open: Choose one not from that list above 1024 and make sure any firewalls between the two computers allow that. See the following examples: http://www.screenage.de/blog/2007/12/30/using-netcat-and-tar-for-network-file-transfer/ http://g33kinfo.com/info/archives/1713 http://www.g-loaded.eu/2006/11/06/netcat-a-couple-of-useful-examples/ Regards Johann -- Johann SpiesTelefoon: 021-808 4699 Databestuurder / Data manager Sentrum vir Navorsing oor Evaluasie, Wetenskap en Tegnologie Centre for Research on Evaluation, Science and Technology Universiteit Stellenbosch. Ye lust, and have not: ye kill, and desire to have, and cannot obtain: ye fight and war, yet ye have not, because ye ask not.James 4:2 E-pos vrywaringsklousule Hierdie e-pos mag vertroulike inligting bevat en mag regtens geprivilegeerd wees en is slegs bedoel vir die persoon aan wie dit geadresseer is. Indien u nie die bedoelde ontvanger is nie, word u hiermee in kennis gestel dat u hierdie dokument geensins mag gebruik, versprei of kopieer nie. Stel ook asseblief die sender onmiddellik per telefoon in kennis en vee die e-pos uit. Die Universiteit aanvaar nie aanspreeklikheid vir enige skade, verlies of uitgawe wat voortspruit uit hierdie e-pos en/of die oopmaak van enige lês aangeheg by hierdie e-pos nie. E-mail disclaimer This e-mail may contain confidential information and may be legally privileged and is intended only for the person to whom it is addressed. If you are not the intended recipient, you are notified that you may not use, distribute or copy this document in any manner whatsoever. Kindly also notify the sender immediately by telephone, and delete the e-mail. The University does not accept liability for any damage, loss or expense arising from this e-mail and/or accessing any files attached to this e-mail. -- To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20120808142242.gc11...@sun.ac.za
Re: which one is faster?
On Wed, 08 Aug 2012 15:14:50 +0800, lina wrote: It's a bit big data to transfer, around 1.1 T, from one server to another server. Are both hosts remote (over Internet) or local (LAN)? I checked that rsync is faster than scp, but in my situations rsync has elapsed for 1 hour, I guess the network is also a problem, That can be because rsync performs better for small chunks of data instead bigger ones. Here I wish to know are there some tools (better default) can use for fast transferring, regardless the security reason, my data is just some data, no need special security care. I would run a bunch of tests to transfer one file (~1 GiB) with different tools (scp, ftp, http, rsync...) and choose whichever gives the fastest results :-) Note: adding a secure channel (ftps/sftp/scp/https...) will delay the transfer. Greetings, -- Camaleón -- To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/jvttfc$kgo$8...@dough.gmane.org
