Re: Rsync flaws
Am 11.04.2011 13:08, schrieb Federico Alves: On 4/10/11 11:35 PM, Larry Brower larry-li...@maxqe.com wrote: On 04/11/2011 05:59 AM, Federico Alves wrote: I am using rsync to send almost 1 TB of sparse files across the LAN to another identical Linux box. If I fire only 1 command, I get about 22 Mb of speed, but if I fire 6 commands in parallel, from different SSH connections, the speed is divided by 6: very, very slow. My command is rsync -S --progress sparsefile jephe@server:newsparsefile The LAN is 1GB and both machines are Scientific Linux 6.0. Is there any way to do this that does not have a bottleneck? Federico This could be a limitation of your disks speed in the servers and not rsync itself The limitation is not my hardware. The servers are both Dell R900 with SAS disk arrays. Also, from a Windows virtual machine, inside the same server, I get around 400 MB speed using FTP transfer, windows to windows. There must be a different way to do this from Linux.The files are sparse files, and I need to keep them that way, that's why I use rsync. Maybe it helps to rule out SSH as the culprit. Can you try to set up a real rsync-server on that machine (package rsync-server)? Take a look at `man rsyncd.conf` for an example configuration. Here is a short walkthrough on how to setup one: http://www.jtanderson.org/linux/centos-5-rsync-server-setup/ (BTW: Why isn't there an init script and default config file in the rsync package?) Alternatively, you can replace ssh with rsh and call `rsync --rsh=rsh ...` If the problem is really rsync, you can try to stream tar over ssh. tar --create --sparse $fileA $fileB | ssh user@server tar --extract --directory $target_dir --preserve-permissions --sparse BTW: Please don't top-post. Put your reply below the message you are quoting. This makes reading long threads easier. Hope this helps, Florian Philipp signature.asc Description: OpenPGP digital signature
Re: Rsync flaws
Am 11.04.2011 10:23, schrieb Florian Philipp: Am 11.04.2011 13:08, schrieb Federico Alves: On 4/10/11 11:35 PM, Larry Brower larry-li...@maxqe.com wrote: On 04/11/2011 05:59 AM, Federico Alves wrote: I am using rsync to send almost 1 TB of sparse files across the LAN to another identical Linux box. If I fire only 1 command, I get about 22 Mb of speed, but if I fire 6 commands in parallel, from different SSH connections, the speed is divided by 6: very, very slow. My command is rsync -S --progress sparsefile jephe@server:newsparsefile The LAN is 1GB and both machines are Scientific Linux 6.0. Is there any way to do this that does not have a bottleneck? Federico This could be a limitation of your disks speed in the servers and not rsync itself The limitation is not my hardware. The servers are both Dell R900 with SAS disk arrays. Also, from a Windows virtual machine, inside the same server, I get around 400 MB speed using FTP transfer, windows to windows. There must be a different way to do this from Linux.The files are sparse files, and I need to keep them that way, that's why I use rsync. Maybe it helps to rule out SSH as the culprit. Can you try to set up a real rsync-server on that machine (package rsync-server)? Take a look at `man rsyncd.conf` for an example configuration. Here is a short walkthrough on how to setup one: http://www.jtanderson.org/linux/centos-5-rsync-server-setup/ (BTW: Why isn't there an init script and default config file in the rsync package?) Ignore that part about package rsync-server. I forgot to take that out. Thought there was a package like that. signature.asc Description: OpenPGP digital signature
Re: [SCIENTIFIC-LINUX-USERS] Rsync flaws
On Mon, Apr 11, 2011 at 4:32 AM, Matt Willsher m...@monki.org.uk wrote: On 11 April 2011 12:08, Federico Alves sa...@minixel.com wrote: The limitation is not my hardware. The servers are both Dell R900 with SAS disk arrays. Also, from a Windows virtual machine, inside the same server, I get around 400 MB speed using FTP transfer, windows to windows. There must be a different way to do this from Linux.The files are sparse files, and I need to keep them that way, that's why I use rsync. Have you tried rsync server on the remote side? I've always found transfers over SSH to be rather slower than I'd like although 22Mb is slower that I'd expect. It comes down to a process of ellimination so try and get SSH out of the equation and see if that helps. If not, check disk performance with iostat (part of the sysstat package) and make sure there isn't a problem with queues or disk utilisation there. Check the network for problems - try a different protocol and some dummy files, make sure there isn't packet loss via netstat. If this helps, you might also review your rsync setups. Sending lots of distinct rsync requests, and thus lots of newly established SSH setups, causes considerable startup overhead for each connection, especially if the machines are not set with valid reverse DNS. (The SSH server looks up the reverse DNS of the connecting client to log the hostname of the connection: this is only really disabled by using 'sshd -u0' in the init script, instead of 'sshd'.)
RPM dependency issue with unixODBC
Hello all, I am trying to package a MonaLisa sensor that comes with its own java binary distribution and would be installed in /usr/local. I can build the package fine but it does not install because of a dependency not satisfied : depcheck: package MLSensor 1.0-1 needs libodbc.so()(64bit) depcheck: package MLSensor 1.0-1 needs libodbcinst.so()(64bit) This is because rpmbuild have autocomputed the dependencies and the resulting RPM have these requirements : libodbc.so()(64bit) libodbcinst.so()(64bit) Unfortunately neither unixODBC nor unixODBC-devel provides these, instead they provide : libodbc.so.1()(64bit) libodbcinst.so.1()(64bit) So my RPM refuses to install (and I cannot use --nodeps, the installation is performed by a tool) = Is this due to the packaging of unixODBC ? Can it be corrected ? The java that comes natively with SL is apparently not compiled with ODBC so it does not have libodbc requirements. This issue is minor and not blocking for me and I am not too sure I have understood the problem. Thanks JM -- Jean-michel BARBET| Tel: +33 (0)2 51 85 84 86 Laboratoire SUBATECH Nantes France| Fax: +33 (0)2 51 85 84 79 CNRS-IN2P3/Ecole des Mines/Universite | E-Mail: bar...@subatech.in2p3.fr
Re: RPM dependency issue with unixODBC
2011/4/11 Jean-Michel Barbet jean-michel.bar...@subatech.in2p3.fr: Hello all, I am trying to package a MonaLisa sensor that comes with its own java binary distribution and would be installed in /usr/local. I can build the package fine but it does not install because of a dependency not satisfied : depcheck: package MLSensor 1.0-1 needs libodbc.so()(64bit) depcheck: package MLSensor 1.0-1 needs libodbcinst.so()(64bit) This is because rpmbuild have autocomputed the dependencies and the resulting RPM have these requirements : libodbc.so()(64bit) libodbcinst.so()(64bit) unixODBC-2.2.14-11.el6.i686 : A complete ODBC driver manager for Linux Repo: sl Matched from: Filename: /usr/lib/libodbc.so unixODBC-2.2.14-11.el6.x86_64 : A complete ODBC driver manager for Linux Repo: sl Matched from: Filename: /usr/lib64/libodbc.so and unixODBC-2.2.14-11.el6.i686 : A complete ODBC driver manager for Linux Repo: sl Matched from: Filename: /usr/lib/libodbcinst.so unixODBC-2.2.14-11.el6.x86_64 : A complete ODBC driver manager for Linux Repo: sl Matched from: Filename: /usr/lib64/libodbcinst.so are you using sl 5 or sl 6? -- Eero
Re: RPM dependency issue with unixODBC
Jean-Michel Barbet wrote: Hello all, I am trying to package a MonaLisa sensor that comes with its own java binary distribution and would be installed in /usr/local. I can build the package fine but it does not install because of a dependency not satisfied : depcheck: package MLSensor 1.0-1 needs libodbc.so()(64bit) depcheck: package MLSensor 1.0-1 needs libodbcinst.so()(64bit) This is because rpmbuild have autocomputed the dependencies and the resulting RPM have these requirements : libodbc.so()(64bit) libodbcinst.so()(64bit) Unfortunately neither unixODBC nor unixODBC-devel provides these, instead they provide : libodbc.so.1()(64bit) libodbcinst.so.1()(64bit) So my RPM refuses to install (and I cannot use --nodeps, the installation is performed by a tool) = Is this due to the packaging of unixODBC ? Can it be corrected ? The java that comes natively with SL is apparently not compiled with ODBC so it does not have libodbc requirements. This issue is minor and not blocking for me and I am not too sure I have understood the problem. Thanks JM Hello Jean-Michel, I suppose you already tried this inelegant workaround: ln -s libodbc.so.1 libodbc.so ln -s libodbcinst.so.1 libodbcinst.so Cheers, J-P
Re: RPM dependency issue with unixODBC
Jean-Pierre Froberger wrote: I suppose you already tried this inelegant workaround: ln -s libodbc.so.1 libodbc.so ln -s libodbcinst.so.1 libodbcinst.so Thank you Eero and JP, The issue is not with the files themselves. The link mentioned above exist. The issue is with the provides list of unixODBC which includes libodbc.so.1()(64bit) but not libodbc.so()(64bit). The RPM I build wants the latter. JM -- Jean-michel BARBET| Tel: +33 (0)2 51 85 84 86 Laboratoire SUBATECH Nantes France| Fax: +33 (0)2 51 85 84 79 CNRS-IN2P3/Ecole des Mines/Universite | E-Mail: bar...@subatech.in2p3.fr
Re: SL60 install on bios raid
On 04/08/2011 04:59 PM, Phil Schaffner wrote: Artem Trunov wrote on 04/07/2011 09:31 AM: ... So, what would be the right way to use kickstart install on such raid? Turn off the BIOS FakeRAID/HostRAID and use software RAID. Phil Agreed. But for fake raid or not, this is what I always do to setup partitioning in a kickstart file. Do the install by hand, partitioning it how I like it, then grab the partitioning information out of the kickstart file generation for you /root/anaconda-ks.cfg Troy -- __ Troy Dawson daw...@fnal.gov (630)840-6468 Fermilab ComputingDivision/SCF/FEF/SLSMS Group __
Re: Rsync flaws
On Mon, Apr 11, 2011 at 05:08, Federico Alves sa...@minixel.com wrote: The limitation is not my hardware. The servers are both Dell R900 with SAS disk arrays. Also, from a Windows virtual machine, inside the same server, I get around 400 MB speed using FTP transfer, windows to windows. There must be a different way to do this from Linux.The files are sparse files, and I need to keep them that way, that's why I use rsync. Well transferring sparse files is going to be slow and it could be hardware (unless you are somehow testing with windows of copying sparse files over). rsync is having to see what real bits are there and what is fluff so it is going to be CPU and disk intensive. -- Stephen J Smoogen. The core skill of innovators is error recovery, not failure avoidance. Randy Nelson, President of Pixar University. Let us be kind, one to another, for most of us are fighting a hard battle. -- Ian MacLaren
