Hello.
After reviewing the results of test, I've modified smb.conf. I've added
max protocol = SMB2 and removed SO_RCVBUF=8192 SO_SNDBUF=8192 from
socket options. The read speed from this server increased to 40MB/s, the
write speed to this server increased to 30MB/s.
On 19.12.2015 00:43, Dan Ritter wrote:
Do you have jumbo packet support on your switch and on all
possible clients? 5000 - 9000 byte ethernet packets can improve
raw throughput, at a (usually minor) cost in latency.
Yes, main switches are set for jumbo frames, but half of computers and
switches are old and can't handle jumbo frames. That's why servers are
not set for jumbo frames, only the switches are ready.
RAIDZ2 is not a high-speed solution, it's a medium-safety
solution. Layering ext4 on top of ZFS blocks doesn't make
it faster, it just extends the reliability of ZFS to the
ext4 fs.
Do you have a requirement for putting ext4 on top of ZFS blocks?
Can you use ZFS instead?
Yes, raidz2 is a compromise for us for performance-safety. I wanted to
allow up two 2 hdd to fail and be able to sustain the data. I've choosed
ext4 on zfs, because this is not only a backup server, but also a file
server, which stores files with ACL support in a Microsoft Active
Directory file access policy. Also I want to set up drdb to sync
partitions to another file server. If ZFS file system will allow this -
I would switch to it.
BUT, this server boots from an md mirror on two separate SSD's. Reading
and writing to this disks give same low speed.
On 19.12.2015 04:14, David Christensen wrote:
> What motherboard? What CPU? What and how much RAM? What disk/RAID?
Its a supermicro server based on SuperMicro X8DTi-F motherboard with 2 x
Intel Xeon E5620 CPU @ 2,4GHz with 18 GB ram.
> What version of Debian? Which kernel? Any other customizations?
Message title states Debian 7 which is Wheezy, Linux 3.2.0-4-amd64 #1
SMP Debian 3.2.68-1+deb7u6 x86_64 GNU/Linux. All updates installed. It's
a standard install with ZFS for software raid and file server.
> But, if the cables ran at 100 MB/s under Windows you're probably okay.
> What is your switch?
Same cable types, same server type on windows gives faster file
transfer. All servers is on gigabit switch.
> iperf should be testing the hardware, device drivers, and kernel on both
> ends. What make/ model Ethernet chips do you have on the two hosts?
>
> # lspci | grep -i ethernet
iperf between two Debian 7 servers gives a good performance almost. The
problem server has Intel 82576 dual gigabit.
On the other server it is Broadcom Corporation NetXtreme BCM5705
Gigabit. The othe server gives 80MB/s on read and 60MB/s on write. Still
not 100MB/s as on windows, but it is 1 cpu older server.
> Is there anything else between the two computers?
Only the switch.
> I assume you ran that on your Samba server (?).
Ofcourse.
> A stream of zeros is very easy to compress and can give misleading
> results. It's better to use random numbers. (A hardware RNG is
useful for this.)
Yes, zeros are easily compressed. I've tried to use /dev/random as
source, but this device give me less then 1KB/s. When I used
/dev/urandom, I've got 8,6MB/s and dd used 100% of one core.
> I assume you're running the latest ZFS on Linux? (ZFS-fuse is
> considerably slower and is behind on features.)
Yes, ZFS on Linux and all latest updates.
> Why did you have to -f (force) creation?
Don't know why I did used this option back then. I think it would be
working withou -f.
> LUKS? On the raw drives, or on partitions? How many CPU's and cores?
> HT? AES-NI?
I used raw drives 8 x 2TB and 8 x 1TB. 2 x CPU gives me 16 cores. This
is BIOS settings:
Ratio CMOS Setting: 18
C1E Support: Disabled
Hardware Prefetcher: Enabled
Adjacent Cache Line Prefetch: Enabled
DCU Prefetcher: Enabled
Data Reuse Optimisation: Enabled
MPS and ACPI MADT ordering: Modern ordering
Max CPUID Value Limit: Disabled
Intel(R) Virtualization Tech: Enabled
Execute-Disable Bit Capability: Enabled
Intel AES-NI: Disabled
Simultaneous Multi-Threading: Enabled
Active Processor Cores: All
Intel(R) EIST tech: Disabled
Intel(R) C-STATE tech: Disabled
Clock Spread Spectrum: Disabled
> Different-sized drives makes things more interesting. What arrangements
> did you try? What did you settle on, and why?
I've read a lot about zfs before setting up this server and used mostly
recommended ashift=12 and other settings to suit my needs.
> Why raidz2?
To allow up two 2 disk to fail.
> Turning off atime is okay if none of the applications that use the file
> systems need it.
Thats why it is off.
> General de-duplication can be risky, but for many backup solutions it is
> a killer feature (along with replication).
18GB is not enough for good deduplication, even I would like to have it.
> A ZFS volume with an ext4 file system? Why?
ACL access list and future use of drdb.
> For backups? I keep mine on dedicated disks in another machine.
This is a backup server with file server.
So hope to cope with windows speed in file transfer.
