Re: 4.4 & Samba write performance
On Nov 20, 2008 2:22pm, tico <[EMAIL PROTECTED]> wrote: Answers to your questions are inline /w the question with some big in attempt to reduce noise. Perhaps I should have phrased my comment better, or possibly I'm wrong altogether, but I don't see where you showed that the problem was with *Samba* throughput (which would obviously be directed to ports@ or even [EMAIL PROTECTED] as you state) versus hardware/OS/configuration. Good point. I guess then it depends on how you look at it. My "performance issue" was noticed from the perspective of samba, but not likely samba's fault at all. What network throughput can your system handle? What network and disk throughput can your clients handle? I don't know. Unless you recommend something else, I'll try iPerf and command line ftp from the windows box to my OpenBSD storage server and see what the numbers are like. What disk I/O throughput can your system handle? Based on the examples in your example from your previous email: # time dd if=/dev/zero of=/home/test bs=1024k count=5000 5000+0 records in 5000+0 records out 524288 bytes transferred in 100.006 secs (52425156 bytes/sec) 1m40.12s real 0m0.03s user 1m16.59s system # time dd if=/dev/zero of=/home/test2 bs=1024k count=500 500+0 records in 500+0 records out 524288000 bytes transferred in 9.856 secs (53193127 bytes/sec) 0m9.96s real 0m0.00s user 0m7.64s system # time dd if=/dev/zero of=/home/test3 bs=1024k count=50 50+0 records in 50+0 records out 52428800 bytes transferred in 0.971 secs (53964967 bytes/sec) 0m1.06s real 0m0.00s user 0m0.77s system And in case you want to know: # cat /etc/fstab /dev/wd0a / ffs rw,softdep 1 1 /dev/wd0d /mnt/wd0d ffs rw,nodev,nosuid,async 1 2 /dev/wd0e /mnt/wd0e ffs rw,nodev,nosuid,async 1 2 /dev/wd0f /mnt/wd0f ffs rw,nodev,nosuid,async 1 2 /dev/wd0g /mnt/wd0g ffs rw,nodev,nosuid,async 1 2 Why async? Although it's a storage server, the data being saved isn't important enough to warrant special care. Due-ly noted however that the use of soft-dep would likely be less resource intensive. What network and disk throughput can your clients handle? Don't do enough client to client copying to know. Though I do remember making a 2Gig MP2 tv recording from client to client and not even consider a performance issue many months ago. 2. What performance comparison is showing you that the write throughput that you *are* achieving is bad? Again, this is from a samba perspective. After initial setup and an untweaked smb.conf file a standard windows to samba server file copy just took too long, something like 170meg file copy was estimated to take 2 minutes before the copy was supposed to complete. That's about 1.42MB/sec file transfer rate. (hard, exact numbers are currently impossible to grab @ the moment because I am not in the network environment that exhibits the problem). How can I make a comparison such as that without the proper hardware on hand? (IE other GigE Nic for OpenBSD) - - suffice it to say that that time constraints are currently in play that prevents such an acquisition and installation. What consistently slow speed did you get on average running a specific test, and what better speed did you get running the exact same test after changing each setting one-by-one? Didn't. I jumped on the bandwagon for a software solution I could have with acceptable immediate results without taking the time for a thorough test. Again time plays a factor there. You'll notice, hopefully, that there are no "tweaks" added to any relevant conf file, Oh I noticed. because the defaults just plain work, and I've not been able to show a bottleneck in real-world performance that is solvable through hackery ... and most importantly, the customer is happy and I don't have to write even more documentation for the next sysadmin that walks through the door. That's why you get paid the big bucks for such consulting work and I don't consult. BTW, if you'd rather move this discussion off list or to misc that's fine with me. and of course, thanks for your input. Enjoy the weekend!
Re: 4.4 & Samba write performance
1. Why is this sent to the "ports@" mailing list ? 2. What performance comparison is showing you that the write throughput that you *are* achieving is bad? I have much nicer systems on nicer 100Mbit managed switches that individual Windows clients are not able to drive at higher than 9MB/sec ~= 72Mbits. If you're ranging from 4-12MBytes/sec, those are good real-world numbers. 3. If you want to test out your problem, why haven't you tested out the network throughput capability of your system/network, separately from the disk I/O capability? /dev/null and /dev/zero exist. use them. 4. Why are you trying to tweak the TCP stack of your system, if you don't know what each piece does, whether or not it's a bottleneck, and what the trade-offs are from adjusting it? There is no "kern.system.go.faster=true" option for a reason. "Premature optimization is the root of all evil" -- Knuth Are you sure that PCI cards sharing the same IRQ is a performance problem? I have systems that serve 100's of megs of data where multiple NICs are sharing the same IRQ. Don't drink the kool-aid that you find on random forums. Only attempt to optimize what you can establish is *actually* a problem. Do you want to see one of my client's fileserver systems that serves data faster than the Windows clients can eat it? --- from a stock smb.conf: [global] workgroup = BHD server string = Samba Server passdb backend = tdbsam log file = /var/log/smbd.%m max log size = 50 os level = 33 preferred master = Yes domain master = Yes dns proxy = No wins support = Yes hosts allow = 192.168.1., 192.168.0., 127. -- # uname -m -r -v 4.4 GENERIC#1915 amd64 # grep -v ^# /etc/sysctl.conf net.inet.ip.forwarding=1# 1=Permit forwarding (routing) of IPv4 packets # em0: flags=8843 mtu 1500 lladdr 00:30:48:9a:17:34 description: internal net (VLAN100 untagged) media: Ethernet autoselect (100baseTX full-duplex) status: active inet6 fe80::230:48ff:fe9a:1734%em0 prefixlen 64 scopeid 0x1 inet 192.168.0.252 netmask 0xff00 broadcast 192.168.0.255 I'm running a single RAID1 volume (should be slower than your single disk) on a pair of SATA drives on an Areca controller: arc0 at pci2 dev 14 function 0 "Areca ARC-1210" rev 0x00: irq 5 arc0: 4 ports, 256MB SDRAM, firmware V1.43 2007-4-17 scsibus0 at arc0: 16 targets, initiator 16 sd0 at scsibus0 targ 0 lun 0: SCSI3 0/direct fix ed sd0: 715255MB, 512 bytes/sec, 1464843264 sec total no exotic filesystem mounts either: /dev/sd0j on /home/art type ffs (local, noatime, nodev, nosuid, softdep) /dev/sd0e on /home type ffs (local, noatime, nodev, nosuid, softdep) /dev/sd0k on /home/sandbox type ffs (local, noatime, nodev, nosuid, softdep) If it ain't broke, don't fix^H^H^H break it. ::yawn:: -Tico Whyzzi wrote: I'm looking to improve my samba writing performance but I Fear I've either tweaked something wrong or not tweaked it enough and as such I'm looking for any and all criticism that might improve my situation. Admittedly I'm running somewhat of a Frankenstein system - older P3 /w too many pci cards that sadly in some ways have to share the same IRQ (the Compaq BIOS in manual IRQ assignment is not very flexible). I have however gone out of my way to disable whatever I am not using - - eg sound, PATA, onboard usb 1.1, etc. And yes that is a CMD Silicon Image 3512 PCI add-in card /w 1TB western digital SATA drive on IRQ 14 -- it was fun to watch the machine boot from the SATA DVDRW. "Out of the box" was performance just short of abysmal. I can't currently remember the exact output numbers but doing a windows to samba copy worked at about 2MB/s (watching systat vmstat as it writes to the HD). By adding the following to OpenBSD: -- net.inet.ip.ifq.maxlen=768 # Maximum input queue (256*number of interfaces) net.inet.icmp.errppslimit=1000 # Maximum outgoing ICMP error messages per sec net.inet.ip.ttl=254 # TTL = "min-ttl" in scrub rule in pf.conf net.inet.tcp.ackonpush=1# acks for packets with the push bit no delay net.inet.tcp.ecn=1 # Explicit Congestion Notification enabled net.inet.tcp.mssdflt=1452 # maximum segment size (1452 from scrub pf.conf) net.inet.tcp.rfc1323=1 # RFC1323 TCP window scaling net.inet.tcp.recvspace=262144 # Increase TCP "recieve" windows size to increas e performance net.inet.tcp.sendspace=262144 # Increase TCP "send" windows size to increase p erformance net.inet.tcp.sack=1 # enable TCP Selective ACK (SACK) Packet Recover y net.inet.udp.recvspace=262144 # Increase UDP "recieve" windows size to increas e performance net.inet.udp.sendspace=262144 # Increase UDP "send" windows size to increase p erformance -- and the following performance option in smb.conf -- socket options = IPTOS_LOWDELAY SO_RCVBUF=65536 S
Re: 4.4 & Samba write performance
Thanks for quick responses. It always stands to show that I forget something when sending out questions like this to list servs. Yes, I am running a Gig connection (albeit cheap gig connection - hence the realtek gig nic on the OpenBSD box and a cheap $60cdn Gig unmanaged 8 port switch). I'll likely play around a bit more over the next day or two and report my findings to list then.
Re: 4.4 & Samba write performance
On 18 Nov 2008 at 10:36, Whyzzi wrote: > I'm looking to improve my samba writing performance but I Fear I've > either tweaked something wrong or not tweaked it enough and as such > I'm looking for any and all criticism that might improve my situation. > > Admittedly I'm running somewhat of a Frankenstein system - older P3 /w > too many pci cards that sadly in some ways have to share the same IRQ > (the Compaq BIOS in manual IRQ assignment is not very flexible). I > have however gone out of my way to disable whatever I am not using - > - eg sound, PATA, onboard usb 1.1, etc. And yes that is a CMD Silicon > Image 3512 PCI add-in card /w 1TB western digital SATA drive on IRQ 14 > -- it was fun to watch the machine boot from the SATA DVDRW. > > "Out of the box" was performance just short of abysmal. I can't > currently remember the exact output numbers but doing a windows to > samba copy worked at about 2MB/s (watching systat vmstat as it writes > to the HD). > > By adding the following to OpenBSD: > -- > net.inet.ip.ifq.maxlen=768 # Maximum input queue (256*number of > interfaces) > net.inet.icmp.errppslimit=1000 # Maximum outgoing ICMP error messages per sec > net.inet.ip.ttl=254 # TTL = "min-ttl" in scrub rule in pf.conf > net.inet.tcp.ackonpush=1# acks for packets with the push bit no delay > net.inet.tcp.ecn=1 # Explicit Congestion Notification enabled > net.inet.tcp.mssdflt=1452 # maximum segment size (1452 from scrub > pf.conf) > net.inet.tcp.rfc1323=1 # RFC1323 TCP window scaling > net.inet.tcp.recvspace=262144 # Increase TCP "recieve" windows size to > increas > e performance > net.inet.tcp.sendspace=262144 # Increase TCP "send" windows size to > increase p > erformance > net.inet.tcp.sack=1 # enable TCP Selective ACK (SACK) Packet > Recover > y > net.inet.udp.recvspace=262144 # Increase UDP "recieve" windows size to > increas > e performance > net.inet.udp.sendspace=262144 # Increase UDP "send" windows size to > increase p > erformance > -- > > and the following performance option in smb.conf > -- >socket options = IPTOS_LOWDELAY SO_RCVBUF=65536 SO_SNDBUF=65536 >sync always = no >read raw = no >write raw = no > -- First you manually set your TCP send/recv buffers to 256K (in sysctl.conf) but then you tell SAMBA to re-set them to only 64K. According to SAMBA docs, socket options get passed directly to the TCP/IP stack, i.e. these are not "internal" settings. OTOH, in all my personal tests I was unable to verify any perfomance gains beyond 64K buffers. Perhaps experts like henning@ or stu@ can chime in and advise what is the highest performance setting... Last but not least, if you are using a Gig switch and your internal computer(s) also have Gig NICs, you can get a boost in network performance by turning on Jumbo frames -- ifconfig re0 mtu NN, where 1500 < NN < 7422 (Realtek limitation). But beware that ALL computers need to have the same Jumbo frame setting or they will stop communicating. (which is probably the reason why Jumbo frames are NOT on by default.) > > I managed to get my internal network interface (re0) to 4000 > interupts/sec and a wd0 write speed between 4MB and 12MB per second. > Again any help anyone can give me that might improve my overall write > speed would be greatly appriciated. > > incase you're wondering the system is for a home network firewall /w > wireless support. dmesg follows: > > OpenBSD 4.4 (GENERIC) #1021: Tue Aug 12 17:16:55 MDT 2008 > [EMAIL PROTECTED]:/usr/src/sys/arch/i386/compile/GENERIC
Re: 4.4 & Samba write performance
On Tue, Nov 18, 2008 at 10:36:04AM -0700, Whyzzi wrote: [cut] > I managed to get my internal network interface (re0) to 4000 > interupts/sec and a wd0 write speed between 4MB and 12MB per second. > Again any help anyone can give me that might improve my overall write > speed would be greatly appriciated. You're using a 100Mbit connection, I'm surprised you got it as high as 12MB/s (96Mbit/s), usually from samba I would expect something around 6, maybe 8. -- viq pgpMctAhyBGCJ.pgp Description: PGP signature
4.4 & Samba write performance
I'm looking to improve my samba writing performance but I Fear I've
either tweaked something wrong or not tweaked it enough and as such
I'm looking for any and all criticism that might improve my situation.
Admittedly I'm running somewhat of a Frankenstein system - older P3 /w
too many pci cards that sadly in some ways have to share the same IRQ
(the Compaq BIOS in manual IRQ assignment is not very flexible). I
have however gone out of my way to disable whatever I am not using -
- eg sound, PATA, onboard usb 1.1, etc. And yes that is a CMD Silicon
Image 3512 PCI add-in card /w 1TB western digital SATA drive on IRQ 14
-- it was fun to watch the machine boot from the SATA DVDRW.
"Out of the box" was performance just short of abysmal. I can't
currently remember the exact output numbers but doing a windows to
samba copy worked at about 2MB/s (watching systat vmstat as it writes
to the HD).
By adding the following to OpenBSD:
--
net.inet.ip.ifq.maxlen=768 # Maximum input queue (256*number of interfaces)
net.inet.icmp.errppslimit=1000 # Maximum outgoing ICMP error messages per sec
net.inet.ip.ttl=254 # TTL = "min-ttl" in scrub rule in pf.conf
net.inet.tcp.ackonpush=1# acks for packets with the push bit no delay
net.inet.tcp.ecn=1 # Explicit Congestion Notification enabled
net.inet.tcp.mssdflt=1452 # maximum segment size (1452 from scrub pf.conf)
net.inet.tcp.rfc1323=1 # RFC1323 TCP window scaling
net.inet.tcp.recvspace=262144 # Increase TCP "recieve" windows size to increas
e performance
net.inet.tcp.sendspace=262144 # Increase TCP "send" windows size to increase p
erformance
net.inet.tcp.sack=1 # enable TCP Selective ACK (SACK) Packet Recover
y
net.inet.udp.recvspace=262144 # Increase UDP "recieve" windows size to increas
e performance
net.inet.udp.sendspace=262144 # Increase UDP "send" windows size to increase p
erformance
--
and the following performance option in smb.conf
--
socket options = IPTOS_LOWDELAY SO_RCVBUF=65536 SO_SNDBUF=65536
sync always = no
read raw = no
write raw = no
--
I managed to get my internal network interface (re0) to 4000
interupts/sec and a wd0 write speed between 4MB and 12MB per second.
Again any help anyone can give me that might improve my overall write
speed would be greatly appriciated.
incase you're wondering the system is for a home network firewall /w
wireless support. dmesg follows:
OpenBSD 4.4 (GENERIC) #1021: Tue Aug 12 17:16:55 MDT 2008
[EMAIL PROTECTED]:/usr/src/sys/arch/i386/compile/GENERIC
cpu0: Intel Pentium III ("GenuineIntel" 686-class) 864 MHz
cpu0:
FPU,V86,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,MMX,FXSR,SSE
real mem = 402026496 (383MB)
avail mem = 380039168 (362MB)
mainbus0 at root
bios0 at mainbus0: AT/286+ BIOS, date 09/13/02, BIOS32 rev. 0 @
0xe7300, SMBIOS rev. 2.3 @ 0xfd33c (50 entries)
bios0: vendor Compaq version "686P2 v3.14" date 09/13/2002
bios0: Compaq Deskpro
apm at bios0 function 0x15 not configured
acpi0 at bios0: rev 0
acpi0: tables DSDT FACP SSDT SSDT SSDT APIC SSDT SSDT SSDT SSDT
acpi0: wakeup devices PCI0(S4) HUB_(S4) COM1(S4) COM2(S4) USB1(S3)
USB2(S3) PBTN(S4)
acpitimer0 at acpi0: 3579545 Hz, 24 bits
acpiprt0 at acpi0: bus 0 (PCI0)
acpiprt1 at acpi0: bus 2 (HUB_)
acpicpu0 at acpi0
acpibtn0 at acpi0: PBTN
bios0: ROM list: 0xc/0x8000 0xc8000/0x4800 0xe/0x1!
cpu0 at mainbus0
pci0 at mainbus0 bus 0: configuration mode 1 (no bios)
pchb0 at pci0 dev 0 function 0 "Intel 82815 Host" rev 0x02
ppb0 at pci0 dev 1 function 0 "Intel 82815 AGP" rev 0x02
pci1 at ppb0 bus 1
vga1 at pci1 dev 0 function 0 "ATI Rage Pro" rev 0x5c
wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation)
wsdisplay0: screen 1-5 added (80x25, vt100 emulation)
agp0 at vga1: aperture at 0x4400, size 0x240
drm at vga1 unsupported
ppb1 at pci0 dev 30 function 0 "Intel 82801BA Hub-to-PCI" rev 0x01
pci2 at ppb1 bus 2
fxp0 at pci2 dev 8 function 0 "Intel 82562" rev 0x01, i82562: irq 5,
address 00:02:a5:01:6b:f8
inphy0 at fxp0 phy 1: i82562EM 10/100 PHY, rev. 0
ral0 at pci2 dev 9 function 0 "Ralink RT2860" rev 0x00: irq 11,
address 00:1b:fc:df:3e:f9
ral0: MAC/BBP RT2860 (rev 0x0101), RF RT2820 (2T3R)
pciide0 at pci2 dev 10 function 0 "CMD Technology SiI3512 SATA" rev 0x01: DMA
pciide0: using irq 14 for native-PCI interrupt
pciide0: port 0: device present, speed: 1.5Gb/s
atapiscsi0 at pciide0 channel 0 drive 0
scsibus0 at atapiscsi0: 2 targets, initiator 7
cd0 at scsibus0 targ 0 lun 0: ATAPI
5/cdrom removable
cd0(pciide0:0:0): using BIOS timings, Ultra-DMA mode 5
pciide0: port 1: device present, speed: 1.5Gb/s
wd0 at pciide0 channel 1 drive 0:
wd0: 16-sector PIO, LBA48, 953869MB, 1953525168 sectors
wd0(pciide0:1:0): using BIOS timings, Ultra-DMA mode 6
ohci0 at pci2 dev 11 function 0 "NEC USB" rev 0x43: irq 10, version 1.0
ohci1 at pci2 dev 11 function 1 "NEC USB" rev 0x43: irq 10, version 1.0
ehc
