no battery = no cache. Performance is abysmal without.
On Tue, Aug 22, 2006 at 02:17:35AM +0200, Robert Urban wrote:
> Hi Folks,
>
> using a simple test program to write sequential blocks to a file, optionally
> opening with O_SYNC, I've tested write performance to a MegaRAID logical
> drive consisting of a RAID-5 set of 4 72GB HP Ultra320 disks and to a RAID-0
> drive consisting of a single 300GB HP Ultra320 disk. The controller
> has 128MB of cache, but I do not have a battery, so cache write policy
> is write-through. The kernel is the "bsd", and not "bsd.mp", but it makes
> no difference. I've tried both.
>
> All tests performed with O_SYNC, to avoid bufcache interaction.
>
> The performance, at least to my perhaps naive eyes, seems abysmal. I'm
> getting
> 1.2MB/sec on the RAID-5 logical drive.
>
> It doesn't matter what blocksize I choose, 32k, 64k, 128k, 256k. The
> chunksize
> on the logical drive is the default, 64k.
>
> On the single HP 300GB Ultra320 RAID-0 drive write performance is also bad:
> 1.7MB/sec.
>
> The systems is (in case you hadn't already guessed) a Compaq ProLiant DL380-G2
> with 1GB main memory and dual 1266MHz CPUs. I will attach dmesg, my test
> program, and the output of "bioctl -iv ami0" below.
>
> I can't imagine the MegaRAID controller is this slow. Any suggestions?
>
> Rob Urban
>
> dmesg:
> ---------------------------------------------------------------
> OpenBSD 3.9 (GENERIC) #617: Thu Mar 2 02:26:48 MST 2006
> [EMAIL PROTECTED]:/usr/src/sys/arch/i386/compile/GENERIC
> cpu0: Intel(R) Pentium(R) III CPU family 1266MHz ("GenuineIntel" 686-class)
> 1.27 GHz
> cpu0:
> FPU,V86,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,M
> MX,FXSR,SSE
> real mem = 1073307648 (1048152K)
> avail mem = 972660736 (949864K)
> using 4278 buffers containing 53768192 bytes (52508K) of memory
> mainbus0 (root)
> bios0 at mainbus0: AT/286+(00) BIOS, date 12/31/99, BIOS32 rev. 0 @ 0xf0000
> pcibios0 at bios0: rev 2.1 @ 0xf0000/0x2000
> pcibios0: PCI BIOS has 9 Interrupt Routing table entries
> pcibios0: PCI Interrupt Router at 000:15:0 ("ServerWorks OSB4" rev 0x00)
> pcibios0: PCI bus #0 is the last bus
> bios0: ROM list: 0xc0000/0x8000 0xc8000/0x2600 0xee000/0x2000!
> cpu0 at mainbus0
> pci0 at mainbus0 bus 0: configuration mode 1 (no bios)
> pchb0 at pci0 dev 0 function 0 "ServerWorks CNB20HE Host" rev 0x23
> pci1 at pchb0 bus 1
> ppb0 at pci1 dev 3 function 0 "Intel i960 RP PCI-PCI" rev 0x05
> pci2 at ppb0 bus 2
> vga1 at pci2 dev 0 function 0 "ATI Mach64 GV" rev 0x7a
> wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation)
> wsdisplay0: screen 1-5 added (80x25, vt100 emulation)
> "Intel 80960RP ATU" rev 0x05 at pci1 dev 3 function 1 not configured
> pchb1 at pci0 dev 0 function 1 "ServerWorks CNB20HE Host" rev 0x01
> pchb2 at pci0 dev 0 function 2 "ServerWorks CNB20HE Host" rev 0x01
> pchb3 at pci0 dev 0 function 3 "ServerWorks CNB20HE Host" rev 0x01
> pci3 at pchb3 bus 7
> ami0 at pci3 dev 4 function 0 "Symbios Logic MegaRAID" rev 0x01: irq 3 Dell
> 518 64b/lhc
> ami0: FW 351X, BIOS v1.10, 128MB RAM
> ami0: 2 channels, 0 FC loops, 2 logical drives
> scsibus0 at ami0: 40 targets
> sd0 at scsibus0 targ 0 lun 0: <AMI, Host drive #00, > SCSI2 0/direct fixed
> sd0: 208110MB, 208110 cyl, 64 head, 32 sec, 512 bytes/sec, 426209280 sec total
> sd1 at scsibus0 targ 1 lun 0: <AMI, Host drive #01, > SCSI2 0/direct fixed
> sd1: 286080MB, 286080 cyl, 64 head, 32 sec, 512 bytes/sec, 585891840 sec total
> scsibus1 at ami0: 16 targets
> uk0 at scsibus1 targ 15 lun 0: <COMPAQ, PROLIANT 4L6I, 1.84> SCSI2
> 3/processor
> fixed
> uk0: unknown device
> scsibus2 at ami0: 16 targets
> "Compaq PCI Hotplug" rev 0x12 at pci3 dev 7 function 0 not configured
> fxp0 at pci0 dev 2 function 0 "Intel 8255x" rev 0x08, i82559: irq 5, address
> 00:08:02:8a:4b:fc
> inphy0 at fxp0 phy 1: i82555 10/100 PHY, rev. 4
> fxp1 at pci0 dev 4 function 0 "Intel 8255x" rev 0x08, i82559: irq 7, address
> 00:08:02:8a:4b:fb
> inphy1 at fxp1 phy 1: i82555 10/100 PHY, rev. 4
> "Compaq Netelligent ASMC" rev 0x00 at pci0 dev 6 function 0 not configured
> piixpm0 at pci0 dev 15 function 0 "ServerWorks OSB4" rev 0x51: SMBus disabled
> pciide0 at pci0 dev 15 function 1 "ServerWorks OSB4 IDE" rev 0x00: DMA
> atapiscsi0 at pciide0 channel 0 drive 0
> scsibus3 at atapiscsi0: 2 targets
> cd0 at scsibus3 targ 0 lun 0: <COMPAQ, CD-ROM SN-124, N102> SCSI0 5/cdrom
> removable
> cd0(pciide0:0:0): using PIO mode 4, DMA mode 2, Ultra-DMA mode 1
> ohci0 at pci0 dev 15 function 2 "ServerWorks OSB4/CSB5 USB" rev 0x04: irq 11,
> version 1.0, legacy support
> usb0 at ohci0: USB revision 1.0
> uhub0 at usb0
> uhub0: ServerWorks OHCI root hub, rev 1.00/1.00, addr 1
> uhub0: 4 ports with 4 removable, self powered
> isa0 at mainbus0
> isadma0 at isa0
> pckbc0 at isa0 port 0x60/5
> pckbd0 at pckbc0 (kbd slot)
> pckbc0: using irq 1 for kbd slot
> wskbd0 at pckbd0: console keyboard, using wsdisplay0
> pms0 at pckbc0 (aux slot)
> pckbc0: using irq 12 for aux slot
> wsmouse0 at pms0 mux 0
> pcppi0 at isa0 port 0x61
> midi0 at pcppi0: <PC speaker>
> spkr0 at pcppi0
> npx0 at isa0 port 0xf0/16: using exception 16
> pccom0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
> fdc0 at isa0 port 0x3f0/6 irq 6 drq 2
> fd0 at fdc0 drive 0: 1.44MB 80 cyl, 2 head, 18 sec
> biomask ef4d netmask efed ttymask ffef
> pctr: 686-class user-level performance counters enabled
> mtrr: Pentium Pro MTRR support
> uhub0: device problem, disabling port 1
> dkcsum: sd0 matches BIOS drive 0x80
> dkcsum: sd1 matches BIOS drive 0x81
> root on sd0a
> rootdev=0x400 rrootdev=0xd00 rawdev=0xd02
>
> write-test.c:
> * write-test.c
> *
> * write-test is meant to test simple writing performance to a *new*
> * file. Blocksize can be set, default is 128KB.
> *
> * Usage:
> *
> * This program performs tests on a *new* file. (it will truncate an
> * existing file.)
> *
> * Although performance results are returned when file-creation is finished,
> * they don't have any relevance for random writes. File-creation is
> * performed using an open(2) call.
> *
> * This program writes blocks of the specified block size, default=128KB.
> *
> * To perform a test:
> *
> * write-test -f <file-1>
> *
> * Compilation:
> *
> * cc -O4 -o write-test write-test.c
> *
> * Version: 1.0
> *
> * author: Robert Urban <[EMAIL PROTECTED]>
> *-----------------------------------------------------------------------*/
>
> #include <stdlib.h>
> #include <stdio.h>
> #include <unistd.h>
> #include <fcntl.h>
> #include <sys/stat.h>
> #include <sys/types.h>
> #include <signal.h>
> #include <time.h>
> #include <sys/time.h>
> #include <errno.h>
>
> #define KILOBYTE (1024)
> #define MEGABYTE (1024 * 1024)
> #define GIGABYTE (1024 * 1024 * 1024)
> #define DEF_BLOCKSIZE (128 * 1024)
> #define MAX_BLOCKSIZE (256 * 1024)
>
>
> /* prototypes */
> void handleSignal(int sig);
> float diff(struct timeval *start, struct timeval *end);
> void initChunk(char ch);
> void usage();
> void writeData();
>
>
> /* globals */
> int allrandom = 1;
> int debug = 0;
> long size = 128; /* 128 MB */
> char filename[128] = { NULL };
> int seconds = 60;
> int io_count = 0;
> float latency_cum = 0.0;
> int blocksize = DEF_BLOCKSIZE;
> long written = 0;
>
> struct timeval bm_start, bm_end;
> struct timeval io_start, io_end;
>
> char *chunk;
> int thefd;
> int blockcount;
> int f_sync = 0;
> int f_directio = 0;
>
> main(int argc, char *argv[]) {
> time_t tloc;
> char ran_args[32];
> int c, errflg = 0;
> struct stat statbuf;
> int num_files = 0;
> struct timeval tv;
>
> gettimeofday(&tv, NULL);
> srand48(tv.tv_usec);
>
> #define ARGS "hdDSf:s:b:"
>
> optarg = NULL;
> while (!errflg && ((c = getopt(argc, argv, ARGS)) != -1))
> switch (c) {
> case 'h':
> usage();
> exit(0);
> case 'b':
> blocksize = atoi(optarg) * KILOBYTE;
> break;
> case 's':
> /* size = atoi(optarg) * MEGABYTE; */
> size = atoi(optarg);
> break;
> case 'd':
> debug = 1;
> break;
> case 'S':
> f_sync = 1;
> break;
> case 'f':
> if (num_files == 1) {
> printf("you cannot specify more than 1
> file.\n");
> exit(1);
> }
> strcpy(filename, optarg);
> num_files++;
> break;
> default:
> printf("unrecognized option: [%c].\n", c);
> errflg++;
> usage();
> exit(-1);
> }
>
> chunk = (char *) malloc(MAX_BLOCKSIZE);
>
> if (num_files != 1) {
> printf("exactly 1 file must be specified using '-f'.\n");
> exit(1);
> }
>
> /* convert size to bytes */
> size *= MEGABYTE;
>
> initChunk('A');
>
> blockcount = size / blocksize;
>
> if (debug) {
> printf("SIZE : %ld\n", size);
> printf("BLOCKS : %d\n", blockcount);
> printf("BLKSIZ : %d\n", blocksize);
> printf("FILE : %s\n", filename);
> printf("SYNC : %s\n", f_sync ? "Yes" : "No");
> }
>
> if (filename == 0) {
> printf("must supply a file using \"-f <filename>\"\n");
> exit(-1);
> }
>
> signal(SIGINT, handleSignal);
> signal(SIGTERM, handleSignal);
> signal(SIGHUP, handleSignal);
> signal(SIGALRM, handleSignal);
>
> gettimeofday(&bm_start, NULL);
> writeData();
> handleSignal(0);
> }
>
> void usage() {
> printf("usage: write-test -f <file> [flags]\n");
> printf("\twhere flags are:\n");
> printf("\t-d turn on debug messages\n");
> printf("\t-S open output file with O_SYNC\n");
> printf("\t-s <size> specify size in Megebytes
> (default=100MB)\n");
> printf("\t-b <blocksize> specify blocksize in Kilobytes
> (default=128)\n");
> }
>
> void markStart()
> {
> gettimeofday(&io_start, NULL);
> }
>
> void markEnd()
> {
> float et;
> gettimeofday(&io_end, NULL);
>
> et = diff(&io_start, &io_end);
> latency_cum += et;
> if (debug) printf("latency: %.3f ms\n", et*1000);
> }
>
> void handleSignal(int sig) {
> float et;
> int index;
>
> gettimeofday(&bm_end, NULL);
> et = diff(&bm_start, &bm_end);
> printf("%d IOs in %.2f seconds, %.2f IOs/sec, %.2f MB/sec\n",
> io_count, et,
> io_count/et,
> ((float)written/et)/(float)MEGABYTE);
>
> printf("latency for %s: %.3f ms\n",
> filename,
> latency_cum/io_count*1000);
>
> close(thefd);
> exit(0);
> }
>
> void writeData() {
> int this_blocksize;
> int ret;
> /* int flags = O_CREAT|O_TRUNC|O_WRONLY|O_SYNC; */
> int flags = O_CREAT|O_TRUNC|O_WRONLY;
> mode_t mode = 0666;
>
> if (f_sync) {
> flags |= O_SYNC;
> }
>
> if ((thefd = open(filename, flags, mode)) == -1) {
> printf("open of \"%s\" failed for writing, errno=%d\n",
> filename, errno);
> exit(-1);
> }
>
> if (debug) printf("entering loop.\n");
>
> while(written < size) {
>
> this_blocksize = ((size - written) < blocksize) ?
> (size - written) : blocksize;
>
> markStart();
> if (debug) printf("writing [%d] bytes...\n", this_blocksize);
> ret = write(thefd, chunk, this_blocksize);
> markEnd();
> if (ret == -1) {
> printf("write failed.\n");
> exit(1);
> }
> written += this_blocksize;
> io_count++;
> }
>
> handleSignal(0);
> }
>
> int spew(char *tag)
> {
> struct timeval tv;
>
> gettimeofday(&tv, NULL);
> printf(" TS[%s]: %lu.%d\n", tag, tv.tv_sec, tv.tv_usec);
>
> return(0);
> }
>
> void initChunk(char ch) {
> memset(chunk, ch, MAX_BLOCKSIZE);
> }
>
> float diff(struct timeval *start, struct timeval *end)
> {
> float d = end->tv_sec - start->tv_sec +
> (end->tv_usec - start->tv_usec)/1000000.0;
>
> return(d);
> }
>
> bioctl -iv ami0:
> ----------------------------------------------------------------
> # bioctl -iv ami0
> Volume Status Size Device
> ami0 0 Online 218219151360 sd0 RAID5
> 0 Online 72834088960 0:0.0 noencl <COMPAQ BD07288277
> HPB2>
> '3KT06BVL000075237CP0'
> 1 Online 72834088960 0:1.0 noencl <COMPAQ BD07285A25
> HPB8>
> '3HZ6Z63500007433D7TT'
> 2 Online 72834088960 0:2.0 noencl <COMPAQ BD07287B4C
> HPB8>
> 'D20LF9XK'
> 3 Online 72834088960 0:3.0 noencl <COMPAQ BD07288277
> HPB2>
> '3KT06BXP00007523V3F7'
> ami0 1 Online 299976622080 sd1 RAID0
> 0 Online 299995496448 0:5.0 noencl <COMPAQ BD30089BBA
> HPB1>
> 'DA01P5B007CG0547'
> ami0 2 Hot spare 72834088960 0:4.0 noencl <COMPAQ BD0728A4B4
> HPBC>
> 'J20JWRZK'
