I'm attaching a writeup of some tests I did a few months ago using AFS
to serve data to heavily-loaded HTTP server. At that time I did not have
an opportunity to follow up very much on some of the unanswered questions,
and didn't feel like I had characterized the situation clearly enough.
The problems that we saw happened when we started the test scenario
against a "cold" cache with a moderate (60Mb) amount of data to retrieve.
In this scenario, the tests would proceed for a few minutes, and then the
AFS client/HTTP server would lose contact with the AFS server. The
addition of an additional database server did not solve the problem.
If anyone has any thoughts on this, I'd appreciate hearing them. Some
differences between these somewhat informal tests that I ran back in May
and what we're going to do now include a change in operating system
(Solaris 2.5.1 instead of 2.4), and Ultras (on 10Mb/sec ethernet) instead
of Sparc 20s. If we can get it, we'll try and run the enhanced release of
2.5.1 (for Web servers) and see what happens.
I believe tcp_max_conn_req was set to 128 during this test, but as I
said, I did this somewhat informally and didn't collect all of the data
that I should have. :-]
What I'm looking for is responses like "Yeah, we saw similar behaviour
when we did something like this and fixed it by <blah>", or "You might try
bumping up <bleh>".
Transarc didn't really provide much help (although they tried to be
helpful) as the support guy didn't feel like he had enough info to really
understand the problem.
-brian
---------------------------------------------------------------------------
| APPENDIX A. AFS/HTTP SERVER PERFORMANCE EVALUATION RESULTS
|
---------------------------------------------------------------------------
Overview -------- This test scenario is designed to do some stress testing
of an HTTP (Web) server reading its data out of AFS versus local disk.
The HTTP client is running some home-grown software (webbash) which allows
us to fork off multiple simultaneous threads which act as HTTP clients,
requesting documents from a list. The document testbed is a set of files
containing random ASCII characters ranging in size from 0k to 1Mb, and
totals initially 2.2Mb. Copying this testbed into subdirectories [a..z]
yields a testbed of 60Mb.
Test Environment
----------------
The test environment consists of three machines isolated from the rest of
the network via an ethernet hub. I connect to the machines via a Cisco
terminal server which is not isolated from the local network. All
machines are running Solaris 2.4 w/o the recommended patches. :-]
[prod-1b] AFS Fileserver Sparc20/128Mb
AFS Client
[ log ] AFS Client / Sparc20/64Mb
HTTP Server
[prod-2a] HTTP Client Sparc20/128Mb
("bash" program)
AFS Client
(later becomes an additional AFS file/dbserver)
Explanation of the Fields Below
-------------------------------
Data Source - local disk or AFS
Client Threads - number of simultaneous threads created by "webbash". Each
thread reads the file list and randomizes it. Each thread will
time out after ten seconds if it does not receive data from the
HTTP server.
Iterations - number of times the test suite iterates over the list of files
to retrieve.
Cache/Data Ratio - the ratio of AFS cache size to data testbed size.
Daemons (afsd) - number of extra afsd processes to run to handle service
requests.
Volume Type - ReadWrite or ReadOnly. Lack of per-file callbacks on ReadOnly
volumes should reduce AFS client-server traffic.
Throughput (Mb/Hr) - As reported by webbash in megabytes per hour.
HTTP Ops/Sec - Number of HTTP operations services per second. This is probably
the real "thoughput benchmark".
Comments - Describes various events that may have happened during the test.
See the key below the table for details on this.
Cache/ Throug-
Data Client Itera- Data Data Daemons Volume hput HTTP Com-
Source Threads tions Ratio Size (afsd) Type (Mb/Hr) Ops/Sec ments
-------------------------------------------------------------------------------
local 10 10 n/a 2.2Mb n/a n/a 4048 15.01
afs 10 10 34/1 2.2Mb 3 RW 4053 15.04
afs 20 10 34/1 2.2Mb 3 RW 3959 14.69 #
afs 50 10 34/1 2.2Mb 3 RW 3795 13.98 #+
afs 100 10 34/1 2.2Mb 3 RW 4306 23.06 #+
afs 200 10 34/1 2.2Mb 3 RW 5208 31.31 #+
afs 10 5 1.25 60Mb 3 RW 3793 37.59 !@
afs 1 5 1.25 60Mb 3 RW 1504 5.57
afs 3 3 1.25 60Mb 3 RW 3155 11.74 $%
afs 5 2 1.25 60Mb 5 RO 2927 12.53 !@
# A second file/db server was added to the AFS cell at this point.
(these were warm cache reads)
afs 10 1 1.25 60Mb 5 RO 3957 14.68 #$
# Then I flushed the volume from the cache (i.e. cold reads)
afs 10 2 1.25 60Mb 5 RO 1524 8.70
!+#+@+
Comments Key:
! - Cell "Thrashing": AFS client loses contact with fs and volserver.
AFS
client freezes.
@ - HTTP Server returned "403 Not Found" Errors (i.e. file did not exist).
# - Large files (500K+) took more than 10 seconds to transfer.
$ - Timeout (>10 sec) trying to read data.
% - HTTP Server returned "504" Error.
+ - Modifer to above. This event occurred many (>10, generally) times.
Problems Observed
-----------------
Interestingly, reading data from a small to moderate local AFS cache
appears to be slightly faster than local disk. This seems to differ from
work done by Michael Stolarchuck at U-M which suggested ways to improve
AFS cache read performance. The difference is small, and may be a
statistical anonmaly.
The biggest problem we've seen with this situation has been timeouts
between the AFS client and server(s) as the data is being fetched into the
cache. Generally what happens is that I can observe an initial burst of
traffic between the AFS client and server, during which I see a large
number of network collisions. After a period of time, the traffic will
come to a halt (I'm observing the lights on the hub). The AFS client/HTTP
server will at this point freeze for a while, and will eventually report
that the server(s) for the realm are unavailable. This will cause the
HTTP server to report that the files the various threads are trying to
fetch do not exist, which will generate the "403 Not Found" errors.
AFS timeouts generally look like this:
afs: Lost contact with file server 198.83.22.104 in cell test.ans.net (all
multi-homed ip addresses down for the server)
afs: Lost contact with file server 198.83.22.104 in cell test.ans.net (all
multi-homed ip addresses down for the server)
afs: file server 198.83.22.104 in cell test.ans.net is back up
(multi-homed address; other same-host interfaces may still be down)
afs: file server 198.83.22.104 in cell test.ans.net is back up
(multi-homed address; other same-host interfaces may still be down)
Once the data is in the cache, the timeouts do not happen. With a larger
number of clients (threads), the timeouts seem to happen more frequently.
When the timeouts do happen, the AFS client machine freezes via serial
console. After a second file/dbserver was added to the cell, timeouts
occurred for both AFS servers, although the initial timeouts were only to
the new fileserver, which held one of the ReadOnly copies of the data I
was trying to fetch.
Only the AFS client itself seems to pause during this time. The main AFS
server and HTTP client do not seem to be affected.
During one test, the AFS client became unpingable and had to be rebooted
by sending a break to the console.
Conclusions
-----------
The artificialness of the test scenario makes it hard to determine whether
or not these problems would occur in Real Life. Multiple, parallel HTTP
GET streams coming at this rate may not be a realistic scenario. I don't
know "how much" traffic each thread really represents in real terms.
Still, we did observe some definite problems with the AFS client that may
or may not be tunable. Adding a second file/dbserver initially appeared
to help the situation, but after flushing the volume from the cache in
reality did nothing. Increasing the number of afsd processes helped
somewhat (it seemed to take longer for the client to lose contact with the
server), but didn't improve the situation very much.
I know what AFS is in use under some relatively heavy usage conditions at
U-M and NCSA. I know that U-M's WWW servers are considered "slow" by
their user populace (I helped administer them for a short time), but this
is probably due to a number of factors, including network congestion and
overburdened fileservers, and the growing pains of a very large AFS cell.
On the other hand, I don't generally consider NCSA's site to be
particularly slow.
Perhaps a more rational way of getting the cache seeded beforehand would
help.
I'd like to see some response from Transarc on this.
--
Brian W. Spolarich - ANS - [EMAIL PROTECTED] - (313)677-7311
Look both ways before crossing the Net.
