Thanks to Owen Garrett, who reminded me that I should have
mentioned a little more details about the client configuration.
My modified SURGE client fetch web pages on an "object" basis,
each object contains multiple pages. For each object the client
uses HTTP/1.1 keepalive, but not pipeline. After an object
being fetched completely, the client close the connection
to the server and reopen a new one for next object.
The delay time I added was between each web pages, so the
client goes to sleep for a little while with the connection
being still open.
FYI, I have attached the client code. Anyone have a wild guess
on what's going on? ;-) Thanks a lot!
-Min
On Mon, Feb 10, 2003 at 01:50:35PM -0600, Min Xu wrote:
> Hi All,
>
> Sorry I am posting this directly to the development list. But
> I think this is not a user setup problem and it is so strange
> maybe only you guys will have a clue on what's going on.
>
> I am a student of UW-Madison. In order to study computer
> architecture of commercial multiprocessor servers, we have used
> APACHE as one of our important workloads.
>
> I am the one who setup the workload on a 14-processor Sun
> Enterprise server. During setup I found a very strange behavior
> of the apache server (running with worker MPM). Essentially the
> strange thing is that:
>
> The server optimal throughput is not achieved by using a
> greedy client, who drive the server with no think time. But
> with tiny amount of think time, much better throughput is
> archievable. Also, with the greedy client, the server's
> performance decreased over time, which seems to be very
> counter-intuitive.
>
> Of course, just give you the short decription above does not
> help you to help me. So I will give you the detail problem
> description and data in the following. With the understanding
> of the source code, maybe you can give me some more hypothesises
> to try on.
>
> Workload background
> -------------------
> The setup of apache workload on is fairly simple comparing with
> some of the other workloads we have (OLTP). In this workload, we
> have a HTTP server and an automatic request generator(SURGE).
> Both of the programs are highly multi-threaded. The server has
> a pool of static text files to be served from a known URL to the
> request generator (the client). The size of the files follows a
> statistical distribution. And the client has multiple threads each
> emulate a user who access a serial of files in fixed order.
>
> In previous setup of the workload, we have removed client think time.
> The basis of that is the following: (we also have to put the server
> and the client on the same machine for other reasons)
>
> The workload (server + client) is a closed queueing system. The
> throughput of the system is ultimately determined by the bottleneck in
> the system. Having think time in the client only increase the parallelism
> in the system. It shouldn't change the maximum throughput too much.
> BTW, our goal is to archieve realistic workload setup with available
> hardware.
>
> If you think about it, for our current server throughput level, say 5000
> trans/sec, if each user have 1 second think time between fetching each
> file, this will need 5000 users to sustain this throughput. On the other
> hand, if we remove the think time from the client, maybe 10 users can also
> generate the same 5000 requests per second. So the difference here is that
> one server has 5000 httpd threads and the other has only 10 httpd threads.
> 10 won't be worse(in terms of server behavior) than 5000, right? Greedy
> client won't be worse(in terms of performance) than the lazy client, right?
>
> Well it is not that simple...
>
>
> I know how to get higher performance, but I don't know why it works!
> --------------------------------------------------------------------
> I have two version of surge clients in my hand. One is the original,
> one is my modified version. The difference between them would be the
> client efficiency. My modified version would fetch files more efficiently
> (because I made it emulate a simpler user) and have less thread
> switching overhead.
>
> However, when I comparing the server throughput using these two clients,
> I got very surprising results, roughly:
>
> old client: 3000 trans/sec
> new client: starts out from 4700 trans/sec, gradually degrade to 2500
> trans/sec after 10-20 minutes of runtime.
>
> And this really puzzled me for a long time. My supposedly performance
> enhancement did not improve the server Xput, but hurt it!
>
> Turns out the reason for this is the new client was too efficient! I
> added the think time between each URL request and new client was able
> to drive the server Xput to as high as 5000 trans/sec. But note, the
> real interesting thing is not the think time, but how sensitive the
> Xput was affected by it.
>
> I'd prefer to call the think time "delay time" in the following because
> I really only introduced very small amount of delay between each file
> fetch. The result can be seen in the following plots:
>
> http://www.cs.wisc.edu/~xu/files/delay_results.eps
> http://www.cs.wisc.edu/~xu/files/side1.eps
> http://www.cs.wisc.edu/~xu/files/side2.eps
>
> In this experiment, instead of using both old and new version of the
> client, I just used the new version with varying delay time and number
> of threads. Since there are two dimensions of freedom in the client,
> the plot is in 3D. The figures side1 and side2 is roughly the 2D
> projection of the Xput vs. thread and Xput vs. delay time.
>
> Each point on the plot is a 30 minutes benchmarking on a 14P MP system.
>
> Clearly, driving the server using no delay time is not optimal. No
> matter using same amount of threads or less number of threads, the
> server Xput is no higher than delayed counterparts. However, you can see,
> the server Xput raise rapidly with client number when delay time is 0.
> On the other hand, with small number clients, server Xput is reverse
> proportional to the the delay time. And with larger clients number,
> server Xput is proportional to delay time.
>
> I don't understand why small(1-3us, with nanosleep on of solaris) delay
> time would help?
>
> Some hypothesises are that apache server itself have some internals to
> slowdown greedy clients. Or Solaris did not schedule the server threads
> well enough to handle short request interval. Or, the greedy client
> consumed too much cpu time?
>
> I'd appreciate any suggestions/comments from you.
>
> -Min
>
> --
> Rapid keystrokes and painless deletions often leave a writer satisfied with
> work that is merely competent.
> -- "Writing Well" Donald Hall and Sven Birkerts
--
Rapid keystrokes and painless deletions often leave a writer satisfied with
work that is merely competent.
-- "Writing Well" Donald Hall and Sven Birkerts
/****************************************************************************/
/* Copyright 1997, Trustees of Boston University. */
/* All Rights Reserved. */
/* */
/* Permission to use, copy, or modify this software and its documentation */
/* for educational and research purposes only and without fee is hereby */
/* granted, provided that this copyright notice appear on all copies and */
/* supporting documentation. For any other uses of this software, in */
/* original or modified form, including but not limited to distribution in */
/* whole or in part, specific prior permission must be obtained from Boston */
/* University. These programs shall not be used, rewritten, or adapted as */
/* the basis of a commercial software or hardware product without first */
/* obtaining appropriate licenses from Boston University. Boston University */
/* and the author(s) make no representations about the suitability of this */
/* software for any purpose. It is provided "as is" without express or */
/* implied warranty. */
/* */
/****************************************************************************/
/* */
/* Author: Paul Barford */
/* modified by Min Xu */
/* Title: Client11s - HTTP/1.1 compliant Client thread with single */
/* pthread. */
/* Revision: 1.0 5/7/98 */
/* */
/* This is the part of the SURGE code which actually makes document */
/* requests. It requires Pthreads be installed in order to operate. */
/* See the SURGE-HOW-TO in order to compile and run properly. */
/* */
/* Surge is set up so that you can multiple Surge client processes can be */
/* run on a single system. This code used to be part of Surgeclient.c but */
/* has been separated so that multiple HTTP flavors can be supported. This */
/* is the code that supports the HTTP/1.1 flavor. This code is meant to */
/* be compiled as a module for Surgeclient.c. The two routine contained */
/* here are what actually make the requests for files from the server. */
/* The ClientThread routine is what is referred to as a user-entity. The */
/* ReaderThread routine is what actually makes a single document request. */
/* */
/* In order to support HTTP/1.1 this code supports multiple persistent */
/* connections. It supports both content length and chunked encoding to */
/* determine file end. It should be used with fewer threads per process */
/* since it will use up to MAXCONCTS per user entity for file transfers. */
/* */
/* This code is compiled in the Makefile as client11.c and compiles into */
/* client.o. It's HTTP/1.0 counterpart is called client10.c */
/* */
/* THESE ARE THE HTTP/1.1 COMPLIANT CLIENTTHREAD ROUTINES */
/* THAT INCLUDE THE ABILITY TO DO PIPELINED REQUESTS AND */
/* PIPELINE READING. */
/* */
/****************************************************************************/
#include <stdio.h>
#include <stdlib.h>
#include <errno.h>
#include <pthread.h>
#include <semaphore.h>
#include <sys/types.h>
#include <sys/time.h>
#include <sys/times.h>
#include <unistd.h>
#include <string.h>
#include <strings.h>
#include <assert.h>
#include "Surgeclient.h"
#include "client11s.h"
#include "magic_call.h"
void *ClientThread (void *);
void ReaderRead (void *);
int Checkconnect (void *);
void Writerequest (int, void *);
/* Each of the following routines and variables are defined in Surgeclient.c */
/* The variables in caps are #defined in Surgeclient.h */
extern int systemno;
extern long object[MATCHSIZE][OBJLIMIT];
extern long seq[NAMESIZE];
extern char httpstart[URLSIZE];
extern char httpend[URLSIZE];
extern int connectsrvr ();
extern void loadoutput (long, long, long, int, long, long, long, long);
extern int fillreadbuff (int, char *, size_t);
extern unsigned long numobjects;
extern unsigned long seq_counter;
extern unsigned long out_counter;
extern unsigned long cnt_counter;
extern unsigned long long *tpmc_counter;
extern long *cnt;
extern unsigned int threadnum;
extern unsigned int thinktime;
static struct timeval system_starttime;
static struct timeval system_endtime;
static struct timezone system_tz;
static unsigned long long last_sum = 0;
static unsigned long long last_reported_sum = 0;
static double last_sec = 0;
static void end_a_transaction(int thread_id) {
int i,j;
unsigned long long sum = 0;
double sec = 0;
tpmc_counter[thread_id] += 1;
if(thread_id == 0) {
for(i = 0 ; i < threadnum ; i++) {
sum += tpmc_counter[i];
}
/* report throughput */
/*
sum = tpmc_counter[thread_id];
if(sum % 1000 == 0) {
*/
/* make sure we don't call gettimeofday() too often */
if(sum - last_sum > 5000) {
gettimeofday (&system_endtime, &system_tz);
/*
printf("start %d:%d\n", system_starttime.tv_sec, system_starttime.tv_usec);
printf("end %d:%d\n", system_endtime.tv_sec, system_endtime.tv_usec);
*/
sec = system_endtime.tv_sec - system_starttime.tv_sec;
sec += (system_endtime.tv_usec - system_starttime.tv_usec)/1000000;
/*
printf("trans: %d\n", sum);
printf("secs: %f\n", sec);
*/
if (sec - last_sec > 60) {
printf("Client [%d] : %f sys_avg_xput %f sys_int_xput\n",
thread_id, sum/sec, (sum-last_reported_sum)/(sec-last_sec));
last_sec = sec;
last_reported_sum = sum;
/* crudely exit if runs for too long */
if(HOUR && sec > (HOUR*3600)) {
printf("Execution time is up!\n");
exit(0);
}
}
last_sum = sum;
}
}
end_transaction_magic_call ();
}
/****************************************************************************/
/* This routine calls for the connection set up and calls the document */
/* retriever routine as a separate thread. It is the first routine which */
/* is executed by the threads and is called from main. */
/****************************************************************************/
void *
ClientThread (void *threadno) {
int numobjs, id, numfiles;
long i, j, objindx, count;
int debugflag = 0;
struct readerdata reader;
/* Begin by initializing all of the variables used in the ClientThread */
id = *(int *) threadno; /* assign local thread id */
count = id; /* count is used to store the object index from OBJFIFO */
reader.conctno = -1;
/* multifacet: to be safe */
if(id == 0) {
gettimeofday (&system_starttime, &system_tz);
}
/* multifacet: get client start time after the last thread started */
if(id == threadnum-1) {
int i;
/* mark start of all the stats */
gettimeofday (&system_starttime, &system_tz);
for(i = 0 ; i < threadnum ; i++) {
tpmc_counter[i] = 0;
}
last_sum = 0;
last_reported_sum = 0;
last_sec = 0;
}
/* Loop until all of the objects in name.txt (OBJFIFO) have been requested */
while (1) {
int stopfileloop = 0;
/* multifacet specific:
* statically partition the request sequence amoung all threads
*/
count += threadnum;
if (count >= numobjects) {
/* to end */
break;
} else {
objindx = seq[count];
}
/*
if(count%1000 == 0) {
printf("SURGEclient %d, thread %d: Object Count %ld\n",systemno, id, count);
}
*/
/* For HTTP1.1 - Get the number of objects to read before closing the */
/* connections and startiang a new set of connections to server. */
/* I call this the "open connection" period. */
numobjs = 0; /* for HTTP1.1 number of objects to get before new concts */
if (numobjs == 0) {
/* multifacet specific */
numobjs = cnt[count];
if (debugflag) {
printf ("SURGEclient %d: Objs in open connect period = %d\n",
systemno, numobjs);
}
}
/* Get the file names associated with the selected object and request */
/* them from the server. If HTTP1.1 then use multiple connections */
/* For pipelined HTTP/1.1 requests, find out how many files are */
/* in the current object and then split these between the */
/* connections */
/* find out how many files */
numfiles = 0; /* Number of files in current object */
while (stopfileloop == 0) {
numfiles++;
if (object[objindx][numfiles] == -1 || numfiles == OBJLIMIT) {
stopfileloop = 1;
}
}
/* xu note:
* get all files using one connection;
* we don't use http pipeline here because Surge client is buggy in
* parsing the pipeline output, which resulting in closing connections
* unnecessarily. Also, being on the same machine, the latency between
* client and server do not make sense to use http pipeline.
*/
if (numfiles > 0) {
static struct timespec rqtp;
rqtp.tv_sec = thinktime / 1000000;
rqtp.tv_nsec = (thinktime % 1000000) * 1000;
/* Be sure server didn't close connection */
while (reader.conctno == -1) {
reader.conctno = connectsrvr ();
}
for (i = 0; i < numfiles; i++) {
reader.threadno = id;
reader.objectno = objindx;
reader.nofiles = 1;
reader.fileno[0] = i;
ReaderRead(&reader);
if( nanosleep(&rqtp, NULL) == -1) { /* sleep for a while */
perror("nanosleep");
}
}
} /* End if numfiles > 0 */
/* For HTTP 1.1: If objects for current "open connection" period */
/* have been transferred, then close all of open connections. numobjs */
/* tracks the number of objects for "open connection" period. If all */
/* objects have not been selected then be sure that connections are */
/* still open - if not, reopen them. */
if (numobjs == 1) {
/* If all objects xfered, close connects */
if (reader.conctno >= 0) {
if ((close (reader.conctno)) < 0) {
printf ("SURGEclient %d: Client %d Error closing connection %ld\n",
systemno, id, 0);
}
reader.conctno = -1;
if (debugflag) {
printf ("SURGEclient %d: Client %d Closed connection %ld\n",
systemno, id, 0);
}
}
numobjs = 0;
} else {
/* If all objects not xfered, be sure all connects still open */
Checkconnect (&reader);
numobjs--; /* decrement # objects in this "open connection" period */
}
} /* End object select while loop */
/* close all connections */
if (reader.conctno != -1) {
close(reader.conctno);
}
pthread_exit (0);
} /* End ClientThread */
/****************************************************************************/
/* This routine is where the action takes place. It sets up the HTTP */
/* command, issues the command, reads the data from the open connection and */
/* takes a time stamp when the transaction finishes and then loads the */
/* results array. For pipelined reads and writes: request the base file */
/* and then read it. Then request all of the remaining files and read them */
/****************************************************************************/
static char g_contlength[MAXTHREAD][20];
static char g_headbuff[MAXTHREAD][300];
static char g_rbuff[MAXTHREAD][READBUFFSZ];
void
ReaderRead (void *readdat) {
int i, c, thrdid, numfiles, filesread, stopflg, conctstat, conctid;
long objindx, fileindx[OBJLIMIT], length = 0;
char *contlength = NULL;
char *headbuff = NULL;
char *rbuff = NULL;
char parseline1[] = "chunked";
char parseline2[] = "Content-Length";
char parseline3[] = "\r\n\r\n";
char parseline4[] = "HTTP";
char *lineptr = NULL;
char *headptr = NULL;
int debugflag = 0;
/* Each time this function is called by the ClientThread it */
/* is responsible for requesting and reading one object from the */
/* server. Begin by loading the object data from the readdat structure */
thrdid = ((struct readerdata *) readdat)->threadno;
objindx = ((struct readerdata *) readdat)->objectno;
numfiles = ((struct readerdata *) readdat)->nofiles;
Checkconnect(readdat);
conctid = ((struct readerdata *) readdat)->conctno;
rbuff = g_rbuff[thrdid];
headbuff = g_headbuff[thrdid];
contlength = g_contlength[thrdid];
bzero(rbuff, READBUFFSZ);
bzero(headbuff, 300);
bzero(contlength, 20);
for (i = 0; i < numfiles; i++) {
fileindx[i] = ((struct readerdata *) readdat)->fileno[i];
if (debugflag) {
printf ("SURGEclient %d: Thread %d Object %ld Fileno %ld\n",
systemno, thrdid, objindx, object[objindx][fileindx[i]]);
}
}
/* Issue HTTP GET call for the specified file to the specified */
/* In the case of pipelined HTTP/1.1, request all files at once. */
filesread = 0;
Writerequest (filesread, readdat);
/* Read the file from the connection. For HTTP1.1 chunked data format */
/* is supported. So, read chunks until the "0 CRLF {footer CRLF} CRLF" */
/* is seen and then consider the transfer to be complete. */
/* For HTTP 1.1 must be able to read data from persistent concts */
/* Read data from connection into the rbuff and then parse to see if */
/* chunked encoding is being used - if yes then read until the "0 CRLF */
/* {footer CRLF} CRLF" is seen otherwise use the content length field */
/* and read that number of bytes of message and then end. By rfc2068 */
/* "Messages MUST NOT include both a Content-Length header field and */
/* the "chunked" transfer coding. If both are received, the */
/* Content-Length MUST be ignored." */
/* Start out by reading the first rbuff of data from the connection */
if(conctid == -1) { printf("Bad connection before fetch file\n"); };
c = fillreadbuff (conctid, rbuff, READBUFFSZ);
if(c == -1) { printf("Bad read of the first file\n"); };
if (debugflag) {
printf ("Read first buffer of data size = %d\n", c);
}
/* Find the start of first message and place lineprt where data starts */
if ((lineptr = strstr (rbuff, parseline3)) == NULL) {
/* cannot find message header */
printf (" Can't find first message start - aborting object transfer\n");
if (debugflag) {
printf("c=%d", c);
for(i=0; i<1000; i++) {
printf("%d", rbuff[i]);
}
for(i=0; i<1000; i++) {
printf("%c", rbuff[i]);
}
printf("rbuff=%s\n", rbuff);
assert(0);
printf ("Surgeclient %d: Thread %d Object %ld Fileno %ld",
systemno, thrdid, objindx, object[objindx][fileindx[filesread]]);
}
filesread = numfiles;
if (close (conctid) < 0) {
printf ("SURGEclient %d: Thrd %d Error closing connect %d\n",
systemno, thrdid, conctid);
}
((struct readerdata *) readdat)->conctno = -1;
} else {
/* Store header for pipelinig */
lineptr += 4; /* Place pointer after the CRLF CRLF in msg header */
if (!(300 > lineptr - rbuff)) { printf("Must increase headbuff size!\n"); };
strncpy (headbuff, rbuff, lineptr - rbuff);
}
/* For pipelined HTTP, read all of the files requested. Check after */
/* each file read that the connection is still opened. If not, reopen */
/* and resend requests for the remaining files. */
while (filesread < numfiles) {
/* Parse the header from first data pkt if chunked encoding is used */
if (strstr (headbuff, parseline1) != NULL) {
/* multifacet: apache don't use chucked encoding */
printf("Strange! Our workload should not use chucked encoding!\n");
if (debugflag) {
printf ("Surgeclient %d: Thread %d Object %ld Fileno %ld", systemno,
thrdid, objindx, object[objindx][fileindx[filesread]]);
printf (" Chunked data transfer\n");
}
/* Look for the end string in each packet - begin at body start of */
/* first packet and check the length of the next chunk. Then look */
/* for end string in subsequent chunks until the end of the packet */
/* Get additional packets of data when necessary until end string */
/* is found. */
stopflg = 0;
while (stopflg == 0 && c != -1) { /* Cycle through chunks until EOF */
if (length + lineptr - rbuff < c - 1) {
lineptr += length; /* Set lineptr to 1st char of next chnk */
i = 0;
bzero (contlength, 20);
while (rbuff[i + lineptr - rbuff] != '\n') { /* Read chunk size */
contlength[i] = rbuff[i + lineptr - rbuff];
i++;
/* If size is on array boundry get next packet of data */
if (i + lineptr - rbuff == READBUFFSZ) {
c = fillreadbuff (conctid, rbuff, READBUFFSZ);
i = 0;
if (debugflag > 1) {
printf ("Read1 chunk buffer of data size = %d\n", c);
}
}
} /* End while of reading next chunk size */
length = strtol (contlength, NULL, 16); /* Chunk size is hex */
if (debugflag > 1) {
printf ("Next chunk size = %ld hex %s\n", length, contlength);
}
lineptr += 3 + i; /* put lineptr at start of data of next chunk */
if (length == 0) { /* Look for CRLF CRLF string to end data. If */
lineptr -= 3 + i; /* its not in this rbuff the get one more. */
if ((lineptr = strstr (lineptr, parseline3)) == NULL) {
c = fillreadbuff (conctid, rbuff, READBUFFSZ);
lineptr = rbuff; /* put lineptr at start of rbuff */
if (debugflag > 1) {
printf ("Read2 chunk buffer of data size = %d\n", c);
}
}
if (c != -1) { /* Write out results if reading was successful */
/* Get the output sequence # and write results structure */
end_a_transaction(thrdid);
if (debugflag) {
printf ("Surgeclient %d: Thread %d Object %ld",
systemno, thrdid, objindx);
printf (" Finished reading file %ld\n",
object[objindx][fileindx[filesread]]);
}
/* Now get the header for the next file which can lay */
/* across a buffer boundry so be careful in constructing */
/* headbuff and place the lineptr at the start of the new */
/* data. NOTE: PIPELINE CHUNKS HAVE NOT BEEN TESTED!!!! */
if (filesread + 1 < numfiles) {
bzero (headbuff, 300);
headptr = lineptr; /* put headptr at start of next header */
if (lineptr >= rbuff + c) { /*Get new pkt if headptr at end */
/* Before we can get a new packet, we much check to */
/* see if the server has closed the connection. If */
/* so, reopen it and rerequest the remaining files */
if ((conctstat = Checkconnect (readdat)) == 1) {
Writerequest (filesread, readdat);
conctid = ((struct readerdata *) readdat)->conctno;
}
c = fillreadbuff (conctid, rbuff, READBUFFSZ);
headptr = rbuff;
if (debugflag) {
printf ("Header is in next packet\n");
printf ("Read3 chunk buffer of data size = %d\n", c);
}
} /* End if headptr at end of rbuff */
lineptr = headptr;
if ((lineptr = strstr (lineptr, parseline3)) != NULL) {
strncpy (headbuff, headptr, lineptr + 4 - headptr);
lineptr += 4; /* put lineptr at start of data */
if (debugflag) {
printf ("Header in Page\n");
}
} else { /* Header has gone over rbuff boundry */
i = c - (headptr - rbuff);
if (i > 300) i = 300; /* Be sure to keep the index within */
if (i < 0) i = 0; /* the length of headbuff */
strncpy (headbuff, headptr, i); /* Copy start of header */
c = fillreadbuff (conctid, rbuff, READBUFFSZ);
strncat (headbuff, rbuff, 300 - i); /* add data to headbuff */
if ((headptr = strstr (headbuff, parseline4)) != NULL) {
if ((headptr = strstr (headptr, parseline3)) != NULL) {
if ((lineptr = strstr (rbuff, parseline3)) != NULL) {
if (debugflag) {
printf ("Header over Boundary\n");
}
lineptr += 4;
} else {
c = -1;
if (debugflag) {
printf ("Error finding data start\n");
}
}
} else {
c = -1;
if (debugflag) {
printf ("Error finding next headr end\n");
}
}
} else {
c = -1;
if (debugflag) {
printf ("Error finding next headr start\n");
}
}
} /* End else if header has gone over rbuff boundry */
} /* End if filesread < numfiles */
} /* End if c != -1 to write results */
stopflg = 1; /* Chunk data end string found so end read */
} /* End if length == 0 */
} else { /* If chunk size is beyond current rbuff get next pkt */
length -= c - (lineptr - rbuff);
c = fillreadbuff (conctid, rbuff, READBUFFSZ);
lineptr = rbuff;
if (debugflag) {
printf ("Read chunk 4 buffer of data size = %d\n", c);
}
}
} /* End while of cycle through chunks */
} else if ((headptr = strstr (headbuff, parseline2)) != NULL) {
/********************************************************************/
/* content length is used to decide when to end */
/********************************************************************/
i = 0; /* A really ugly way to parse out "Content-Length: " */
bzero (contlength, 20);
while (headbuff[i + headptr + 16 - headbuff] >= '0' &&
headbuff[i + headptr + 16 - headbuff] <= '9' ) {
contlength[i] = headbuff[i + headptr + 16 - headbuff];
i++;
}
if (!(20 > i)) { printf("Must increase contlength buffer size!\n"); };
length = strtol (contlength, NULL, 10); /* Content length is decimal */
if (!(length > 0)) { printf("File length < 0??\n"); };
if (debugflag) {
printf ("Surgeclient %d: Thread %d Object %ld Fileno %ld", systemno,
thrdid, objindx, object[objindx][fileindx[filesread]]);
printf (" Content length used length = %ld\n", length);
}
/* Test to see if end of message is in current rbuff of data */
/* xu note: length does not include the header! */
length -= c - (lineptr - rbuff);
if (debugflag > 1) {
printf ("Data remaining after this packet = %ld\n", length);
}
/* If data still remains to be read, read until length = 0 */
while (length > 0 && c >= 0) {
c = fillreadbuff (conctid, rbuff, READBUFFSZ);
if (debugflag > 1) {
printf ("Length = %ld Value of next read is %d\n", length, c);
}
length -= c;
if (debugflag > 1) {
printf ("Read Cont 1 buffer of data size = %d\n", c);
}
}
if (c != -1) {
/* Get the output sequence # and write results structure */
end_a_transaction(thrdid);
if (debugflag) {
printf ("Surgeclient %d: Thread %d Object %ld",
systemno, thrdid, objindx);
printf (" Finished reading file %ld\n",
object[objindx][fileindx[filesread]]);
}
/* Now get the header for the next file which can lay */
/* across a buffer boundry so be careful in constructing */
/* headbuff and place the lineptr at the start of the new */
/* data. */
if (filesread + 1 < numfiles) {
if(!(length <= 0)) { printf("Length should be negtive\n"); };
bzero (headbuff, 300);
/* Put headptr at start of next (length < 0) */
headptr = rbuff + c + length;
if (headptr == rbuff + c) { /* Get new pkt if ptr at end of buff */
/* Before we can get a new packet, we must check to see if */
/* the server has closed the connection. If so, reopen it */
/* and rerequest the remaining files. */
if ((conctstat = Checkconnect (readdat)) == 1) {
Writerequest (filesread, readdat);
conctid = ((struct readerdata *) readdat)->conctno;
}
c = fillreadbuff (conctid, rbuff, READBUFFSZ);
if (c == -1) { printf("Error reading buffer contains next header\n"); };
headptr = rbuff; /* next packet the get it */
if (debugflag) {
printf ("Header is in next packet\n");
}
} /* End if headptr at end of rbuff */
if ((lineptr = strstr (headptr, parseline3)) != NULL) {
lineptr += 4; /* put lineptr at start of data */
if (!(300 > lineptr - headptr)) { printf("Need bigger headbuff\n"); };
strncpy (headbuff, headptr, lineptr - headptr);
if (debugflag) {
printf ("Header in Page\n");
}
} else { /* Header has gone over rbuff boundry */
i = c - (headptr - rbuff);
if (i > 299) i = 299; /* Be sure to keep the index with in */
if (i < 0) i = 0; /* the length of headbuff */
strncpy (headbuff, headptr, i); /* Copy start of header */
c = fillreadbuff (conctid, rbuff, READBUFFSZ);
if (c == -1) { printf("Bad read of next header buffer\n"); };
strncat (headbuff, rbuff, 299 - i); /* add data to headbuff */
if ((headptr = strstr (headbuff, parseline3)) != NULL) {
if ((lineptr = strstr (rbuff, parseline3)) != NULL) {
if (debugflag) {
printf ("Header over rbuff Boundary\n");
}
lineptr += 4;
} else {
if (debugflag) {
for(i = 0 ; i<READBUFFSZ; i++) {
printf("%c", rbuff[i]);
if(i>=10 && rbuff[i-10] == 0) break;
}
printf("\n");
for(i = 0 ; i<READBUFFSZ; i++) {
printf("%d", rbuff[i]);
if(i>=10 && rbuff[i-10] == 0) break;
}
printf("c=%d", c); printf("\nheadbuff=%s", headbuff);
printf("\nrbuff=%s", rbuff); fflush(stdout); assert(0);
}
printf ("Error finding data start\n");
c = -1;
}
} else {
printf ("Error finding next header\n");
c = -1;
if (debugflag) {
printf("c=%d", c); printf("\nheadbuff=%s", headbuff);
printf("\nrbuff=%s", rbuff); fflush(stdout); assert(0);
}
}
}
} else { /* End if filesread + 1 < numfiles */
if(!(length == 0)) {
printf("Trailing data\n");
c = -1;
}
}
} else { /* End if c != -1 to write results */
printf("Error during fetch rest of the file!\n");
}
} else {
if (debugflag) {
printf ("SURGEclient %d: Thread %d Objct %ld Fileno %ld", systemno,
thrdid, objindx, object[objindx][fileindx[filesread]]);
printf("c=%d", c); printf("\nheadbuff=%s", headbuff); printf("\nrbuff=%s",
rbuff); fflush(stdout); assert(0);
}
printf (" Can't find Chunk Size or Content Length - abort obj\n");
c = -1;
} /* End of length encoding selection */
/* If the last file was successfully read, increment the filesread */
/* counter and reset the start times to the last end time. If c = -1 */
/* then there was a real problem so close connect and rereques the */
/* remaining files. */
if (c != -1) { /* If c != -1 then the file was successfully read */
filesread++; /* Increment the # of files read from pipe */
} else {
if (debugflag) {
printf ("SURGEclient %d: Thread %d Objct %ld Fileno %ld", systemno,
thrdid, objindx, object[objindx][fileindx[filesread]]);
}
filesread = numfiles;
if (close (conctid) < 0) {
printf ("SURGEclient %d: Thrd %d Error closing connect %d\n", systemno,
thrdid, conctid);
}
((struct readerdata *) readdat)->conctno = -1;
}
} /* End while filesread < numfiles for pipelined reading of an object */
} /* End ReaderRead */
/****************************************************************************/
/* This routine checks the TCP connection for the specified reader. If the */
/* connection is closed, it reopens it. It returns a 0 if it did not have */
/* to reopen the connection, it returns a 1 if it did have to reopen. */
/* The inputs to this routine are the client thread number and the reader */
/* number which specify which connection to check. */
/****************************************************************************/
int
Checkconnect (void *readdat)
{
int conctid, thrdid;
fd_set fdvar;
struct timeval timeout;
int debugflag = 0;
conctid = ((struct readerdata *) readdat)->conctno;
thrdid = ((struct readerdata *) readdat)->threadno;
timeout.tv_sec = 0; /* Return immediately with value from select */
timeout.tv_usec = 0;
if (conctid == -1)
{ /* conct can be set to -1 when aborted in ReaderThread */
while (conctid == -1) /* Reopen connect */
{
conctid = connectsrvr ();
}
((struct readerdata *) readdat)->conctno = conctid;
return (1); /* Did have to reopen connection */
}
else
{ /* If connect is supposed to be open, use select to test status */
FD_ZERO (&fdvar);
FD_SET (conctid, &fdvar);
if (select (conctid + 1, &fdvar, NULL, NULL, &timeout) > 0)
{
if ((close (conctid)) < 0)
printf ("SURGEclient %d: Error closing connect %d\n", systemno, conctid);
conctid = -1;
while (conctid == -1) /* Reopen connect */
{
conctid = connectsrvr ();
}
((struct readerdata *) readdat)->conctno = conctid;
if (debugflag)
{
printf ("SURGEclient %d: Thrd %d", systemno, thrdid);
printf (" Reopened conct %d\n", conctid);
}
return (1); /* Did have to reopen connection */
}
else
{
return (0); /* Didn't have to reopen connection */
}
}
} /* End Checkconnect */
/****************************************************************************/
/* This routine is used by the pipelined version of client11.c in order to */
/* wirte the multiple HTTP requests in a single buffer. It takes as input */
/* the reader structure which contains the file names, and where to start */
/* makeing requests of the files. */
/****************************************************************************/
void
Writerequest (int start, void *readdat)
{
int i, thrdid, conctid, numfiles;
long objindx;
char wbuff[WRITEBUFFSZ];
char objbuff[64];
char url[URLSIZE];
int debugflag = 0;
thrdid = ((struct readerdata *) readdat)->threadno;
objindx = ((struct readerdata *) readdat)->objectno;
numfiles = ((struct readerdata *) readdat)->nofiles;
conctid = ((struct readerdata *) readdat)->conctno;
wbuff[0] = 0;
/* Generate complete URL string by inserting file number in http string */
/* For pipeline requests, place all requests in the write buffer */
for (i = start; i < numfiles; i++) {
bzero (url, URLSIZE);
strcpy (url, httpstart);
sprintf (objbuff, "%ld",
object[objindx][((struct readerdata *) readdat)->fileno[i]]);
strncat (url, objbuff, sizeof (objbuff));
strncat (url, httpend, sizeof (httpend));
if (i == 0) {
strcpy (wbuff, url);
} else {
strncat (wbuff, url, sizeof (url));
}
if (!(strlen(wbuff) < WRITEBUFFSZ)) { printf("Consider bigger writebuff\n"); };
}
if (debugflag) {
printf ("SURGEclient %d: Thrd %d Object %ld",
systemno, thrdid, objindx);
printf (" Requested fileidxs %d to %d\n", start, numfiles);
}
/* Issue HTTP GET call for the specified file to the specified connectin */
write (conctid, wbuff, strlen (wbuff));
} /* End Writerequest */
/****************************************************************************/
/* Copyright 1997, Trustees of Boston University. */
/* All Rights Reserved. */
/* */
/* Permission to use, copy, or modify this software and its documentation */
/* for educational and research purposes only and without fee is hereby */
/* granted, provided that this copyright notice appear on all copies and */
/* supporting documentation. For any other uses of this software, in */
/* original or modified form, including but not limited to distribution in */
/* whole or in part, specific prior permission must be obtained from Boston */
/* University. These programs shall not be used, rewritten, or adapted as */
/* the basis of a commercial software or hardware product without first */
/* obtaining appropriate licenses from Boston University. Boston University*/
/* and the author(s) make no representations about the suitability of this */
/* software for any purpose. It is provided "as is" without express or */
/* implied warranty. */
/* */
/****************************************************************************/
/* */
/* Author: Paul Barford */
/* modified by Min Xu */
/* Title: Surgeclient header file */
/* Revision: 1.0 5/7/98 */
/* */
/* With the HTTP/1.1 compliant version of SURGE, it is necessary to make */
/* a separate header file with variable definitions that may be used by */
/* the clientXX.c files. */
/* */
/* This is part of the client that use a single client thread for each */
/* client. */
/* Currently this file is identical to client11p.h. */
/* */
/****************************************************************************/
/****************************************************************************/
/* These are the values which you might want to alter for each SURGE test. */
/* These will go into a config file at some point. */
/* NOTE: You can be sure that you have your SIZE values properly set by */
/* insuring that NAMESIZE >= wc objout.txt and OUTSIZE >= total requests for*/
/* files (shown after running zipf.c) and MATCHSIZE >= wc mout.txt and */
/* OBJLIMIT >= LIMIT from object.c and URLSIZE >= max size of all chars */
/* which will make up any HTTP command string. READBUFFSZ is the size of */
/* the buffer that accepts data from the server. */
/****************************************************************************/
#define READBUFFSZ 8192 /* size of buffer that data is read into */
#define WRITEBUFFSZ 8192 /* size of buffer that requests are written */
/****************************************************************************/
/* Define the structures used for passing values to the reader routine */
/****************************************************************************/
struct readerdata { /* This is the data passed to each reader thread */
int conctno; /* Socket connection for this reader */
int threadno; /* calling thread id */
int nofiles; /* number of files to be retrieved in pipeline */
long objectno; /* index of object selected */
long fileno[OBJLIMIT]; /* index of file selected */
};
