Author: rjung Date: Tue May 16 15:52:39 2006 New Revision: 407096 URL: http://svn.apache.org/viewcvs?rev=407096&view=rev Log: Adding attribute "distance" to balanced workers.
Express Preferences between the balanced workers of an lb worker. A load balancer will never choose some balanced worker in case there is another usable worker with lower distance. Only in case all workers below a given distance are in error, disabled or stopped, workers of a larger distance are eligible for balancing. Modified: tomcat/connectors/trunk/jk/native/TODO tomcat/connectors/trunk/jk/native/common/jk_lb_worker.c tomcat/connectors/trunk/jk/native/common/jk_lb_worker.h tomcat/connectors/trunk/jk/native/common/jk_shm.h tomcat/connectors/trunk/jk/native/common/jk_status.c tomcat/connectors/trunk/jk/native/common/jk_util.c tomcat/connectors/trunk/jk/xdocs/config/workers.xml Modified: tomcat/connectors/trunk/jk/native/TODO URL: http://svn.apache.org/viewcvs/tomcat/connectors/trunk/jk/native/TODO?rev=407096&r1=407095&r2=407096&view=diff ============================================================================== --- tomcat/connectors/trunk/jk/native/TODO (original) +++ tomcat/connectors/trunk/jk/native/TODO Tue May 16 15:52:39 2006 @@ -2,26 +2,14 @@ $Id$ -1) Counter types in shm -======================= +1) Optimize "distance" +====================== -Why are elected and errors of type size_t? +Sorting the list of balanced workers by distance would be nice, but: +How to combine the sorting with the offset implementation (especially +useful for strategy BUSYNESS under low load). - /* Number of times the worker was elected */ - volatile size_t elected; - /* Number of non 200 responses */ - volatile size_t errors; - -Shouldn't they be jk_uint64_t? - -2) Implement "distance" -======================= - -Add a new attribut "distance" to balanced workers. -Without session Id or stickyness the lb worker will -choose between al usable workers of the smallest distance. - -3) Reduce number of string comparisons in most_suitable +2) Reduce number of string comparisons in most_suitable ======================================================== a) redirect/domains @@ -50,7 +38,7 @@ Could we use hashes instead of string comparisons all the time? I'm not sure, if a good enough hash takes longer than a string comparison though. -4) Optimization of JK_WORKER_USABLE +3) Optimization of JK_WORKER_USABLE ==================================== We use that one quite a lot. Since it is now a combination of four @@ -63,7 +51,7 @@ every time we change one of the four other flags). But in terms of performance that happens rarely. -5) Code separation between factory, validate and init in lb +4) Code separation between factory, validate and init in lb ============================================================ The factory contains: @@ -88,18 +76,23 @@ p->lb_workers[i].s->maintain_time = time(NULL); -6) Refactor Logging +5) Refactor Logging ==================== a) Use the same code files for the request logging functions in Apache 1.3 and 2.0. b) Use the same code files for piped logging in Apache 1.3 and 2.0. -7) ajpget +6) ajpget ========== Combine ajplib and Apache ab to an ajp13 commandline client ajpget. +7) Manage lb method and locking via jk_status +============================================= + +It's not yet contained in the shm. + 8) Parsing workers.properties ============================= @@ -108,7 +101,7 @@ no information will be logged, e.g. when attributes contain typos. 9) Persisting workers.properties -================================= +================================ Make workers.properties persist from inside status worker. @@ -154,7 +147,7 @@ to be generated at the end or directly after service. 11) Access/Modification Time in shm -================================== +=================================== a) [Discussion] What will this generally be used for? At the moment, only jk_status "uses" it, but it only sets the values, it never asks for them. @@ -173,7 +166,7 @@ or the second call in jk_status coulkd be removed. 12) "Destroy" functionality -========================== +=========================== [Hint] Destroy on a worker never seems to free shm, but I think that was already a flaw without shm. Modified: tomcat/connectors/trunk/jk/native/common/jk_lb_worker.c URL: http://svn.apache.org/viewcvs/tomcat/connectors/trunk/jk/native/common/jk_lb_worker.c?rev=407096&r1=407095&r2=407096&view=diff ============================================================================== --- tomcat/connectors/trunk/jk/native/common/jk_lb_worker.c (original) +++ tomcat/connectors/trunk/jk/native/common/jk_lb_worker.c Tue May 16 15:52:39 2006 @@ -95,28 +95,17 @@ JK_TRACE_EXIT(l); } -/* Get the correct lb_value when recovering/starting/enabling a worker. - * This function needs to be externally synchronized! +/* Reset all lb values. */ -jk_uint64_t restart_value(lb_worker_t *p, jk_logger_t *l) +void reset_lb_values(lb_worker_t *p, jk_logger_t *l) { int i = 0; - jk_uint64_t curmax = 0; JK_TRACE_ENTER(l); if (p->lbmethod != JK_LB_BYBUSYNESS) { for (i = 0; i < p->num_of_workers; i++) { - if (p->lb_workers[i].s->lb_value > curmax) { - curmax = p->lb_workers[i].s->lb_value; - } + p->lb_workers[i].s->lb_value = 0; } } - if (JK_IS_DEBUG_LEVEL(l)) - jk_log(l, JK_LOG_DEBUG, - "restarting worker with lb_value %" - JK_UINT64_T_FMT, - curmax); - JK_TRACE_EXIT(l); - return curmax; } /* Retrieve the parameter with the given name */ @@ -364,6 +353,7 @@ jk_logger_t *l) { unsigned int i; + int d = 0; jk_uint64_t curmin = 0; worker_record_t *candidate = NULL; @@ -378,9 +368,12 @@ * not in error state, stopped, disabled or busy. */ if (JK_WORKER_USABLE(p->lb_workers[i].s)) { - if (!candidate || p->lb_workers[i].s->lb_value < curmin) { + if (!candidate || p->lb_workers[i].s->distance < d || + (p->lb_workers[i].s->lb_value < curmin && + p->lb_workers[i].s->distance == d)) { candidate = &p->lb_workers[i]; curmin = p->lb_workers[i].s->lb_value; + d = p->lb_workers[i].s->distance; } } } @@ -400,6 +393,7 @@ unsigned int i; unsigned int j; unsigned int offset; + int d = 0; jk_uint64_t curmin = 0; /* find the least busy worker */ @@ -415,9 +409,12 @@ * not in error state, stopped, disabled or busy. */ if (JK_WORKER_USABLE(p->lb_workers[i].s)) { - if (!candidate || (p->lb_workers[i].s->lb_value < curmin)) { + if (!candidate || p->lb_workers[i].s->distance < d || + (p->lb_workers[i].s->lb_value < curmin && + p->lb_workers[i].s->distance == d)) { candidate = &p->lb_workers[i]; curmin = p->lb_workers[i].s->lb_value; + d = p->lb_workers[i].s->distance; next_offset = i + 1; } } @@ -914,6 +911,8 @@ if (p->lb_workers[i].s->lb_factor < 1) { p->lb_workers[i].s->lb_factor = 1; } + p->lb_workers[i].s->distance = + jk_get_distance(props, worker_names[i]); if ((s = jk_get_worker_jvm_route(props, worker_names[i], NULL))) strncpy(p->lb_workers[i].s->jvm_route, s, JK_SHM_STR_SIZ); else Modified: tomcat/connectors/trunk/jk/native/common/jk_lb_worker.h URL: http://svn.apache.org/viewcvs/tomcat/connectors/trunk/jk/native/common/jk_lb_worker.h?rev=407096&r1=407095&r2=407096&view=diff ============================================================================== --- tomcat/connectors/trunk/jk/native/common/jk_lb_worker.h (original) +++ tomcat/connectors/trunk/jk/native/common/jk_lb_worker.h Tue May 16 15:52:39 2006 @@ -57,6 +57,13 @@ /* The exponent x is JK_LB_DECAY_MULT*#MAINT_INTV_SINCE_LAST_MAINT */ #define JK_LB_DECAY_MULT (1) +static const char *lb_locking_type[] = { + JK_LB_LM_DEFAULT, + JK_LB_LM_PESSIMISTIC, + "unknown", + NULL +}; + static const char *lb_method_type[] = { JK_LB_METHOD_REQUESTS, JK_LB_METHOD_TRAFFIC, @@ -97,7 +104,7 @@ int JK_METHOD lb_worker_factory(jk_worker_t **w, const char *name, jk_logger_t *l); -jk_uint64_t restart_value(lb_worker_t *p, jk_logger_t *l); +void reset_lb_values(lb_worker_t *p, jk_logger_t *l); void update_mult(lb_worker_t * p, jk_logger_t *l); #ifdef __cplusplus Modified: tomcat/connectors/trunk/jk/native/common/jk_shm.h URL: http://svn.apache.org/viewcvs/tomcat/connectors/trunk/jk/native/common/jk_shm.h?rev=407096&r1=407095&r2=407096&view=diff ============================================================================== --- tomcat/connectors/trunk/jk/native/common/jk_shm.h (original) +++ tomcat/connectors/trunk/jk/native/common/jk_shm.h Tue May 16 15:52:39 2006 @@ -71,6 +71,8 @@ char domain[JK_SHM_STR_SIZ+1]; /* worker redirect route */ char redirect[JK_SHM_STR_SIZ+1]; + /* worker distance */ + volatile int distance; /* current status of the worker */ volatile int is_disabled; volatile int is_stopped; @@ -96,13 +98,13 @@ /* Number of bytes transferred to remote */ volatile jk_uint64_t transferred; /* Number of times the worker was elected */ - volatile size_t elected; + volatile jk_uint64_t elected; /* Number of non 200 responses */ - volatile size_t errors; + volatile jk_uint32_t errors; /* Number of recovery attempts */ - volatile size_t recoveries; + volatile jk_uint32_t recoveries; /* Number of recovery failures */ - volatile size_t recovery_errors; + volatile jk_uint32_t recovery_errors; }; typedef struct jk_shm_worker jk_shm_worker_t; Modified: tomcat/connectors/trunk/jk/native/common/jk_status.c URL: http://svn.apache.org/viewcvs/tomcat/connectors/trunk/jk/native/common/jk_status.c?rev=407096&r1=407095&r2=407096&view=diff ============================================================================== --- tomcat/connectors/trunk/jk/native/common/jk_status.c (original) +++ tomcat/connectors/trunk/jk/native/common/jk_status.c Tue May 16 15:52:39 2006 @@ -448,7 +448,7 @@ jk_puts(s, "</tr>\n</table>\n<br/>\n"); jk_puts(s, "<table><tr>" "<th>Name</th><th>Type</th><th>jvmRoute</th><th>Host</th><th>Addr</th>" - "<th>Stat</th><th>F</th><th>M</th><th>V</th><th>Acc</th><th>Err</th>" + "<th>Stat</th><th>D</th><th>F</th><th>M</th><th>V</th><th>Acc</th><th>Err</th>" "<th>Wr</th><th>Rd</th><th>Busy</th><th>Max</th><th>RR</th><th>Cd</th></tr>\n"); for (j = 0; j < lb->num_of_workers; j++) { worker_record_t *wr = &(lb->lb_workers[j]); @@ -472,11 +472,12 @@ wr->s->in_recovering, wr->s->is_busy), "</td>", NULL); + jk_printf(s, "<td>%d</td>", wr->s->distance); jk_printf(s, "<td>%d</td>", wr->s->lb_factor); jk_printf(s, "<td>%" JK_UINT64_T_FMT "</td>", wr->s->lb_mult); jk_printf(s, "<td>%" JK_UINT64_T_FMT "</td>", wr->s->lb_value); - jk_printf(s, "<td>%u</td>", wr->s->elected); - jk_printf(s, "<td>%u</td>", wr->s->errors); + jk_printf(s, "<td>%" JK_UINT64_T_FMT "</td>", wr->s->elected); + jk_printf(s, "<td>%" JK_UINT32_T_FMT "</td>", wr->s->errors); jk_putv(s, "<td>", status_strfsize(wr->s->transferred, buf), "</td>", NULL); jk_putv(s, "<td>", status_strfsize(wr->s->readed, buf), @@ -511,7 +512,9 @@ jk_puts(s, "<input type=\"hidden\" name=\"lb\" "); jk_printf(s, "value=\"%u\">\n</table>\n", i); - jk_puts(s, "<table>\n<tr><td>Load factor:</td><td><input name=\"wf\" type=\"text\" "); + jk_puts(s, "<table>\n<tr><td>Distance:</td><td><input name=\"wx\" type=\"text\" "); + jk_printf(s, "value=\"%d\"/></td><tr>\n", wr->s->distance); + jk_puts(s, "<tr><td>Load factor:</td><td><input name=\"wf\" type=\"text\" "); jk_printf(s, "value=\"%d\"/></td><tr>\n", wr->s->lb_factor); jk_puts(s, "<tr><td>Route Redirect:</td><td><input name=\"wr\" type=\"text\" "); jk_putv(s, "value=\"", wr->s->redirect, NULL); @@ -577,14 +580,15 @@ } /* Display legend */ jk_puts(s, "<hr/><table>\n" - "<tr><th>Name</th><td>Worker route name</td></tr>\n" + "<tr><th>Name</th><td>Worker name</td></tr>\n" "<tr><th>Type</th><td>Worker type</td></tr>\n" - "<tr><th>jvmRoute</th><td>Worker JVM Route</td></tr>\n" + "<tr><th>jvmRoute</th><td>Worker JVM route</td></tr>\n" "<tr><th>Addr</th><td>Backend Address info</td></tr>\n" "<tr><th>Stat</th><td>Worker status</td></tr>\n" - "<tr><th>F</th><td>Load Balancer Factor</td></tr>\n" - "<tr><th>M</th><td>Load Balancer Multiplicity</td></tr>\n" - "<tr><th>V</th><td>Load Balancer Value</td></tr>\n" + "<tr><th>D</th><td>Worker distance</td></tr>\n" + "<tr><th>F</th><td>Load Balancer factor</td></tr>\n" + "<tr><th>M</th><td>Load Balancer multiplicity</td></tr>\n" + "<tr><th>V</th><td>Load Balancer value</td></tr>\n" "<tr><th>Acc</th><td>Number of requests</td></tr>\n" "<tr><th>Err</th><td>Number of failed requests</td></tr>\n" "<tr><th>Wr</th><td>Number of bytes transferred/min</td></tr>\n" @@ -656,11 +660,12 @@ wr->s->in_recovering, wr->s->is_busy) ); + jk_printf(s, " distance=\"%d\"", wr->s->distance); jk_printf(s, " lbfactor=\"%d\"", wr->s->lb_factor); jk_printf(s, " lbmult=\"%" JK_UINT64_T_FMT "\"", wr->s->lb_mult); jk_printf(s, " lbvalue=\"%" JK_UINT64_T_FMT "\"", wr->s->lb_value); - jk_printf(s, " elected=\"%u\"", wr->s->elected); - jk_printf(s, " errors=\"%u\"", wr->s->errors); + jk_printf(s, " elected=\"%" JK_UINT64_T_FMT "\"", wr->s->elected); + jk_printf(s, " errors=\"%" JK_UINT32_T_FMT "\"", wr->s->errors); jk_printf(s, " transferred=\"%" JK_UINT64_T_FMT "\"", wr->s->transferred); jk_printf(s, " readed=\"%" JK_UINT64_T_FMT "\"", wr->s->readed); jk_printf(s, " busy=\"%u\"", wr->s->busy); @@ -747,14 +752,22 @@ jk_shm_lock(); wr->s->is_disabled = i; wr->s->is_stopped = j; - wr->s->lb_value = restart_value(lb, l); + reset_lb_values(lb, l); + /* unlock the shared memory */ + jk_shm_unlock(); if (i+j==0) { jk_log(l, JK_LOG_INFO, - "worker %s restarted in status worker with lb_value %" + "worker %s restarted in status worker" JK_UINT64_T_FMT, - wr->s->name, - wr->s->lb_value); + wr->s->name); } + } + i = status_int("wx", s->query_string, wr->s->distance); + if (wr->s->distance!=i) { + /* lock shared memory */ + jk_shm_lock(); + wr->s->distance = i; + reset_lb_values(lb, l); /* unlock the shared memory */ jk_shm_unlock(); } Modified: tomcat/connectors/trunk/jk/native/common/jk_util.c URL: http://svn.apache.org/viewcvs/tomcat/connectors/trunk/jk/native/common/jk_util.c?rev=407096&r1=407095&r2=407096&view=diff ============================================================================== --- tomcat/connectors/trunk/jk/native/common/jk_util.c (original) +++ tomcat/connectors/trunk/jk/native/common/jk_util.c Tue May 16 15:52:39 2006 @@ -57,6 +57,7 @@ #define SOCKET_KEEPALIVE_OF_WORKER ("socket_keepalive") #define RECYCLE_TIMEOUT_OF_WORKER ("recycle_timeout") #define LOAD_FACTOR_OF_WORKER ("lbfactor") +#define DISTANCE_OF_WORKER ("distance") /* deprecated directive. Use balance_workers instead */ #define BALANCED_WORKERS ("balanced_workers") #define BALANCE_WORKERS ("balance_workers") @@ -82,6 +83,7 @@ #define WORKER_MAINTAIN_PROPERTY_NAME ("worker.maintain") #define DEFAULT_MAINTAIN_TIME (60) #define DEFAULT_LB_FACTOR (1) +#define DEFAULT_DISTANCE (0) #define LOG_FORMAT ("log_format") #define TOMCAT32_BRIDGE_NAME ("tomcat32") @@ -748,6 +750,19 @@ return jk_map_get_int(m, buf, DEFAULT_LB_FACTOR); } +int jk_get_distance(jk_map_t *m, const char *wname) +{ + char buf[1024]; + + if (!m || !wname) { + return DEFAULT_DISTANCE; + } + + MAKE_WORKER_PARAM(DISTANCE_OF_WORKER); + + return jk_map_get_int(m, buf, DEFAULT_DISTANCE); +} + int jk_get_is_sticky_session(jk_map_t *m, const char *wname) { int rc = JK_TRUE; @@ -781,7 +796,7 @@ char buf[1024]; const char *v; if (!m || !wname) { - return DEFAULT_LB_FACTOR; + return JK_LB_BYREQUESTS; } MAKE_WORKER_PARAM(METHOD_OF_WORKER); Modified: tomcat/connectors/trunk/jk/xdocs/config/workers.xml URL: http://svn.apache.org/viewcvs/tomcat/connectors/trunk/jk/xdocs/config/workers.xml?rev=407096&r1=407095&r2=407096&view=diff ============================================================================== --- tomcat/connectors/trunk/jk/xdocs/config/workers.xml (original) +++ tomcat/connectors/trunk/jk/xdocs/config/workers.xml Tue May 16 15:52:39 2006 @@ -373,6 +373,16 @@ </p> </directive> +<directive name="distance" default="0" required="false"> +Express Preferences between the balanced workers of an lb worker. +A load balancer will never choose some balanced worker +in case there is another usable worker with lower distance. +<p> +Only in case all workers below a given distance are in error, disabled or stopped, +workers of a larger distance are eligible for balancing. +</p> +</directive> + <directive name="domain" default="" required="false"> Domain directive can be used only when the worker is a member of the load balancer. Workers that share the same domain name are treated as single worker. If sticky_session --------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]