Re: Question about new mod_jk jk_lb_worker.c
I would be pleased to use rotatelogs, but it doesn't work with mod_jk.log (at least not last time I checked). David Rees wrote: Rainer Jung wrote, On 11/14/2004 7:31 AM: 0) Any ideas on rotating the mod_jk log file? Use cronolog http://cronolog.org/ or the rotatelogs program included with Apache. -Dave - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Question about new mod_jk jk_lb_worker.c
Unfortunately I don't understand. Trying to combine rotatelogs with mod_jk.log in the following way (apache 1.3): JkLogFile |/home/jung/mkb/apache/bin/rotatelogs /home/jung/mkb/apache/logs/mod_jk.log 600 doesn't work. It does neither log, nor spawn a rotatelogs process. Inside apaches CustomLog rotatelogs works, but there is no obvious way to combine CustomLog with JkLogFile. Mladen Turk wrote: David Rees wrote: Rainer Jung wrote, On 11/14/2004 7:31 AM: 0) Any ideas on rotating the mod_jk log file? Use cronolog http://cronolog.org/ or the rotatelogs program included with Apache. For that (or any kind of rotatelogs) we would need to spawn the separate process that'll do the logging. The only reasonable solution is to use the 'native' logging and then use rotatelogs program. MT. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Question about new mod_jk jk_lb_worker.c
Rainer Jung wrote: Unfortunately I don't understand. Trying to combine rotatelogs with mod_jk.log in the following way (apache 1.3): JkLogFile |/home/jung/mkb/apache/bin/rotatelogs /home/jung/mkb/apache/logs/mod_jk.log 600 doesn't work. It does neither log, nor spawn a rotatelogs process. Of course it doesn't. As said: The only reasonable solution is to use the 'native' logging and then use rotatelogs program.. It means that it'll need to be done to support the native logger like JK2 does. Seems like 1.2.8 feature to me, cause it's a lot backporting. That way the request logging will go to access.log and error logging to error.log, and youl'll be able to use rotatelogs on them. MT. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Question about new mod_jk jk_lb_worker.c
Rainer Jung wrote, On 11/16/2004 1:11 AM: Unfortunately I don't understand. Trying to combine rotatelogs with mod_jk.log in the following way (apache 1.3): Doh. I've been using Apache 2 for so long I forgot that it doesn't work with Apache 1.3. It does work using Apache 2 / mod_jk. -Dave - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Question about new mod_jk jk_lb_worker.c
David Rees wrote: Rainer Jung wrote, On 11/14/2004 7:31 AM: 0) Any ideas on rotating the mod_jk log file? Use cronolog http://cronolog.org/ or the rotatelogs program included with Apache. For that (or any kind of rotatelogs) we would need to spawn the separate process that'll do the logging. The only reasonable solution is to use the 'native' logging and then use rotatelogs program. MT. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Question about new mod_jk jk_lb_worker.c
Hi Mladen, 0) Any ideas on rotating the mod_jk log file? 4) Open Problem This should work now with the latest patches. Excellent! Actually I tried to understand the new principles. Using an old style paper computer I can see that the values for lb_value are periodic. But I must confirm, that I did not mathematically understand the algorithm, maybe due to being a little ill. Are the mathematically foundations behind that simple algorithm described anywhere? 1) Limiting new application sessions if load is to high. There is a problem with that. I made a implementation counting the number of busy childs/threads from scoreboard (took me entire day), but again we should count the number of connections to tomcat, cause the apache might be serving static content. Anyhow the idea is great and I'll implement it in the new mod_proxy for Apache 2.2 where we have extra slots in the scoreboard. Sad but we can not do that inside mod_jk unless we implement our own shared memory, that was prover to be bogus in jk2. Apache serving static content is not a problem according to our experience. Static content usually serves in very well under a second (depending mostly on internet speed). The idea here is to detect a problem with the application getting slow, e.g. because of backend systems not responding fast enough. In this situation we want to limit creation of new sessions. Counting the static requests doesn't really matter for us. Example: During normal operation there are 5 static requests in work and 10 dynamic ones (that take much longer to complete). When there is a problem with backend systems we will have 15 static ones, but more than 100 dynamic ones. So either counting or ommiting the static ones seems to make no big difference. You might want to take a look at the attached patch patch_overload.txt for jk/native/apache1.3/mod_jk.c version 1.52. I don't have a patch for Apache 2. The Patch places every change inside #ifdef OVERLOAD. Also there are some DEBUG-Log-Statements put inside #ifdef DEBUG which I assume can now be done more consistently with your TRACE features. Of course the best observable value would be the number of requests belonging to the same webapp. So some possible enhancement would be to count only requests with a fix URL prefix (that's not contained in the patch). 2) Multi-Cluster-Routing Can you write some use case for that and perhaps some simple algo too. What about sticky-sessions and forcing failower if we do not have session replication? Use case: Enterprise application with redundant internet connections A and B. A consists of two Apache instances A.a1 und A.a2, B of B.a1 and B.a2. Behind are 4 Tomcat A.t1, A.t2, B.t1, B.t2. A.t1 and A.t2 are clustered, B.t1 and B.t2 are clustered. mod_jk uses load balancing with sticky sessions. All Apaches can connect to any Tomcat, but A.t1 is local for A.a1, A.t2 for A.a2, B.t1 for B.a1 and B.t2 for B.a2: A.a1 A.a2 B.a1 B.a2 || X || X || X || A.t1---A.t2 B.t1---B.t2 A.t1 and A.t2 are put into the same domain A in workers.properties, B.t1 and B.t2 are put into the same domain B in workers.properties. Now if you shutdown e.g. tomcat B.t1 for service/update (or if it breaks) All apaches will know from the domain configuration, that sticky requests for B.t1 have to go to B.t2. This is important, since only on that tomcat the replicated sessions from B.t1 will exist. Without domains you have to put all the Tomcats in one cluster. But then all sessions are replicated to all tomcats. We have a production side using 3x3=9 tomcats and a cluster with 9 nodes would mean too much overhead. The implementation uses and additional worker attribute domain, and if a session has a jvmRoute, is sticky and the correct worker is in error state another worker with the same domain attribute is chosen. I have an implementation, but I have to adapt to your code changes. The patch will only concern common/jk_util.c for configuration and common/jk_lb_worker.c. I could provide the code, if you are interested. For applications using sessions but without session replication I don't see a way how to successfully fail over. Maybe I don't understand that part of your question? 3) Idle connection disconnect Use worker mpm. We just can not make maintainer thread for non-treaded mpm's like apache1.2 or prefork. I don't use a seperate thread. I implemented this feature for Apache 1.3. The principles are the following (and I can sent complete code if you are interested): a) In common/service.h in jk_worker define an additional JK_METHOD check to be implemented by the workers: /* * Check all workers on a regular basis, * even if they do not process requests. */ int (JK_METHOD *check)(jk_worker_t *w, int force, jk_logger_t *l); b) In common/jk_worker.c provide an entry point wc_check, that will call the check method for all workers in the worker_map:
Re: Question about new mod_jk jk_lb_worker.c
Forgot to attach the patch for the overload feature. It is attached now.*** mod_jk.c.1.52 Sun Nov 14 15:00:20 2004 --- mod_jk.c.1.52.overload Sun Nov 14 15:18:59 2004 *** *** 38,43 --- 38,46 #include util_script.h #include util_date.h #include http_conf_globals.h + #ifdef OVERLOAD + #include scoreboard.h + #endif /* OVERLOAD */ /* * Jakarta (jk_) include files *** *** 63,68 --- 66,75 #define JK_DURATION (jakarta.worker.duration) #define JK_MAGIC_TYPE (application/x-jakarta-servlet) #define NULL_FOR_EMPTY(x) ((x !strlen(x)) ? NULL : x) + #ifdef OVERLOAD + #define EMPTY_FOR_NULL(x) ((x) ? x : ) + #define NEITHER_NULL_NOR_EMPTY(x) (x x[0] != '\0') + #endif /* OVERLOAD */ /* * If you are not using SSL, comment out the following line. It will make *** *** 134,139 --- 141,170 int envvars_in_use; table *envvars; + #ifdef OVERLOAD + /* + * Configuration object for the mod_overload module. Parameters are + * + * overload_uri URI that triggers load check, e.g. starting + * URI for a new application session + * overload_uri_regexp URI RegExp that triggers load check, e.g. starting + * URI pattern for a new application session + * overload_uri_match String representation of overload_uri_match + * overload_max_busy_slots maximum number of busy children allowed + * when doing load check + * overload_error_page URI of the error page shown (or redirected to), + * if there are too many children busy during load check + * overload_is_external Flag to indicate, that overload redirect should + * be done externally insted of an internal redirect + */ + char *overload_uri; + char *overload_uri_match; + regex_t *overload_uri_regexp; + int overload_max_busy_slots; + char *overload_error_page; + int overload_is_external; + #endif /* OVERLOAD */ + server_rec *s; } jk_server_conf_t; *** *** 1404,1409 --- 1435,1556 return NULL; } + #ifdef OVERLOAD + /* + * JkOverloadURI directive Handling + * + * Take this config parameter only, if the string is not null and not empty + */ + static const char *set_overload_uri(cmd_parms *cmd, + void *dummy, + char *uri) + { + jk_server_conf_t *conf = + (jk_server_conf_t *)ap_get_module_config(cmd-server-module_config, + jk_module); + + if ( NEITHER_NULL_NOR_EMPTY(uri) ) { + conf-overload_uri = uri; + } + + return NULL; + } + + /* + * JkOverloadURIMatch directive Handling + * + * Take this config parameter only, if the string is not null and not empty + */ + static const char *set_overload_uri_match(cmd_parms *cmd, + void *dummy, + char *uri_match) + { + jk_server_conf_t *conf = + (jk_server_conf_t *)ap_get_module_config(cmd-server-module_config, + jk_module); + + if ( NEITHER_NULL_NOR_EMPTY(uri_match) ) { + conf-overload_uri_regexp = ap_pregcomp(cmd-pool, uri_match, REG_EXTENDED); + if (conf-overload_uri_regexp == NULL) { + ap_log_error(APLOG_MARK, APLOG_NOERRNO|APLOG_DEBUG, cmd-server, + Overload configure: + could not compile regexp %s, + uri_match); + conf-overload_uri_match = NULL; + conf-overload_uri_regexp = NULL; + } else { + conf-overload_uri_match = uri_match; + } + } + + return NULL; + } + + /* + * JkOverloadMaxBusySlots Directive Handling + * + * Take this config parameter only, if the string is not null and not empty. + * We just take atoi of the string, so the integer value is the initial integer + * in the string + */ + static const char *set_overload_max_busy_slots(cmd_parms *cmd, + void *dummy, + char *max_busy_slots) + { + jk_server_conf_t *conf = + (jk_server_conf_t *)ap_get_module_config(cmd-server-module_config, + jk_module); + + if ( NEITHER_NULL_NOR_EMPTY(max_busy_slots) ) { + conf-overload_max_busy_slots = atoi(max_busy_slots); + } + + return NULL; + } + + /* + * JkOverloadErrorPage Directive Handling + * + * Take this config parameter only, if the string is not null and not empty. + */ + static
Re: Question about new mod_jk jk_lb_worker.c
Rainer Jung wrote: Hi Mladen, 0) Any ideas on rotating the mod_jk log file? Implementing inside apache's error.log is the only solution. But that'll have to wait for 1.2.8. Are the mathematically foundations behind that simple algorithm described anywhere? The idea behind this scheduler is the following: lbfactor is how much we expect this worker to work, or the worker's work quota. lbstatus is how urgent this worker has to work to fulfill its quota of work. We distribute each worker's work quota to the worker, and then look which of them needs to work most urgently (biggest lbstatus). This worker is then selected for work, and its lbstatus reduced by the total work quota we distributed to all workers. Thus the sum of all lbstatus does not change.(*) If some workers are disabled, the others will still be scheduled correctly. If a balancer is configured as follows: worker abcd lbfactor 25 25 25 25 lbstatus 0000 And b gets disabled, the following schedule is produced: lbstatus -500 25 25 lbstatus -250 -25 50 lbstatus 0000 (repeat) That is it schedules: a c d a c d a c d ... The following asymmetric configuration works as one would expect: worker ab lbfactor 70 30 lbstatus -30 30 lbstatus 40 -40 lbstatus 10 -10 lbstatus -20 20 lbstatus -50 50 lbstatus 20 -20 lbstatus -10 10 lbstatus -40 40 lbstatus 30 -30 lbasatus 00 (repeat) That is after 10 schedules, the schedule repeats and 7 a are selected with 3 b interspersed. 1) Limiting new application sessions if load is to high. Of course the best observable value would be the number of requests belonging to the same webapp. So some possible enhancement would be to count only requests with a fix URL prefix (that's not contained in the patch). Can you open a bugzilla report and enter those patches so they don't get lost. I very much like the idea and I have apache2 code for counting busy workers: int server_limit, thread_limit; /* Get the number of busy workers from scoreboard */ static int jk_server_busy_workers() { int i, j, res; int busy = 0; worker_score *ws_record; process_score *ps_record; if (!ap_exists_scoreboard_image()) { return 0; } for (i = 0; i server_limit; ++i) { ps_record = ap_get_scoreboard_process(i); for (j = 0; j thread_limit; ++j) { int indx = (i * thread_limit) + j; ws_record = ap_get_scoreboard_worker(i, j); res = ws_record-status; if (!ps_record-quiescing ps_record-pid) { if (res == SERVER_READY res != SERVER_DEAD res != SERVER_STARTING res != SERVER_IDLE_KILL) busy++; } } } return busy; } inside jk_post_config +ap_mpm_query(AP_MPMQ_HARD_LIMIT_THREADS, thread_limit); +ap_mpm_query(AP_MPMQ_HARD_LIMIT_DAEMONS, server_limit); 3) Idle connection disconnect Use worker mpm. We just can not make maintainer thread for non-treaded mpm's like apache1.2 or prefork. I don't use a seperate thread. I implemented this feature for Apache 1.3. The principles are the following (and I can sent complete code if you are interested): You are running that on unix or windows? a) In common/service.h in jk_worker define an additional JK_METHOD check to be implemented by the workers: Please, do another bugzilla entry for that. http://issues.apache.org/bugzilla/enter_bug.cgi?product=Tomcat%205 Component: Native:JK Severity: Enhancement Thank's for the discussion!! Thank you! Regards, Mladen - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Question about new mod_jk jk_lb_worker.c
Rainer Jung wrote, On 11/14/2004 7:31 AM: 0) Any ideas on rotating the mod_jk log file? Use cronolog http://cronolog.org/ or the rotatelogs program included with Apache. -Dave - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Question about new mod_jk jk_lb_worker.c
Hi Mladen, I include my original posting. Hi Mladen, great! I don't know how hard the following is to achieve, but it is the most prominent problem around mod_jk-Logging I know of: mod_jk does neither support Apaches rotatelogs nor does it have a log rotation capability on its own. If you or any other committer has an idea how to introduce rotating logfiles, that would be a major achievement. I have 3 more features I would be willing to provide. Maybe you or the other committers can comment, if they think that would be useful. All of them are already implemented for mod_jk/apache 1.3 and in production under high load. 1) Limiting new application sessions if load is to high. You can configure a start URL for your application (regexp based). If a request X matches this URL, then mod_jk checks the scoreboard how many apache requests are being processed simultaneously at that moment. There is a configurable limti, and if that limit is reached, the request X will not be sent to tomcat, but instead be answered by some configurable local static response (containing the info, that the load is to high and the user should try again later). Alternatively one can configure X to be answered by some external redirect. 2) Multi-Cluster-Routing Tomcat-Cluster does only allow session replication to all nodes in the cluster. Once you work with more than 3-4 nodes there is too much overhead and risk in replicating sessions to all nodes. We split all nodes into clustered groups. I introduced a new worker attribute domain to let get_most_suitable_worker know, to which other nodes the session gets replicated (the workers in the same domain). This enhancement has another benefit: Once mod_jk connects an apache process to a tomcat instance, the tomcat jk connector will need one thread for this connection as long as the apache process is alive. Now assume you have multiple Apaches and Tomcats. The Tomcats are clustered and mod_jk uses sticky sessions. Now you are going to shut down (maintenance) one tomcat. All Apache will start connections to all tomcats. You end up with all tomcats getting connections from all apache processes, so the number of threads needed inside the tomcats will explode. If you group the tomcats the connections normally will stay inside the groups (combine 2) with a routing load balancer to the apaches). 3) Idle connection disconnect This one also comes from the fact, that a mod_jk connection consumes one tomcat connector thread, even if its idle for a long time. The idle connection check in mod_jk only works, if the connection is going to be used. Then it is checked if it was idle for too long and probably is beeing closed and reconnected. I implemented a feature, where in configurable intervals all workers of an lb worker are checked for idle connections, even the ones which are not returned by get_most_suitable_worker. The rationale behind it is the same as in the second part of 2). 4) Open Problem I didn't check your new code, but at least before there was the problem, that a recovered worker that was offline a long time (in means of load) got all the work after recovery. Of course there was a fixed conatsnt of load added to it after recovery to compensate, but working with a constant was not really solving the problem. It looks like one should virtually add load to a worker, even during it's error time, so that it will get normal load after recovering. Thanks for any comments! Rainer Mladen Turk wrote: Rainer Jung wrote: Hi Mladen, hi everyone, i have a proposal for a slight enhancement concerning the log format of mod_jk. Maybe you could consider including it in your recently revised code. It contains only changes to jk_util.c and I attach a patch relativ to version 1.32 of the file. Hi Rainer. You are definitely on my track :) . I'm planning to add one additional log level, and clear all the log messages. 1. ERROR - fatal operations 2. WARN - problems but not fatal 3. INFO - production logging 4. DEBUG - protocol tracing 5. TRACE - function call logging 1) Include the log level of a message in the log line. 2) Include the PID of the logging process in the log file. I'd like to make a customizable log line, in the JkRequestLogFormat fashion to be able to add PID/ThreadId, level, etc... MT. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] Mladen Turk wrote: Peter Rossbach wrote: Hello Mladen, I have see your checkin's and Rainer Jung very fine mod_jk extension concept mail. I have two questions about lb changes: a) Why you not change the lb_value value after successful recovery at service() function ? After a longer fail the recovered worker get for a long time all new sessions... OK. Will check. Seems very reasonable. See also Rainers Jung mail Can you forward that. Seems I can not
Re: Question about new mod_jk jk_lb_worker.c
Rainer Jung wrote: I include my original posting. Hi Rainer, First of all thank you for ideas. They are great! 1) Limiting new application sessions if load is to high. There is a problem with that. I made a implementation counting the number of busy childs/threads from scoreboard (took me entire day), but again we should count the number of connections to tomcat, cause the apache might be serving static content. Anyhow the idea is great and I'll implement it in the new mod_proxy for Apache 2.2 where we have extra slots in the scoreboard. Sad but we can not do that inside mod_jk unless we implement our own shared memory, that was prover to be bogus in jk2. 2) Multi-Cluster-Routing Can you write some use case for that and perhaps some simple algo too. What about sticky-sessions and forcing failower if we do not have session replication? 3) Idle connection disconnect Use worker mpm. We just can not make maintainer thread for non-treaded mpm's like apache1.2 or prefork. 4) Open Problem I didn't check your new code, but at least before there was the problem, that a recovered worker that was offline a long time (in means of load) This should work now with the latest patches. Best regards, Mladen. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Question about new mod_jk jk_lb_worker.c
Hello Mladen, I have two use case for the Multi Cluster Routing: Use Case 1: More Scaling cluster = A tomcat standard we replicated the session to all tomcat node at a cluster. This replication strategie not scale very well, but when we split the tomcat nodes to some domain and the lb knows that, the system scale better. Apache 1 domain and cluster 1 worker w1.1 for T1.1 worker w1.2 for T1.2 worker w1.3 for T1.3 domain and cluster 2 worker w2.1 for T2.1 worker w2.2 for T2.2 worker w2.3 for T2.3 When a worker w1 at domain 1 failed the made the next try to w1.2 and w1.3, only this worker fail also, the balancer give the w2.x worker a chance. Ok, the client lose the session but than you have really hardware problems... Use Case 2: Switch smoothly to next software generation ( preferred domain) Release a new software at runtime without drop the current user sessions. Start complete new tomcat instances with new application release. Say the Loadbalancer that all new sessions start to domain 2 workers, but as you lost a worker at domain 1 (old release) switch to worker at same domain. Apache 1 domain and cluster 1 ( Software Release 1) worker w1.1 for T1.1 worker w1.2 for T1.2 worker w1.3 for T1.3 domain and cluster 2 ( Software Release 2) - preferred worker w2.1 for T2.1 worker w2.2 for T2.2 worker w2.3 for T2.3 Both use case work perfect with the mod_jk2 level concept, with a little patch. Only the level limit is a real problem for scaling well.. --- I check also the newest jk_lb_worker and it works fine for me... The increment technic is simple and powerful :-) Some examples w1+ w2 lb_factor 1 lb_value 0 values after calc request | w1 w2| comment 1 |0* 2 |w1 get the Session 2 |0 0* |w2 get the Session 3 |0 0* |w2 get the Session, w1 is in error state w1 lb_factor 3 lb_value 0 w2 lb_factor 1 lb_value 0 values after calc request | w1 w2 |comment 1| -1* 1| w1 get the Session 2| -2* 2| w1 get the Session 3| -3 3| w1 get the Session, w1 is first lb worker 4| 0 0*| w2 get the Session 5| -1* 1| w1 get the Session 6| -1 0* | w2 get the Session, w1 is in error 7| -1 0* | w2 get the Session, w1 is in error 7| -1* 1| w1 get the Session, w1 recover great. regards peter Mladen Turk schrieb: Rainer Jung wrote: I include my original posting. Hi Rainer, First of all thank you for ideas. They are great! 1) Limiting new application sessions if load is to high. There is a problem with that. I made a implementation counting the number of busy childs/threads from scoreboard (took me entire day), but again we should count the number of connections to tomcat, cause the apache might be serving static content. Anyhow the idea is great and I'll implement it in the new mod_proxy for Apache 2.2 where we have extra slots in the scoreboard. Sad but we can not do that inside mod_jk unless we implement our own shared memory, that was prover to be bogus in jk2. 2) Multi-Cluster-Routing Can you write some use case for that and perhaps some simple algo too. What about sticky-sessions and forcing failower if we do not have session replication? 3) Idle connection disconnect Use worker mpm. We just can not make maintainer thread for non-treaded mpm's like apache1.2 or prefork. 4) Open Problem I didn't check your new code, but at least before there was the problem, that a recovered worker that was offline a long time (in means of load) This should work now with the latest patches. Best regards, Mladen. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Question about new mod_jk jk_lb_worker.c
Peter Rossbach wrote: Hello Mladen, I have see your checkin's and Rainer Jung very fine mod_jk extension concept mail. I have two questions about lb changes: a) Why you not change the lb_value value after successful recovery at service() function ? After a longer fail the recovered worker get for a long time all new sessions... OK. Will check. Seems very reasonable. See also Rainers Jung mail Can you forward that. Seems I can not find it. b) I have play around with the mod_jk2 lb algo and find the level feature (hot-standby) very usefull. I'll have to check whether it will break something. Some patch would be more then welcome :). MT. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]