On 10.08.2016 11:34, Jayaram Ponnusamy wrote:
Hi André,
Kindly please find my answer below:
On Tue, Aug 9, 2016 at 11:49 AM, André Warnier (tomcat) <a...@ice-sa.com>
wrote:
On 09.08.2016 09:48, Jayaram Ponnusamy wrote:
Thanks André,
Kindly please find my answer below:
On Tue, Aug 9, 2016 at 12:02 AM, André Warnier (tomcat) <a...@ice-sa.com>
wrote:
On 08.08.2016 11:18, Jayaram Ponnusamy wrote:
Thanks André,
1. ANS-Question 1 = Both Tomcat & WebServer installed on different
Machines, Attached Architecture diagram of our environment. Our Tomcat
are
configured as Application Clustering, and WebServer1 is pointing to
Tomcat1
and WebServer2 is pointing to Tomcat2
Tomcat's IP (Node1): 10.100.0.31
WebServer's IP (Node1): 10.100.15.21
Tomcat's IP (Node2): 10.100.0.32
WebServer's IP (Node2): 10.100.15.22
2. ANS-Question 2 = Thanks, Thats fine :)
3. ANS-Question 3 = Attached Architecture diagram for your reference..
There are a lot of separate pieces in your configuration, and the
problem
could be at any level. So we must do this step-by-step, systematically,
and
try to find out exactly where things are not working properly. The
keywords
here are "step-by-step" and "sytematically", because with so many pieces,
if we start making changes left and right without first understanding
what
the problem is, we will never find out.
First, I'll make some comments based on the configuration schema that you
provided, and you will tell me if this is right or wrong, ok ? And I have
some questions in between..
Please answer the questions below directly under each question, to make
it
easier to follow.
1) Note : I do not know what you mean exactly by "Application
Clustering",
and I don't know exactly how an F5 load-balancer works. But I will
assume
that in this case, we can consider only the chain from the user
workstation
to Tomcat #1, and that any results can be extrapolated to other cases.
I also assume that Tomcat #1 and Tomcat #2 run the same applications, but
that they are totally independent from eachother, and that Tomcat #1 and
Tomcat #2 never talk to one another directly (true/false ?) ANS: TRUE
2) I assume that the problems that you are mentioning are happening when
a
user workstation client accesses the Tomcat application as follows :
(You may need to reformat this as 2 text lines, to see it correctly)
user wks <-- HTTP(S) --> F5 <-- HTTP --> Apache #1 + mod_jk <-- AJP -->
Tomcat <-- webapp
(1) (2) (3)
where :
- HTTP(S) is the protocol used on the connection (1) between the client
workstation and the F5 load-balancer
- HTTP(S) is the protocol used on the connection (2) between the F5
load-balancer and Apache webserver #1 (IP 10.100.15.21)
- AJP is the protocol used on the connection (3) between Apache webserver
#1 ((IP 10.100.15.21) and Tomcat #1 (IP 10.100.0.31) (*)
- "Tomcat" actually means "Tomcat and its AJP <Connector>, listening on
port 9009
All correct ?
ANS::
||=> WebServer1 (http://web1abc.com/sites)====>Tomcat1 (
http://tom1abc:9090/sites)
F5 (http://f5.abc.com) =||
||=> WebServer2 (http://web1abc.com/sites)====>Tomcat1 (
http://tom21abc:9090/sites)
Application Clustering Means, While installing the Second Node (Portal
Application on Tomcat) There is an option to choose install as Application
cluster. Primarily Tomcat will not talk each other, but Portal Application
will consider both are same with clustering
2) To access the application, the user enters a URL in his browser.
Can you give us an example of such a URL ? You can change the real name
of
the server for privacy, but then be consistent about this in the future,
because it is important to follow what is going on.
So let's say for example that the F5 is known to the outside world as "
myF5.mycompany.com", and let's always keep using that same hostname for
it in the future.
So the client enters the URL : http://myF5.mycompany.com.. ?
(Add the port if any, and a sample URL)
ANS:
Client Access URL: http://f5abc.com/sites/en_US/about,
http://f5abc.com/sites/en_US/offices, http://f5abc.com/sites/ar_QA/n
ewsroom
Through Tomcat : http://tom1abc:9090/sites/en_US/about,
http://tom1abc:9090/sites/en_US/offices,
http://tom1abc:9090/sites/ar_QA/newsroom
3) the F5 decides, on the base of some criteria, to forward this HTTP
request to Apache webserver #1 or Apache webserver #2.
But for the sake of this example, let's assume that it chooses Apache #1
this time.
Question 3.1 : how does the F5 forward this request to Apache webserver
#1
? does it just copy the original request transparently, and send it to
the
IP address 10.100.15.21 ?
Or does it change the hostname to which the request is addressed ?
(Originally, the client addressed the request to hostname "
myF5.mycompany.com" (which resolves to the IP address of the F5); so
normally, if the F5 does not modify the hostname and sends it "as is" to
Apache #1, then the Apache #1 configuration should have a <VirtualHost>
with a ServerName matching "myF5.mycompany.com" also. And if it does
not,
then Apache #1 can only have 1 VirtualHost, the "default VirtualHost",
otherwise this would not work at all.)
ANS: F5 Not changing the Hostname, and keep as it is
If the F5 changes the hostname of the request, then to what does it
change
it ?
Question 3.2 : does Apache #1 have only 1 <VirtualHost> ?
If yes, what are the ServerName and ServerAlias lines in the
configuration
of that <VirtualHost> ? ANS: Yes its having only one <VirtualHost>
Initilly
we kept the ServerName as hostname only, some forum recommends to keep IP
so i modified it. Now i reverted it back to Hostname
ServerName cmspliv1.com
ServerAlias www.cmspliv1.com
#ServerName 10.100.0.31
If there are more than one <VirtualHost>, can you provide their
ServerName/ServerAlias, in the same order in which the multiple
<VirtualHost> are listed in the Apache #1 configuration ?
4)
Question 4.1 : is there anything (other than a network cable or other
passive equipment) between Apache #1 and Tomcat #1 ? (firewall for
example) ANS:
Only Firewall, but both are in different subNet
5) Tomcat #1 configuration :
Question 5.1 : The server.xml of Tomcat #1 has only 1 <Host> tag.
True/false ? ANS: TRUE
Question 5.2 : the server.xml of Tomcat #1 contains a tag similar to this
one :
<Engine name="Catalina" defaultHost="localhost">
What is the value of the "defaultHost" attribute in your server.xml ?
ANS: defaultHost value is "localhost"
Question 5.3 : the server.xml of Tomcat #1 has only one <Host> tag.
True/false ?
Please paste the content of that <Host> tag here : ANS: <Host
name="localhost" appBase="webapps" unpackWARs="true" autoDeploy="true">
6) Apache httpd :
Apache httpd 2.2.x has several possible configurations, in terms of how
many processes/threads it is using. See this page :
http://httpd.apache.org/docs/2.2/mpm.html
Question 6.1 : Do you know which "model" your Apache #1 is using ? (this
is also called the "MPM"). Is it :
- prefork
or
- worker
or
- event
or
- is your Apache host a Windows machine ? (then the MPM can only be
"mpm_winnt"). ANS: prefork.c
7) previously, you indicated that "when you access Tomcat directly", you
do not see these problems. With reference to your schema (and/or the
simple schema that I showed in (2) above), can you explain how the client
accesses Tomcat directly ? ANS: Actually Tomcat URL is not Public URL so,
clients cannot access it from outside. But we enabled access only to our
premisis to access all servers.
So we will use tomcat hostname & port followed by the context. eg:
http://tom1abc:9090/sites or http://10.100.0.31:9090/sites
Thank you for the information provided above.
Some of it is still a bit confusing however.
So let me summarise what is clear :
1) Tomcat #1 and Tomcat #2 have only 1 <Host> each, and in each case it is
"localhost", which is also the "default host" in each Tomcat.
(Basically, that means that it does not matter how a HTTP or AJP request
*arrives* to Tomcat, it is always executed the same way.
2) Apache #1 and Apache #2 also have only 1 <VirtualHost> each, and it is
thus the default VirtualHost for each one of them. So for Apache also, it
does not matter how a request arrives to it; it is always executed in the
context of this one VirtualHost, no matter which "Host:" header is in the
request, and no matter which "ServerName" or "ServerAlias" there is in the
configuration of that one <VirtualHost>.
3) When a request "is sent directly to Tomcat", it means actually that you
are bypassing the F5, bypassing Apache, and accessing Tomcat directly, from
inside your local network, and through the HTTP Connector of Tomcat (port
9090).
In that case, the processing of the request is also "normal, fast".
4) When a request is "sent through Apache", it means actually that it
comes from the external network, goes through the F5, from there to Apache,
and from there to Tomcat.
In that case, you also see various problems, such as slow request
response, connections lost, etc..
What is less clear, is the question of the firewall(s). Your answer "it is
in a different subnet" is not very helpful.
Referring to your schema (I try to re-attach it), where exactly is that
firewall ?
And do the requests that come through the F5, Apache, and to Tomcat,
actually go through that firewall ?
It does not matter if that firewall has any "rules" at all; I just want to
know if the traffic from F5 to Tomcat and vice-versa, goes through the
firewall or not, and where in the schema the firewall is :
- between the user and the F5 ?
- between the F5 and Apache ?
- between Apache and Tomcat ?
- somewhere else ?
Question # 1 :
From inside your local network, you can access the Tomcats directly,
without going through the F5 nor Apache.
Can you also access the Apaches directly, without going through the F5 ?
(In other words : do the Apache also have an "internal name" which is not
a public DNS name, and points directly to the Apache internal IP address ?)
If yes, what happens when you access the Tomcat applications through
Apache only ?
- does it work ? (I mean here : can you access the applications that way ?)
- do you have the same issues as when you access the applications through
the F5 ?
*ANS: Yes We are facing the slowness by accessing Apache & F5 URL *
No, I mean this :
workstation <- HTTP -> internal Apache <- AJP -> Tomcat
(in other words, without the F5).
Can you do that ? and does this also show the same problems ?
You also did not answer the question about the firewall, where it is exactly.
Question # 2 :
Does your Tomcat application use sessions ? and did you take this into
account in your configuration ?
To explain, an example :
1) user1 starts an "application session" by accessing :
http://f5.abc.com/sites/en_US/offices
<http://f5abc.com/sites/en_US/offices> (thus, through the F5 LB).
2) the F5 LB directs this first request to Apache/Tomcat #1.
3) Tomcat #1 starts a new session (session-id="abc123") for this client
4) Tomcat #1 returns a response. This response goes back to the user1, via
Apache #1 and via the F5.
5) user1 gets the response page, and clicks on a second link in that page..
That link points again to : http://f5.abc.com/sites/en_US/offices/xyz
<http://f5abc.com/sites/en_US/offices/xyz>.. (thus, to the F5 LB).
6) the F5 LB knows nothing about the session, so it sends this second
request to Apache/Tomcat #2 this time.
7) Tomcat #2 receives this second request, which includes the session-id
"abc123".
This Tomcat knows nothing about session-id "abc123", so it starts a new
session for user1 (session-id="xyz987").
What happens then in your application ? *ANS: As Per my understanding
Sticky_Session is not enabled, Kindly please assist to fix this issue.*
The above does not happen if you start a session by accessing Tomcat
directly (e.g. to "http://tom1abc:9090/sites/en_US/offices";) because
then, all follow-up actions will always be addressed to the same host, thus
the same Tomcat, who knows this session.
There is a mechanism called "sticky sessions", which is designed to avoid
this issue. Does that ring a bell ?
Kindly please check my mod_jk.log during connection lost. like if no one
accessing webserver URL (web1abc.com or web2abc.com) then webserver
disconnected from tomcat, URL will be active after restarts webServer.
About the log below :
I suggest that you consult this page :
http://tomcat.10.x6.nabble.com/mod-jk-Problem-td2136175.html
(it comes from one of the web archives of this Tomcat Users List)
I found that page by searching Google for :
"jk_handler::mod_jk.c Aborting connection for worker"
(one of the messages taken from your logfile below)
Many of the explanations given in that page, also apply here.
In particular, the message :
Aborting connection for worker=prod_live_svr
That message may come for a variety of reasons, maybe as simple as the client (the user at
his browser), clicking away because the server is taking too long to respond
(which may be due simply to your application being too slow).
The point is : do not *assume* that because you see a message which says "aborting
connection", that this is a mod_jk or Tomcat problem.
See further comments below.
In your log, there many lines like :
>>> [Wed Aug 10 10:53:11 2016]prod_live_svr web1abc.com 0.148376
which indicate requests that came in the same time period, and that have been handled
normally (and fast).
And then there are also lines like :
>>> [Wed Aug 10 11:01:18 2016]prod_live_svr f5.abc.com 67.194556
which indicate a request that took *67 seconds* before Tomcat returned a response to
mod_jk. That seems like a slow application, and indeed if real human clients must wait so
long before they see the beginning of a response, many of them would give up and navigate
somewhere else (which would cause an "aborted connection" log message).
What is interesting, is that the lines that seem "fast", all seem to indicate that the
"client" in that case is "web1abc.com" (Apache ?), while the lines that are comparatively
much slower, say "f5.abc.com" (the F5 load-balancer ?).
Is there any particular reason for that (apparently systematic) difference ?
With the timestamps below, you should have a look at the corresponding Apache access log,
and find out which exact requests corresponds to which line in the log below, and where
these requests are coming from.
[Wed Aug 10 10:53:11 2016]prod_live_svr web1abc.com 0.349541
[Wed Aug 10 10:53:11 2016][12875:1182443264] [info] jk_handler::mod_jk.c
(2608): Aborting connection for worker=prod_live_svr
[Wed Aug 10 10:53:11 2016]prod_live_svr web1abc.com 0.148376
Look at the Apache access log. Which request was that ? from where ?
[Wed Aug 10 10:53:11 2016]prod_live_svr web1abc.com 0.274469
[Wed Aug 10 10:53:11 2016][12883:1182443264] [info] jk_handler::mod_jk.c
(2608): Aborting connection for worker=prod_live_svr
[Wed Aug 10 10:53:11 2016]prod_live_svr web1abc.com 0.128201
[Wed Aug 10 10:53:11 2016]prod_live_svr web1abc.com 0.008925
[Wed Aug 10 10:53:11 2016]prod_live_svr web1abc.com 0.011010
[...] some similar lines removed
[Wed Aug 10 10:53:38 2016][12971:1182443264] [warn]
map_uri_to_worker_ext::jk_uri_worker_map.c (962): Uri * is invalid. Uri
must start with /
[Wed Aug 10 11:01:07 2016]prod_live_svr f5.abc.com 7.471178
[Wed Aug 10 11:01:18 2016]prod_live_svr f5.abc.com 67.194556
Look at the Apache access log. Which request was that ? from where ?
Is this Tomcat *webapp* response time normal, for this kind of request ?
Think also : the above seems to be a *succesful* request, for which Tomcat generated a
response in 67 seconds, and the client waited long enough to receive that response.
There may be a lot of similar requests which take even longer, and where the client loses
patience and goes away before receiving the response.
In such a case, you would have "client connection aborted" lines in your logs
also.
(you may need to increase the mod_jk JkLogLevel to see that).
[Wed Aug 10 11:05:43 2016]prod_live_svr f5.abc.com 17.388826
[Wed Aug 10 11:06:01 2016]prod_live_svr f5.abc.com 1.831268
[Wed Aug 10 11:13:47 2016]prod_live_svr f5.abc.com 17.888968
[Wed Aug 10 11:18:03 2016]prod_live_svr f5.abc.com 0.018467
[Wed Aug 10 11:21:17 2016]prod_live_svr f5.abc.com 16.999038
[Wed Aug 10 11:27:43 2016]prod_live_svr f5.abc.com 16.393558
[Wed Aug 10 11:41:17 2016]prod_live_svr f5.abc.com 17.675784
[Wed Aug 10 11:48:04 2016][12875:1182443264] [info]
ajp_send_request::jk_ajp_common.c (1496): (prod_live_svr) all endpoints are
disconnected, detected by connect check (1), cping (0), send (0)
The above is a normal message. It just means that mod_jk wants to send a request to
Tomcat, but it finds that all Tomcat connections in the pool are disconnected.
(This may be because Tomcat dropped the connection, because no more requests were coming
on that connection for a long time).
This is not a problem per se. mod_jk will re-create a connection to Tomcat, and this
should just take a couple of milliseconds. So it is not the reason for the slowness.
Note also that mod_jk labels these message as "[info]". So they are not errors.
[Wed Aug 10 11:48:40 2016][13638:1182443264] [info]
ajp_send_request::jk_ajp_common.c (1496): (prod_live_svr) all endpoints are
disconnected, detected by connect check (1), cping (0), send (0)
[...] some similar lines removed
[Wed Aug 10 12:03:03 2016][13755:1182443264] [info]
ajp_send_request::jk_ajp_common.c (1496): (prod_live_svr) all endpoints are
disconnected, detected by connect check (1), cping (0), send (0)
[Wed Aug 10 12:03:09 2016][14112:1182443264] [warn]
map_uri_to_worker_ext::jk_uri_worker_map.c (962): Uri * is invalid. Uri
must start with /
For this one, you should look at the Apache access log, to find out which request to
Apache corresponds to the above line. It is probably just a request by a DAV client, or a
monitoring utility (or a hacker scanning your webserver).
As a kind of temporary conclusion, I would say this :
I am not in front of your system, and I do not really know your full setup, nor your
application. So I am doing my best to try to understand what happens, but I can only have
a limited view on the issue, from the data that you are providing here. (So also, any data
which you are *not* providing, may give me a false impression).
I am not going to be able to give you "the solution".
You need to figure out by yourself, based on all the information that you have collected
so far, where the problem really is.
For that, remember the basic schema :
client <--> F5 <--> Apache + mod_jk <--> Tomcat <--> webapp <--> anything
behind webapp
(and maybe with some firewall in-between somewhere above).
And remember that there are 3 parts :
1) the request :
client --> F5 --> Apache + mod_jk --> Tomcat --> webapp --> anything behind
webapp
2) now the webapp is working to produce a response
3) the response (when the webapp starts producing it) :
client <-- F5 <-- Apache + mod_jk <-- Tomcat <-- webapp <-- anything behind
webapp
Then you need to simplify, and be systematic : start with the simplest
configuration, like
client <--> Tomcat <--> webapp
and with one specific application URL, always the same, repeat 10 times.
Look at the logs (only of Tomcat in this case, or maybe also the application log), and
check the times.
Then add one component :
client <--> Apache + mod_jk <--> Tomcat <--> webapp
do the *same* 10 client requests, and look at the logs again (this time, also
of Apache)
Then add one component again :
client <--> F5 <--> Apache + mod_jk <--> Tomcat <--> webapp
and do the same, and compare.
*Somehere* there is the problem.
At this point, we just do not know yet where.
/Maybe/ the problem is at the Apache <-> Tomcat level.
But at the moment, we do not know this yet. It could also be that your Tomcat is totally
overwhelmed, because the webapp takes too long to respond to each request, and it does not
have enough resources to respond to many similar requests at the same time.
One more tip : in general, for the "access log" kind of messages, the timestamp that you
see is the time at which the request *response* is sent. That means that if a request took
30 seconds to process, the actual request came in 30 seconds earlier than this timestamp.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
