Re: slow jetty shutdown

[Trygve Sanne Hardersen]

On Wed, Oct 27, 2010 at 3:22 PM, Trygve Sanne Hardersen wrote:
>
>
> I've noticed that Jetty logs the following warning on startup:
>
> # Acceptors should be <=2*availableProcessors
>
> Geronimo applies the maxThreads parameter as the number of acceptor threads
> in Jetty, and I believe the default value of 50 is way too high. I've tried
> using 4 on my system, and this makes shutdowns much faster (12 seconds vs.
> 104 seconds with Maven). This is still slow though.
>

I have attached a patch that sets the number of acceptor threads to 2 per
connector, and exposes this as well as the acceptor queue size to the
config-substitutions.properties file. Jetty standalone uses 2 acceptor
threads by default, and with these settings the server shutdown is pretty
fast. I also believe they are more correct, and they remove the startup
warning mentioned above.

I did not find a way to use separate settings for the HTTP, HTTPS and AJP
connectors. I believe that requires separate parameter names.

Thanks!

Trygve


geronimo-2.2-1027647-jetty-acceptors.patch
Description: Binary data

Re: slow jetty shutdown

[Trygve Sanne Hardersen]

Many thanks to both of you!

I tried to apply Shawn's trick, but it gives me a nullpointer:

# java.lang.NullPointerException
#at
org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:447)
#at
org.eclipse.jetty.io.nio.SelectorManager.doStop(SelectorManager.java:254)

I tried to return before this happens, but that results in the same slow
shutdown. The same goes for calling doSelect() before stop().

I've noticed that Jetty logs the following warning on startup:

# Acceptors should be <=2*availableProcessors

Geronimo applies the maxThreads parameter as the number of acceptor threads
in Jetty, and I believe the default value of 50 is way too high. I've tried
using 4 on my system, and this makes shutdowns much faster (12 seconds vs.
104 seconds with Maven). This is still slow though.

I've noticed that the Jetty Acceptor thread calls the same close method in
SelectChannelConnector that is already called from the doStop method in
AbstractConnector when the server is stopping. This seems strange to me, but
disabling it had no effect.

Trygve

On Wed, Oct 27, 2010 at 7:15 AM, Shawn Jiang  wrote:

> Thanks Rick !
>
> I just took a look at this problem,  In method [1], there are about 50
> SelectSet to close in the loop.   Each one takes 1 second to complete
> because SelectSet.stop() uses a Closer to do the closing job asynchronously
> and wait for the closing for 1 second before closing it directly.   The
> asynchronous closing logic only get executed after SelectSet.select() is
> called.
>
> Adding set.doSelect() after  set.stop() at method [1]  did the trick.  I
> did not find a good way in geronimo side to fix this problem.
>
>
> [1] org.eclipse.jetty.io.nio.SelectorManager.doStop()
>
>  protected void doStop() throws Exception
> {
>
> SelectSet[] sets= _selectSet;
> _selectSet=null;
> if (sets!=null)
> {
> for (SelectSet set : sets)
> {
> if (set!=null)
> set.stop();
>
> if (set!=null)
> set.doSelect();
> }
> }
>
> super.doStop();
>
> }
>
> On Wed, Oct 27, 2010 at 1:01 AM, Rick McGuire  wrote:
>
>> On 10/26/2010 12:10 PM, Trygve Sanne Hardersen wrote:
>>
>>> Thanks for you suggestions.
>>>
>>> I have been using Jetty trunk internally for a couple of weeks, and did
>>> my own changes to the Geronimo codebase to make them work. These are much
>>> the same as the ones that have been added to Geronimo now, so I have
>>> replaced them with your changes.
>>>
>>> I haven't had any problems running the new Jetty once the changes to
>>> Geronimo were in place, but I'll try to debug/kill the server during
>>> shutdown and let you know what I find.
>>>
>>
>> I just committed some fixes that finally allow the server to start and I'm
>> seeing some exceptions and a very slow shutdown time now.  From the
>> exceptions, I have a feeling that there have been some lifecycle changes in
>> jetty that are resulting in somethings getting restarted every time a filter
>> is removed.  This will probably take a little more work to sort out.
>>
>> Rick
>>
>>
>>> Trygve
>>>
>>>
>>> On Tue, Oct 26, 2010 at 4:47 PM, Rick McGuire >> rick...@gmail.com>> wrote:
>>>
>>>On 10/26/2010 10:21 AM, Kevan Miller wrote:
>>>
>>>On Oct 26, 2010, at 9:15 AM, Trygve Sanne Hardersen wrote:
>>>
>>>Hi, I have been playing with a newer Jetty version for
>>>Geronimo 2.2 lately, and I'm happy to see that the branch
>>>has been updated to 7.2.0.v20101020. However the server
>>>shutdown is terribly slow now (takes more than a minute).
>>>
>>>Any idea what might be causing this? I have checked that
>>>the standalone Jetty distribution does not have the same
>>>problem, nor does Geronimo 2.2 r1027366. I'm on OS X 10.6
>>>with Java 6.
>>>
>>>I haven't run with the newer jetty. Can you send a kill -3 to
>>>the server process during server shutdown to capture the java
>>>thread stack traces? Should give some indication about what's
>>>happening during your slow shutdown...
>>>
>>>
>>>I can't even get the server to start up with the new Jetty.  There
>>>appear to be some interface changes that still need to be
>>>accounted for (working on a potential fix).  Not much has been
>>>done with the new version yet...the switch was done less than 24
>>>hours ago primarily to see if there are any issues with making the
>>>switch.
>>>
>>>Rick
>>>
>>>--kevan
>>>
>>>
>>>
>>
>
>
> --
> Shawn

Re: slow jetty shutdown

[Shawn Jiang]

Thanks Rick !

I just took a look at this problem,  In method [1], there are about 50
SelectSet to close in the loop.   Each one takes 1 second to complete
because SelectSet.stop() uses a Closer to do the closing job asynchronously
and wait for the closing for 1 second before closing it directly.   The
asynchronous closing logic only get executed after SelectSet.select() is
called.

Adding set.doSelect() after  set.stop() at method [1]  did the trick.  I did
not find a good way in geronimo side to fix this problem.


[1] org.eclipse.jetty.io.nio.SelectorManager.doStop()

 protected void doStop() throws Exception
{

SelectSet[] sets= _selectSet;
_selectSet=null;
if (sets!=null)
{
for (SelectSet set : sets)
{
if (set!=null)
set.stop();

if (set!=null)
set.doSelect();
}
}

super.doStop();
}

On Wed, Oct 27, 2010 at 1:01 AM, Rick McGuire  wrote:

> On 10/26/2010 12:10 PM, Trygve Sanne Hardersen wrote:
>
>> Thanks for you suggestions.
>>
>> I have been using Jetty trunk internally for a couple of weeks, and did my
>> own changes to the Geronimo codebase to make them work. These are much the
>> same as the ones that have been added to Geronimo now, so I have replaced
>> them with your changes.
>>
>> I haven't had any problems running the new Jetty once the changes to
>> Geronimo were in place, but I'll try to debug/kill the server during
>> shutdown and let you know what I find.
>>
>
> I just committed some fixes that finally allow the server to start and I'm
> seeing some exceptions and a very slow shutdown time now.  From the
> exceptions, I have a feeling that there have been some lifecycle changes in
> jetty that are resulting in somethings getting restarted every time a filter
> is removed.  This will probably take a little more work to sort out.
>
> Rick
>
>
>> Trygve
>>
>>
>> On Tue, Oct 26, 2010 at 4:47 PM, Rick McGuire > rick...@gmail.com>> wrote:
>>
>>On 10/26/2010 10:21 AM, Kevan Miller wrote:
>>
>>On Oct 26, 2010, at 9:15 AM, Trygve Sanne Hardersen wrote:
>>
>>Hi, I have been playing with a newer Jetty version for
>>Geronimo 2.2 lately, and I'm happy to see that the branch
>>has been updated to 7.2.0.v20101020. However the server
>>shutdown is terribly slow now (takes more than a minute).
>>
>>Any idea what might be causing this? I have checked that
>>the standalone Jetty distribution does not have the same
>>problem, nor does Geronimo 2.2 r1027366. I'm on OS X 10.6
>>with Java 6.
>>
>>I haven't run with the newer jetty. Can you send a kill -3 to
>>the server process during server shutdown to capture the java
>>thread stack traces? Should give some indication about what's
>>happening during your slow shutdown...
>>
>>
>>I can't even get the server to start up with the new Jetty.  There
>>appear to be some interface changes that still need to be
>>accounted for (working on a potential fix).  Not much has been
>>done with the new version yet...the switch was done less than 24
>>hours ago primarily to see if there are any issues with making the
>>switch.
>>
>>Rick
>>
>>--kevan
>>
>>
>>
>


-- 
Shawn

Re: slow jetty shutdown

[Trygve Sanne Hardersen]

I've debugged the server shutdown and it hangs here;

org.eclipse.jetty.server.nio.SelectChannelConnector # close

, on the "synchronized(this)" call on line #127. Eventually it enters the
synchronized block, but it takes long. Both the HTTP and HTTPS connectors
are effected, but not the AJP connector.

On the jetty-minimal server I only see warnings.

Trygve

On Tue, Oct 26, 2010 at 7:01 PM, Rick McGuire  wrote:

> On 10/26/2010 12:10 PM, Trygve Sanne Hardersen wrote:
>
>> Thanks for you suggestions.
>>
>> I have been using Jetty trunk internally for a couple of weeks, and did my
>> own changes to the Geronimo codebase to make them work. These are much the
>> same as the ones that have been added to Geronimo now, so I have replaced
>> them with your changes.
>>
>> I haven't had any problems running the new Jetty once the changes to
>> Geronimo were in place, but I'll try to debug/kill the server during
>> shutdown and let you know what I find.
>>
>
> I just committed some fixes that finally allow the server to start and I'm
> seeing some exceptions and a very slow shutdown time now.  From the
> exceptions, I have a feeling that there have been some lifecycle changes in
> jetty that are resulting in somethings getting restarted every time a filter
> is removed.  This will probably take a little more work to sort out.
>
> Rick
>
>
>> Trygve
>>
>>
>> On Tue, Oct 26, 2010 at 4:47 PM, Rick McGuire > rick...@gmail.com>> wrote:
>>
>>On 10/26/2010 10:21 AM, Kevan Miller wrote:
>>
>>On Oct 26, 2010, at 9:15 AM, Trygve Sanne Hardersen wrote:
>>
>>Hi, I have been playing with a newer Jetty version for
>>Geronimo 2.2 lately, and I'm happy to see that the branch
>>has been updated to 7.2.0.v20101020. However the server
>>shutdown is terribly slow now (takes more than a minute).
>>
>>Any idea what might be causing this? I have checked that
>>the standalone Jetty distribution does not have the same
>>problem, nor does Geronimo 2.2 r1027366. I'm on OS X 10.6
>>with Java 6.
>>
>>I haven't run with the newer jetty. Can you send a kill -3 to
>>the server process during server shutdown to capture the java
>>thread stack traces? Should give some indication about what's
>>happening during your slow shutdown...
>>
>>
>>I can't even get the server to start up with the new Jetty.  There
>>appear to be some interface changes that still need to be
>>accounted for (working on a potential fix).  Not much has been
>>done with the new version yet...the switch was done less than 24
>>hours ago primarily to see if there are any issues with making the
>>switch.
>>
>>Rick
>>
>>--kevan
>>
>

Re: slow jetty shutdown

[Rick McGuire]

On 10/26/2010 12:10 PM, Trygve Sanne Hardersen wrote:

Thanks for you suggestions.

I have been using Jetty trunk internally for a couple of weeks, and 
did my own changes to the Geronimo codebase to make them work. These 
are much the same as the ones that have been added to Geronimo now, so 
I have replaced them with your changes.


I haven't had any problems running the new Jetty once the changes to 
Geronimo were in place, but I'll try to debug/kill the server during 
shutdown and let you know what I find.


I just committed some fixes that finally allow the server to start and 
I'm seeing some exceptions and a very slow shutdown time now.  From the 
exceptions, I have a feeling that there have been some lifecycle changes 
in jetty that are resulting in somethings getting restarted every time a 
filter is removed.  This will probably take a little more work to sort out.


Rick



Trygve

On Tue, Oct 26, 2010 at 4:47 PM, Rick McGuire > wrote:


On 10/26/2010 10:21 AM, Kevan Miller wrote:

On Oct 26, 2010, at 9:15 AM, Trygve Sanne Hardersen wrote:

Hi, I have been playing with a newer Jetty version for
Geronimo 2.2 lately, and I'm happy to see that the branch
has been updated to 7.2.0.v20101020. However the server
shutdown is terribly slow now (takes more than a minute).

Any idea what might be causing this? I have checked that
the standalone Jetty distribution does not have the same
problem, nor does Geronimo 2.2 r1027366. I'm on OS X 10.6
with Java 6.

I haven't run with the newer jetty. Can you send a kill -3 to
the server process during server shutdown to capture the java
thread stack traces? Should give some indication about what's
happening during your slow shutdown...


I can't even get the server to start up with the new Jetty.  There
appear to be some interface changes that still need to be
accounted for (working on a potential fix).  Not much has been
done with the new version yet...the switch was done less than 24
hours ago primarily to see if there are any issues with making the
switch.

Rick

--kevan

Re: slow jetty shutdown

[Trygve Sanne Hardersen]

Thanks for you suggestions.

I have been using Jetty trunk internally for a couple of weeks, and did my
own changes to the Geronimo codebase to make them work. These are much the
same as the ones that have been added to Geronimo now, so I have replaced
them with your changes.

I haven't had any problems running the new Jetty once the changes to
Geronimo were in place, but I'll try to debug/kill the server during
shutdown and let you know what I find.

Trygve

On Tue, Oct 26, 2010 at 4:47 PM, Rick McGuire  wrote:

> On 10/26/2010 10:21 AM, Kevan Miller wrote:
>
>> On Oct 26, 2010, at 9:15 AM, Trygve Sanne Hardersen wrote:
>>
>>  Hi, I have been playing with a newer Jetty version for Geronimo 2.2
>>> lately, and I'm happy to see that the branch has been updated to
>>> 7.2.0.v20101020. However the server shutdown is terribly slow now (takes
>>> more than a minute).
>>>
>>> Any idea what might be causing this? I have checked that the standalone
>>> Jetty distribution does not have the same problem, nor does Geronimo 2.2
>>> r1027366. I'm on OS X 10.6 with Java 6.
>>>
>> I haven't run with the newer jetty. Can you send a kill -3 to the server
>> process during server shutdown to capture the java thread stack traces?
>> Should give some indication about what's happening during your slow
>> shutdown...
>>
>
> I can't even get the server to start up with the new Jetty.  There appear
> to be some interface changes that still need to be accounted for (working on
> a potential fix).  Not much has been done with the new version yet...the
> switch was done less than 24 hours ago primarily to see if there are any
> issues with making the switch.
>
> Rick
>
>  --kevan
>
>

Re: slow jetty shutdown

[Rick McGuire]

On 10/26/2010 10:21 AM, Kevan Miller wrote:

On Oct 26, 2010, at 9:15 AM, Trygve Sanne Hardersen wrote:


Hi, I have been playing with a newer Jetty version for Geronimo 2.2 lately, and 
I'm happy to see that the branch has been updated to 7.2.0.v20101020. However 
the server shutdown is terribly slow now (takes more than a minute).

Any idea what might be causing this? I have checked that the standalone Jetty 
distribution does not have the same problem, nor does Geronimo 2.2 r1027366. 
I'm on OS X 10.6 with Java 6.

I haven't run with the newer jetty. Can you send a kill -3 to the server 
process during server shutdown to capture the java thread stack traces? Should 
give some indication about what's happening during your slow shutdown...


I can't even get the server to start up with the new Jetty.  There 
appear to be some interface changes that still need to be accounted for 
(working on a potential fix).  Not much has been done with the new 
version yet...the switch was done less than 24 hours ago primarily to 
see if there are any issues with making the switch.


Rick


--kevan

Re: slow jetty shutdown

[Kevan Miller]

On Oct 26, 2010, at 9:15 AM, Trygve Sanne Hardersen wrote:

> Hi, I have been playing with a newer Jetty version for Geronimo 2.2 lately, 
> and I'm happy to see that the branch has been updated to 7.2.0.v20101020. 
> However the server shutdown is terribly slow now (takes more than a minute).
> 
> Any idea what might be causing this? I have checked that the standalone Jetty 
> distribution does not have the same problem, nor does Geronimo 2.2 r1027366. 
> I'm on OS X 10.6 with Java 6.

I haven't run with the newer jetty. Can you send a kill -3 to the server 
process during server shutdown to capture the java thread stack traces? Should 
give some indication about what's happening during your slow shutdown...

--kevan

slow jetty shutdown

[Trygve Sanne Hardersen]

Hi, I have been playing with a newer Jetty version for Geronimo 2.2 lately,
and I'm happy to see that the branch has been updated to 7.2.0.v20101020.
However the server shutdown is terribly slow now (takes more than a minute).

Any idea what might be causing this? I have checked that the standalone
Jetty distribution does not have the same problem, nor does Geronimo 2.2
r1027366. I'm on OS X 10.6 with Java 6.

Thanks

Trygve