Sorry to keep replying to myself but this fix does appear to work... It kept
my open file handles down to 5 or 6 for the duration of the load suite.
public UrlResourceStream(final URL url)
{
// Save URL
this.url = url;
URLConnection connection = null;
try
{
connection = url.openConnection();
contentLength = connection.getContentLength();
contentType = connection.getContentType();
>>if (connection instanceof JarURLConnection)
>>{
>> JarURLConnection jarUrlConnection =
(JarURLConnection)connection;
>> URL jarFileUrl =
jarUrlConnection.getJarFileURL();
>> URLConnection jarFileConnection =
jarFileUrl.openConnection();
>> try
>> {
>> lastModified =
jarFileConnection.getLastModified();
>> }
>> finally
>> {
>>
jarFileConnection.getInputStream().close();
>> }
>>}
>>else
>>{
>> lastModified = connection.getLastModified();
>>}
try
{
file = new File(new URI(url.toExternalForm()));
}
catch (Exception ex)
{
log.debug("cannot convert url: " + url + " to
file (" + ex.getMessage()
+
"), falling back to the inputstream for
polling");
}
if (file != null && !file.exists())
{
file = null;
}
}
catch (IOException ex)
{
// It should be impossible to get here or the original
URL
// couldn't have been constructed. But we re-throw with
details
// anyway.
final IllegalArgumentException illegalArgumentException
= new
IllegalArgumentException(
"Invalid URL parameter " + url);
illegalArgumentException.initCause(ex);
throw illegalArgumentException;
}
finally
{
// if applicable, disconnect
if (connection != null)
{
if (connection instanceof HttpURLConnection)
{
((HttpURLConnection)connection).disconnect();
}
else
{
try
{
connection.getInputStream().close();
}
catch (Exception ex)
{
// ignore
}
}
}
}
}
adambender wrote:
>
> Is it possible to use this
> http://www.mail-archive.com/wicket-u...@lists.sourceforge.net/msg20951.html
> workaround in the URLResourceStream constructor - similar to how it is
> done in URLResourceStream#lastModifiedTime()?
>
>
>
> adambender wrote:
>>
>> For what it is worth I saw a file being created in the URLResourceStream
>> constructor at line 88 but it doesn't look like this is the source of any
>> problems, its just a way to figure out how to check the lastModified time
>> later on.
>>
>> While I can't claim any kind of expertise on the inner workings of Wicket
>> it seems that this problem would be solved if it wasn't required to check
>> the last modification time on those resources which are embedded in the
>> Wicket jar. Of course I have no idea what else this would affect so it
>> may not be so simple. It may also be possible to just cache the stream or
>> the actual .js string for use by later requests so that it is not
>> necessary to try and reload files like this from the jar every time they
>> are requested.
>>
>> For our use what we are likely going to do is pull the .js files out and
>> place them where httpd can serve them and then use some url rewriting
>> magic to prevent Wicket from serving them.
>>
>>
>> Adam
>>
>>
>> igor.vaynberg wrote:
>>>
>>> i just dont see what we can do about this... we are calling
>>> connection.getLastModified(), we are not creating a file, etc...
>>>
>>> -igor
>>>
>>> On Wed, Oct 21, 2009 at 10:18 AM, adambender <adamben...@gmail.com>
>>> wrote:
>>>>
>>>> This issue looks like it is directly related to
>>>> http://issues.apache.org/jira/browse/WICKET-438 and the access of the
>>>> lastModified header. Every time a URLResourceStream is constructed the
>>>> lastModified time is requested at line 85 and the number of file
>>>> handles
>>>> goes up by one. The solution to the jira issue indicated that upgrading
>>>> the
>>>> version of linux fixed the problem but it doesnt seem to be the case
>>>> with OS
>>>> X or Red Hat Enterprise Linux...
>>>>
>>>>
>>>> adambender wrote:
>>>>>
>>>>> I was able to distill the problem down to what I think is the core
>>>>> issue
>>>>> here.
>>>>>
>>>>> When an AJAX Wicket page is rendered it contains a reference to two
>>>>> .js
>>>>> files as Igor had mentioned, the Web URLs of these files look like
>>>>> this:
>>>>>
>>>>> <App Context
>>>>> Root>/resources/org.apache.wicket.markup.html.WicketEventReference/wicket-event.js
>>>>> <App Context
>>>>> Root>/resources/org.apache.wicket.markup.html.WicketEventReference/wicket-ajax.js
>>>>>
>>>>> These requests are of course fielded by the Wicket filter which ends
>>>>> up
>>>>> trying to load the resources using the UrlResourceStream. In order to
>>>>> actually load these files Wicket gets the JAR URL for each file,
>>>>> converts
>>>>> them to a URIs and then passes them to the File constructor. The
>>>>> problem
>>>>> is that the URI that is created is opaque -
>>>>> http://www.docjar.com/docs/api/java/net/URI.html see here for a good
>>>>> explanation in the case of wicket-event.js the URI string is like the
>>>>> following :
>>>>>
>>>>> jar:file:/<absolute-path-tojar>/wicket-1.4.1.jar!/org/apache/wicket/markup/html/wicket-event.js
>>>>>
>>>>> It is similar for the case of wicket-ajax.js
>>>>>
>>>>> A quick glance at the File constructor shows that an opaque URL cannot
>>>>> be
>>>>> used to create a File as it is considered non-hierarchical. This of
>>>>> course
>>>>> leaves you with lots of requests to load this file, none of which can
>>>>> be
>>>>> completed and results in a lot of open URL connections which can never
>>>>> be
>>>>> closed.
>>>>>
>>>>> I guess my question is, should this loading mechanism be working
>>>>> because
>>>>> it doesnt seem like it could the way it is currently written? Is it
>>>>> possible to configure how Wicket finds these files embedded in the
>>>>> jar?
>>>>> Have I missed something?
>>>>>
>>>>>
>>>>> Adam Bender-2 wrote:
>>>>>>
>>>>>> Thanks for the explanation I think that helps shed some light... The
>>>>>> tests are actually JMeter tests driving load by emulating a web
>>>>>> browser - the application the are testing is running in Deployment
>>>>>> mode set up as though it were a production server. After a little
>>>>>> more
>>>>>> digging I have been able to determine that is only a problem on pages
>>>>>> which use Ajax - and it looks like there is a related debug message
>>>>>> coming from an exception handler regarding the wicket-event.js file
>>>>>> not being accessible (URI is not hierarchical). This would actually
>>>>>> explain why the problem only manifests when the JMeter tests are set
>>>>>> to request embedded assets as wicket ajax pages embed requests for
>>>>>> additional pieces of javascript - which in the case of
>>>>>> wicket-event.js
>>>>>> are causing exceptions to be thrown and the JVM bug you mentioned is
>>>>>> probably preventing these resources from being cleaned up properly.
>>>>>>
>>>>>> Does that sound right?
>>>>>>
>>>>>> Adam
>>>>>>
>>>>>>
>>>>>> On Oct 20, 2009, at 8:41 PM, Igor Vaynberg wrote:
>>>>>>
>>>>>>> when you are requesting an embedded resource wicket needs to stream
>>>>>>> the file out of a jar, the way the jvm handles that is by creating a
>>>>>>> jar url connection object that streams the file. unfortunately,
>>>>>>> there
>>>>>>> is a bug in the jdk where the url connection does not have a close
>>>>>>> method and so wicket or any other java app cannot release the
>>>>>>> connection. this is addressed, afair, in jdk 7.
>>>>>>>
>>>>>>> i have many apps deployed in production and have not managed yet to
>>>>>>> run out of the handles with the limit set about 4K. not sure why
>>>>>>> this
>>>>>>> is different in your case. you mentioned "tests", are those unit
>>>>>>> tests
>>>>>>> and is wicket there running in deployment mode?
>>>>>>>
>>>>>>> the resource watcher, which should be disabled, will cause the
>>>>>>> handles
>>>>>>> to run out because it continuously monitors markup files for changes
>>>>>>> (hot reloading in dev mode) and everytime it checks a markup file in
>>>>>>> a
>>>>>>> jar it creates the url connection and leaks it. by default it is
>>>>>>> disabled in deployment mode but if you manually set the poll
>>>>>>> frequency
>>>>>>> in your settings it will be reenabled.
>>>>>>>
>>>>>>> -igor
>>>>>>>
>>>>>>>
>>>>>>> On Tue, Oct 20, 2009 at 4:07 PM, Adam Bender <a...@magpieti.com>
>>>>>>> wrote:
>>>>>>>> We have run with a limit as high as 10,000 files and our tests can
>>>>>>>> bring it
>>>>>>>> to the limit in 20 minutes, but that still doesn't explain why so
>>>>>>>> many
>>>>>>>> copies of the jar are needed - and only when we are also requesting
>>>>>>>> embedded
>>>>>>>> assets...
>>>>>>>>
>>>>>>>> On Oct 20, 2009, at 5:04 PM, Major Péter wrote:
>>>>>>>>
>>>>>>>>> resolution -> solution...
>>>>>>>>> Just set the ulimit in the initscript and run it with start-stop-
>>>>>>>>> daemon,
>>>>>>>>> it will make the problem disappear (but still there would be many
>>>>>>>>> open
>>>>>>>>> files...)
>>>>>>>>>
>>>>>>>>> Peter
>>>>>>>>>
>>>>>>>>> 2009-10-21 00:38 keltezéssel, Major Péter írta:
>>>>>>>>>>
>>>>>>>>>> Hi,
>>>>>>>>>>
>>>>>>>>>> we also had this issue, because (I think) we are running two
>>>>>>>>>> different
>>>>>>>>>> wicket application in only one glassfish domain. Our resolution
>>>>>>>>>> was,
>>>>>>>>>> that we are running now the glassfish domains with custom init
>>>>>>>>>> scripts
>>>>>>>>>> with ulimit settings. Maybe this will help for you.
>>>>>>>>>>
>>>>>>>>>> Best Regards,
>>>>>>>>>> Peter
>>>>>>>>>>
>>>>>>>>>> 2009-10-21 00:14 keltezéssel, Martin Grigorov írta:
>>>>>>>>>>>
>>>>>>>>>>> Hi Adam,
>>>>>>>>>>>
>>>>>>>>>>> You may try to debug what is the problem with
>>>>>>>>>>> https://wiki.sdn.sap.com/wiki/display/Java/JPicus
>>>>>>>>>>>
>>>>>>>>>>> El mar, 20-10-2009 a las 15:39 -0600, Adam Bender escribió:
>>>>>>>>>>>>
>>>>>>>>>>>> Greetings all,
>>>>>>>>>>>>
>>>>>>>>>>>> Recently I have been performance testing a Wicket application
>>>>>>>>>>>> (1.4.1)
>>>>>>>>>>>> and I
>>>>>>>>>>>> am running into the dreaded Too Many Files Open issue. I have
>>>>>>>>>>>> searched
>>>>>>>>>>>> the
>>>>>>>>>>>> mailing lists and most of the trouble around this issue seems
>>>>>>>>>>>> to come
>>>>>>>>>>>> from
>>>>>>>>>>>> being in DEVELOPMENT mode so I made sure we were in DEPLOYMENT
>>>>>>>>>>>> mode.
>>>>>>>>>>>> When I
>>>>>>>>>>>> run 'lsof -p xxxx | grep wicket-1.4.1.jar' I see over 1000
>>>>>>>>>>>> entries and
>>>>>>>>>>>> it's
>>>>>>>>>>>> growing monotonically. This seems really unusual - why would
>>>>>>>>>>>> wicket
>>>>>>>>>>>> need
>>>>>>>>>>>> 1000 copies of this jar open? An additional bit of weirdness
>>>>>>>>>>>> came up
>>>>>>>>>>>> when
>>>>>>>>>>>> our load tests stopped loading the embedded assets (css, js and
>>>>>>>>>>>> images)
>>>>>>>>>>>> in
>>>>>>>>>>>> each page - this seemed to cap the number of lsof entries at
>>>>>>>>>>>> 5... This
>>>>>>>>>>>> is
>>>>>>>>>>>> even weirder because our app serves these static items from
>>>>>>>>>>>> httpd
>>>>>>>>>>>> without
>>>>>>>>>>>> tomcat ever knowing about them. How could our loading of
>>>>>>>>>>>> embedded items
>>>>>>>>>>>> really affect the number of file handles wicket needs?
>>>>>>>>>>>>
>>>>>>>>>>>> For completeness we are running this app on Red Hat Enterprise
>>>>>>>>>>>> Linx 5
>>>>>>>>>>>> with
>>>>>>>>>>>> Tomcat 6.0.20 and Java 1.6
>>>>>>>>>>>>
>>>>>>>>>>>> Adam
>>>>>>>>>
>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>> To unsubscribe, e-mail: users-unsubscr...@wicket.apache.org
>>>>>>>>> For additional commands, e-mail: users-h...@wicket.apache.org
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> ---------------------------------------------------------------------
>>>>>>>> To unsubscribe, e-mail: users-unsubscr...@wicket.apache.org
>>>>>>>> For additional commands, e-mail: users-h...@wicket.apache.org
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> ---------------------------------------------------------------------
>>>>>>> To unsubscribe, e-mail: users-unsubscr...@wicket.apache.org
>>>>>>> For additional commands, e-mail: users-h...@wicket.apache.org
>>>>>>>
>>>>>>
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: users-unsubscr...@wicket.apache.org
>>>>>> For additional commands, e-mail: users-h...@wicket.apache.org
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>
>>>> --
>>>> View this message in context:
>>>> http://www.nabble.com/Too-Many-Files-Wicket-1.4.1-tp25983047p25996646.html
>>>> Sent from the Wicket - User mailing list archive at Nabble.com.
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: users-unsubscr...@wicket.apache.org
>>>> For additional commands, e-mail: users-h...@wicket.apache.org
>>>>
>>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: users-unsubscr...@wicket.apache.org
>>> For additional commands, e-mail: users-h...@wicket.apache.org
>>>
>>>
>>>
>>
>>
>
>
--
View this message in context:
http://www.nabble.com/Too-Many-Files-Wicket-1.4.1-tp25983047p26001456.html
Sent from the Wicket - User mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@wicket.apache.org
For additional commands, e-mail: users-h...@wicket.apache.org
