Re: Refactoring class loader access to resources

[Romain Manni-Bucau]

worse than it, we download the artifact then new File(...).toURI().toURL()
:)

*Romain Manni-Bucau*
*Twitter: @rmannibucau *
*Blog: **http://rmannibucau.wordpress.com/*
*LinkedIn: **http://fr.linkedin.com/in/rmannibucau*
*Github: https://github.com/rmannibucau*



2013/9/25 Mark Thomas 

> On 24/09/2013 06:44, Romain Manni-Bucau wrote:
> > Hi
> >
> > about the expanding, tomee forces it so an option can work. I agree with
> > you "theorically" components shouldn't do it but in practises they have
> to
> > (there is no real solutions on this side. That's why all APIs about
> > scanning ends with a SPI :().
> >
> > About addURL I think you are right too and addURL() impl can handle the
> > conversion.
>
> OK. I think I can see a way to make this work then.
>
> > About jars.txt here is the history about it
> > https://issues.apache.org/jira/browse/TOMEE-794. In 2 words you can pu
> tin
> > WEB-INF a file called jars.txt and each line represent a jar. It can be a
> > mvn url, a http url, a file url or simply a path (with a hack for *.jar
> as
> > in the virtual webapp class loader of tomcat). A part of the magic is
> done
> > in start and addURL methods of
> >
> http://svn.apache.org/repos/asf/tomee/tomee/trunk/tomee/tomee-catalina/src/main/java/org/apache/tomee/catalina/LazyStopWebappClassLoader.java
> > (of
> > course this supposes getURLs impact addURL then).
> >
> > Hope it is clearer
>
> Much clearer thanks. I'm assuming that TomEE is going to convert the
> mvn:... coordinates into a URL (http, file, etc) that that JRE can
> understand.
>
> Mark
>
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: dev-h...@tomcat.apache.org
>
>

Re: Refactoring class loader access to resources

[Mark Thomas]

On 24/09/2013 06:44, Romain Manni-Bucau wrote:
> Hi
> 
> about the expanding, tomee forces it so an option can work. I agree with
> you "theorically" components shouldn't do it but in practises they have to
> (there is no real solutions on this side. That's why all APIs about
> scanning ends with a SPI :().
> 
> About addURL I think you are right too and addURL() impl can handle the
> conversion.

OK. I think I can see a way to make this work then.

> About jars.txt here is the history about it
> https://issues.apache.org/jira/browse/TOMEE-794. In 2 words you can pu tin
> WEB-INF a file called jars.txt and each line represent a jar. It can be a
> mvn url, a http url, a file url or simply a path (with a hack for *.jar as
> in the virtual webapp class loader of tomcat). A part of the magic is done
> in start and addURL methods of
> http://svn.apache.org/repos/asf/tomee/tomee/trunk/tomee/tomee-catalina/src/main/java/org/apache/tomee/catalina/LazyStopWebappClassLoader.java
> (of
> course this supposes getURLs impact addURL then).
> 
> Hope it is clearer

Much clearer thanks. I'm assuming that TomEE is going to convert the
mvn:... coordinates into a URL (http, file, etc) that that JRE can
understand.

Mark


-
To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org
For additional commands, e-mail: dev-h...@tomcat.apache.org

Re: Refactoring class loader access to resources

[Romain Manni-Bucau]

Hi

about the expanding, tomee forces it so an option can work. I agree with
you "theorically" components shouldn't do it but in practises they have to
(there is no real solutions on this side. That's why all APIs about
scanning ends with a SPI :().

About addURL I think you are right too and addURL() impl can handle the
conversion.

About jars.txt here is the history about it
https://issues.apache.org/jira/browse/TOMEE-794. In 2 words you can pu tin
WEB-INF a file called jars.txt and each line represent a jar. It can be a
mvn url, a http url, a file url or simply a path (with a hack for *.jar as
in the virtual webapp class loader of tomcat). A part of the magic is done
in start and addURL methods of
http://svn.apache.org/repos/asf/tomee/tomee/trunk/tomee/tomee-catalina/src/main/java/org/apache/tomee/catalina/LazyStopWebappClassLoader.java
(of
course this supposes getURLs impact addURL then).

Hope it is clearer

*Romain Manni-Bucau*
*Twitter: @rmannibucau *
*Blog: **http://rmannibucau.wordpress.com/*
*LinkedIn: **http://fr.linkedin.com/in/rmannibucau*
*Github: https://github.com/rmannibucau*



2013/9/24 Mark Thomas 

> On 24/09/2013 05:05, Romain Manni-Bucau wrote:
> > Hi
> >
> > about the handler: a lot of container (i particularly know tomee but not
> > only) use string representation of URL to find file to then work on the
> > file (either during scanning or sometimes for some more advanced "hack").
> > If you introduce something not known by these framework it will just
> fail.
> > That's the reason why scanning is not really portable and why vfs (for
> > JBoss for instance) exists. You are doing the same in tomcat (which is
> sad
> > since tomcat was until today simple on this aspect).
>
> I'd argue that components shouldn't be doing that as they have no idea
> what URLs might be coming back.
>
> I think the way to handle this is an option on the Context (so the
> default can be changed in conf/context.xml) to expand JARs from a WAR
> into the work dir and generate URLs to those files if all of the
> following are true:
> - running in a packed WAR
> - getURLs() is called
> - the option is enabled
>
> The JARs will be deleted on context stop.
>
> What I am trying to avoid here is the automatic copying of resources to
> the work dir. I'm also trying to ensure that all resource access goes
> through WebResources as I want to add some memory leak detection code
> their to catch things like InputStreams that are never closed that lead
> to locked files. Having spent ages trying to track down the root cause
> of a locked file, I see this sort of feature as extremely useful. In the
> same way the original memory leak detection code drove improvements in
> Tomcat and 3rd party libraries, I think this code do the same.
>
> > About addURL: we use it in tomee to enrich the classloader in multiple
> > places (but i saw some tomcat extensions doing it too). Our needs are:
> > 1) add container jar in a particular deployment mode (tomee as a webapp)
> > 2) add container jar for some particular features (jsf, jpa, ... - some
> > classes needs to be loaded from the webapp)
> > 3) use a descriptor to define the container (typically jars.txt lists all
> > jar of the loader with mvn/file/http/... syntax). Then our scanning just
> > reuse getURLs().
>
> I think this can be handled by addURL() being used to create the
> appropriate dir/JAR resource and adding it to WebResources but I'm not
> sure about the last one. Could you expand on that use case a little?
>
> Cheers,
>
> Mark
>
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: dev-h...@tomcat.apache.org
>
>

Re: Refactoring class loader access to resources

[Mark Thomas]

On 24/09/2013 05:05, Romain Manni-Bucau wrote:
> Hi
> 
> about the handler: a lot of container (i particularly know tomee but not
> only) use string representation of URL to find file to then work on the
> file (either during scanning or sometimes for some more advanced "hack").
> If you introduce something not known by these framework it will just fail.
> That's the reason why scanning is not really portable and why vfs (for
> JBoss for instance) exists. You are doing the same in tomcat (which is sad
> since tomcat was until today simple on this aspect).

I'd argue that components shouldn't be doing that as they have no idea
what URLs might be coming back.

I think the way to handle this is an option on the Context (so the
default can be changed in conf/context.xml) to expand JARs from a WAR
into the work dir and generate URLs to those files if all of the
following are true:
- running in a packed WAR
- getURLs() is called
- the option is enabled

The JARs will be deleted on context stop.

What I am trying to avoid here is the automatic copying of resources to
the work dir. I'm also trying to ensure that all resource access goes
through WebResources as I want to add some memory leak detection code
their to catch things like InputStreams that are never closed that lead
to locked files. Having spent ages trying to track down the root cause
of a locked file, I see this sort of feature as extremely useful. In the
same way the original memory leak detection code drove improvements in
Tomcat and 3rd party libraries, I think this code do the same.

> About addURL: we use it in tomee to enrich the classloader in multiple
> places (but i saw some tomcat extensions doing it too). Our needs are:
> 1) add container jar in a particular deployment mode (tomee as a webapp)
> 2) add container jar for some particular features (jsf, jpa, ... - some
> classes needs to be loaded from the webapp)
> 3) use a descriptor to define the container (typically jars.txt lists all
> jar of the loader with mvn/file/http/... syntax). Then our scanning just
> reuse getURLs().

I think this can be handled by addURL() being used to create the
appropriate dir/JAR resource and adding it to WebResources but I'm not
sure about the last one. Could you expand on that use case a little?

Cheers,

Mark


-
To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org
For additional commands, e-mail: dev-h...@tomcat.apache.org

Re: Refactoring class loader access to resources

[Romain Manni-Bucau]

Hi

about the handler: a lot of container (i particularly know tomee but not
only) use string representation of URL to find file to then work on the
file (either during scanning or sometimes for some more advanced "hack").
If you introduce something not known by these framework it will just fail.
That's the reason why scanning is not really portable and why vfs (for
JBoss for instance) exists. You are doing the same in tomcat (which is sad
since tomcat was until today simple on this aspect).

About addURL: we use it in tomee to enrich the classloader in multiple
places (but i saw some tomcat extensions doing it too). Our needs are:
1) add container jar in a particular deployment mode (tomee as a webapp)
2) add container jar for some particular features (jsf, jpa, ... - some
classes needs to be loaded from the webapp)
3) use a descriptor to define the container (typically jars.txt lists all
jar of the loader with mvn/file/http/... syntax). Then our scanning just
reuse getURLs().

If you want to discuss of it a bit more dynamically i'm still on IRC
(freenode #openejb) for some hours.

*Romain Manni-Bucau*
*Twitter: @rmannibucau *
*Blog: **http://rmannibucau.wordpress.com/*
*LinkedIn: **http://fr.linkedin.com/in/rmannibucau*
*Github: https://github.com/rmannibucau*



2013/9/24 Mark Thomas 

> On 23/09/2013 21:52, Romain Manni-Bucau wrote:
> > I already explained how t8 loader and WebResource breaks libs based on
> std
> > URLClassLoader
>
> I've looked back through the dev list archives and can't find any
> specific information that describes the problem in any detail. I'm
> fairly confident that whatever is missing / broken can be provided/fixed
> but that isn't going to happen without a clear explanation of what the
> problem is.
>
>
> > Here is the class in xbean
> >
> http://svn.apache.org/repos/asf/geronimo/xbean/trunk/xbean-finder/src/main/java/org/apache/xbean/finder/ClassLoaders.java
> >
> > I know we can workaround it but then we miss classes in the loader (was
> the
> > reason to use getURLs). Another thing is getURLs is insanely fast
> compared
> > to the other algorithm.
> >
> > Here is the xbean case but a lot of libs do it cause that's the only real
> > way to find urls from loaders.
> >
> > Another thing is jar:war urls will totally break common scanners (all
> since
> > it was not existing, isnt it?). So tomee, spring, owb, weld, ... will be
> > broken
>
> Why? They will be valid URLs in that JVM because the appropriate handler
> will have been registered. I can certainly imagine some scenarios where
> TomcatURLStreamHandlerFactory needs some tweaks to get it to play nicely
> in a broader container environment. Let us know what those tweaks are
> and they can be implemented.
>
> > Finally getURLs still say it respects addURL which doesnt look right
> (sadly)
>
> I see what happened on this. addURL() was removed as part of the
> resources refactoring that removed addRepository(). It looks like
> addURL() and the associated plumbing needs to be added back.
>
> What is the use case for needing to call addURL() on this class loader?
> It adds complexity to the class loader that I'd be happy to avoid if I
> could. Is the use case is something that can/should be handled through
> the new resources implementation?
>
> Mark
>
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: dev-h...@tomcat.apache.org
>
>

Re: Refactoring class loader access to resources

[Mark Thomas]

On 23/09/2013 21:52, Romain Manni-Bucau wrote:
> I already explained how t8 loader and WebResource breaks libs based on std
> URLClassLoader

I've looked back through the dev list archives and can't find any
specific information that describes the problem in any detail. I'm
fairly confident that whatever is missing / broken can be provided/fixed
but that isn't going to happen without a clear explanation of what the
problem is.


> Here is the class in xbean
> http://svn.apache.org/repos/asf/geronimo/xbean/trunk/xbean-finder/src/main/java/org/apache/xbean/finder/ClassLoaders.java
> 
> I know we can workaround it but then we miss classes in the loader (was the
> reason to use getURLs). Another thing is getURLs is insanely fast compared
> to the other algorithm.
> 
> Here is the xbean case but a lot of libs do it cause that's the only real
> way to find urls from loaders.
> 
> Another thing is jar:war urls will totally break common scanners (all since
> it was not existing, isnt it?). So tomee, spring, owb, weld, ... will be
> broken

Why? They will be valid URLs in that JVM because the appropriate handler
will have been registered. I can certainly imagine some scenarios where
TomcatURLStreamHandlerFactory needs some tweaks to get it to play nicely
in a broader container environment. Let us know what those tweaks are
and they can be implemented.

> Finally getURLs still say it respects addURL which doesnt look right (sadly)

I see what happened on this. addURL() was removed as part of the
resources refactoring that removed addRepository(). It looks like
addURL() and the associated plumbing needs to be added back.

What is the use case for needing to call addURL() on this class loader?
It adds complexity to the class loader that I'd be happy to avoid if I
could. Is the use case is something that can/should be handled through
the new resources implementation?

Mark


-
To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org
For additional commands, e-mail: dev-h...@tomcat.apache.org

Re: Refactoring class loader access to resources

[Romain Manni-Bucau]

I already explained how t8 loader and WebResource breaks libs based on std
URLClassLoader

Here is the class in xbean
http://svn.apache.org/repos/asf/geronimo/xbean/trunk/xbean-finder/src/main/java/org/apache/xbean/finder/ClassLoaders.java

I know we can workaround it but then we miss classes in the loader (was the
reason to use getURLs). Another thing is getURLs is insanely fast compared
to the other algorithm.

Here is the xbean case but a lot of libs do it cause that's the only real
way to find urls from loaders.

Another thing is jar:war urls will totally break common scanners (all since
it was not existing, isnt it?). So tomee, spring, owb, weld, ... will be
broken.

Finally getURLs still say it respects addURL which doesnt look right (sadly)
Le 23 sept. 2013 22:35, "Mark Thomas"  a écrit :

> On 23/09/2013 13:29, Romain Manni-Bucau wrote:
> > Xbean (so openwebbeans), groovy (so spring lang, groovy etc...)
>
> That doesn't help. It needs to be of the form "With Tomcat 7 we got...
> but with Tomcat 8 we get...".
>
> It might also need a "and it doesn't work because..." depending on how
> obvious the breakage is.
>
> Mark
>
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: dev-h...@tomcat.apache.org
>
>

Re: Refactoring class loader access to resources

[Mark Thomas]

On 23/09/2013 13:29, Romain Manni-Bucau wrote:
> Xbean (so openwebbeans), groovy (so spring lang, groovy etc...)

That doesn't help. It needs to be of the form "With Tomcat 7 we got...
but with Tomcat 8 we get...".

It might also need a "and it doesn't work because..." depending on how
obvious the breakage is.

Mark


-
To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org
For additional commands, e-mail: dev-h...@tomcat.apache.org

Re: Refactoring class loader access to resources

[Romain Manni-Bucau]

Xbean (so openwebbeans), groovy (so spring lang, groovy etc...)
Le 23 sept. 2013 22:27, "Mark Thomas"  a écrit :

> On 23/09/2013 13:21, Romain Manni-Bucau wrote:
> > The main issue of t8 is it breaks some scanning features when relying on
> > URLClassLoader (getURLs()), not perfect but common in libs...
>
> Example(s) please.
>
> Mark
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: dev-h...@tomcat.apache.org
>
>

Re: Refactoring class loader access to resources

[Mark Thomas]

On 23/09/2013 13:21, Romain Manni-Bucau wrote:
> The main issue of t8 is it breaks some scanning features when relying on
> URLClassLoader (getURLs()), not perfect but common in libs...

Example(s) please.

Mark

-
To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org
For additional commands, e-mail: dev-h...@tomcat.apache.org

Re: Refactoring class loader access to resources

[Romain Manni-Bucau]

The main issue of t8 is it breaks some scanning features when relying on
URLClassLoader (getURLs()), not perfect but common in libs...
Le 23 sept. 2013 19:00, "Mark Thomas"  a écrit :

> On 23/09/2013 09:08, Rainer Jung wrote:
> > On 23.09.2013 17:27, Mark Thomas wrote:
> >> I've mentioned on a couple of different threads that refactoring how the
> >> class loader accesses resources was on my TODO list. I haven't done any
> >> implementation yet but I have some ideas that I wanted to get some
> >> feedback on before I start work.
> >>
> >> The things I have in mind at this point are:
> >> - Extracting anything into the work directory feels like a hack
> >> - The class loader should use the same API as every other component
> >> - The class loader needs URLs that actually work
> >> - The anti-locking options are responsible for a number of the hacks in
> >>   the class loader
> >>
> >> The solution I have in mind is along the following lines:
> >> - Add a new getClassLoaderResource() to the WebResources interfaces.
> >>   This will implement the standard WEB-INF/classes then JARs in
> >>   WEB-INF/lib search order. It should be able to do this without any
> >>   extraction into the work directory.
> >
> > A useful feature in TC 6/7 was to add search paths outside of the war to
> > look for config files in order to provide config options special to
> > deployment environments without working with individual war files (via
> > VirtualWebappLoader).
> >
> > I know it is possible with the current resource implementation in TC 8,
> > but will it also be possible after this change, i.e. if the webapp
> > retrieves a config file as a resource via the webapp class loader, can
> > we still configure additional directories to be searched for the
> resource?
>
> Yes (assuming I don't mess up the implementation).
>
> Mark
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: dev-h...@tomcat.apache.org
>
>

Re: Refactoring class loader access to resources

[Mark Thomas]

On 23/09/2013 09:08, Rainer Jung wrote:
> On 23.09.2013 17:27, Mark Thomas wrote:
>> I've mentioned on a couple of different threads that refactoring how the
>> class loader accesses resources was on my TODO list. I haven't done any
>> implementation yet but I have some ideas that I wanted to get some
>> feedback on before I start work.
>>
>> The things I have in mind at this point are:
>> - Extracting anything into the work directory feels like a hack
>> - The class loader should use the same API as every other component
>> - The class loader needs URLs that actually work
>> - The anti-locking options are responsible for a number of the hacks in
>>   the class loader
>>
>> The solution I have in mind is along the following lines:
>> - Add a new getClassLoaderResource() to the WebResources interfaces.
>>   This will implement the standard WEB-INF/classes then JARs in
>>   WEB-INF/lib search order. It should be able to do this without any
>>   extraction into the work directory.
> 
> A useful feature in TC 6/7 was to add search paths outside of the war to
> look for config files in order to provide config options special to
> deployment environments without working with individual war files (via
> VirtualWebappLoader).
> 
> I know it is possible with the current resource implementation in TC 8,
> but will it also be possible after this change, i.e. if the webapp
> retrieves a config file as a resource via the webapp class loader, can
> we still configure additional directories to be searched for the resource?

Yes (assuming I don't mess up the implementation).

Mark

-
To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org
For additional commands, e-mail: dev-h...@tomcat.apache.org

Re: Refactoring class loader access to resources

[Rainer Jung]

On 23.09.2013 17:27, Mark Thomas wrote:
> I've mentioned on a couple of different threads that refactoring how the
> class loader accesses resources was on my TODO list. I haven't done any
> implementation yet but I have some ideas that I wanted to get some
> feedback on before I start work.
> 
> The things I have in mind at this point are:
> - Extracting anything into the work directory feels like a hack
> - The class loader should use the same API as every other component
> - The class loader needs URLs that actually work
> - The anti-locking options are responsible for a number of the hacks in
>   the class loader
> 
> The solution I have in mind is along the following lines:
> - Add a new getClassLoaderResource() to the WebResources interfaces.
>   This will implement the standard WEB-INF/classes then JARs in
>   WEB-INF/lib search order. It should be able to do this without any
>   extraction into the work directory.

A useful feature in TC 6/7 was to add search paths outside of the war to
look for config files in order to provide config options special to
deployment environments without working with individual war files (via
VirtualWebappLoader).

I know it is possible with the current resource implementation in TC 8,
but will it also be possible after this change, i.e. if the webapp
retrieves a config file as a resource via the webapp class loader, can
we still configure additional directories to be searched for the resource?

> - Class loader resources will not be cached.
> - Remove all of the current anti-locking options.
> - Wrap all InputStreams returned by the class loader and use this
>   wrapper to track whether or not the InputStream has been closed.
> - Update the memory leak protection code to close any class loader
>   InputStreams that have not been closed when the web application is
>   stopped.
> - Add a debug option that captures the stack trace of the code that
>   creates the class loader InputStream that can be reported when
>   closing leaked streams.
> - Add the same InputStream tracking to InputStreams provided by the
>   standard resource implementation.
> 
> Obviously details TBD as the implementation evolves.

Regards,

Rainer


-
To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org
For additional commands, e-mail: dev-h...@tomcat.apache.org

Refactoring class loader access to resources

[Mark Thomas]

I've mentioned on a couple of different threads that refactoring how the
class loader accesses resources was on my TODO list. I haven't done any
implementation yet but I have some ideas that I wanted to get some
feedback on before I start work.

The things I have in mind at this point are:
- Extracting anything into the work directory feels like a hack
- The class loader should use the same API as every other component
- The class loader needs URLs that actually work
- The anti-locking options are responsible for a number of the hacks in
  the class loader

The solution I have in mind is along the following lines:
- Add a new getClassLoaderResource() to the WebResources interfaces.
  This will implement the standard WEB-INF/classes then JARs in
  WEB-INF/lib search order. It should be able to do this without any
  extraction into the work directory.
- Class loader resources will not be cached.
- Remove all of the current anti-locking options.
- Wrap all InputStreams returned by the class loader and use this
  wrapper to track whether or not the InputStream has been closed.
- Update the memory leak protection code to close any class loader
  InputStreams that have not been closed when the web application is
  stopped.
- Add a debug option that captures the stack trace of the code that
  creates the class loader InputStream that can be reported when
  closing leaked streams.
- Add the same InputStream tracking to InputStreams provided by the
  standard resource implementation.

Obviously details TBD as the implementation evolves.

Thoughts?

Mark

-
To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org
For additional commands, e-mail: dev-h...@tomcat.apache.org