Re: Compilation Time | More questions ClassLoader
On Thu, 2007-05-24 at 10:15 +0200, Martin Schlienger wrote: > Thanks to all for your contribution. We have now a version that works > and without further optimizations we can run jamvm with a local > classpath of ~ 800ko and still be able to use all the classes in GNU > Classpath by loading them from a remote http location. Sounds interesting. Can you provide a patch? - twisti
Re: Compilation Time | More questions ClassLoader
Thanks to all for your contribution. We have now a version that works and without further optimizations we can run jamvm with a local classpath of ~ 800ko and still be able to use all the classes in GNU Classpath by loading them from a remote http location. Indeed there were several problems having a seperate ClassLoader among them: - all classes from a same package must be loaded from the same classloader due to a security concern. Bypassing this security in jamvm occurs further problems. - classes loaded by bootstrap and classes loaded by system class loader or any other one are not loaded in the same place within jamvm. Loading the class bytes in java and then providing them to the bootstrap class loader won't work either: -When resolving symbolic links, the VM re-use the same class loader as the class that is loading and in this case it is "null" so the only place it will look for the further classes is glibj.zip So: - Since we didn't want to code an HTTP classloader in C, the easiest solution we found is calling static java code from C using JamVM functions: => have a static method that getBytesFromURL in class ClassLoader (could have been another class) => Call this static method from JamVM / Bootstrap ClassLoader just after it has search in the local zip file and if the class was still not found. It is short, simple, doesn't need any other modifications in class/methods/function signature. It works well, so again, thanks for your advice.
Re: Compilation Time | More questions ClassLoader
Yep thanks, I have understood the principle. But this was quite blurred how to provide the bytes to the bootstrap classLoader. We have to change not only VMClassLoader.loadClass signature but some of the VM function behavior. This might not be too complicated though, since we now have quite understand how it works. 2007/5/9, Andrew Haley <[EMAIL PROTECTED]>: Martin Schlienger writes: > 2007/5/4, Tom Tromey <[EMAIL PROTECTED]>: > > Yup. I'd suggest that instead of thinking of the solution in terms of > > writing a class loader, think about it in terms of writing some code > > that simply returns a byte[] that is the class contents. > > This is already done. Although this downloading code is called from > loadClass. The byte[] array is then transformed to a Class object > thanks to defineClass methods. > > > > > > Then call this downloading code from VMClassLoader.loadClass. > > > > How you implement the downloading is up to you, but it turns out to be > > pretty simple to re-use URLClassLoader for this... just treat the > > .class file at the URL as a resource, not a class, and don't use the > > loadClass family of calls on your delegate URLClassLoader at all. > > Basically when a new() is performed and the class not already loaded, > doesn't the VM calls loadClass() family of calls? How to avoid > loadClass() from ClassLoader, download our byte[] (as explained above) > and create a Class object out of it without calling defineClass() ? > > > > > Martin> We have done some basic tests and this seemed to work. However, when > > Martin> trying with a more complete program, it appears that it throws > > Martin> IllegalAccessException making me think that some security is added > > Martin> when using the defineClass() on core API. > > > > For this sort of a thing more details are needed. At least a full > > stack trace. But if you follow the above this may be moot. > > > > I think we got more on this one and every problem seems to be related. > Whether the class code is loaded from bootstrap (thanks to native code > VMClassLoader.loadClass() ) or thanks to a Java ClassLoader, the Class > object is finally created with native function defineClass() -called > directly from native loadClass in the first case (bootstrap) or called > with VMClassLoader.defineClass() in the second case (Java ClassLoader) > but it is finally the same function. > > There are two differences when calling native defineClass() in the two cases: > - Protection Domain, which we could easely set to null without any difference > - The classloader itself > > Our debbuging shows that the VM is actually able to find a Class from > the ClassLoader without calling loadClass() but just using the > reference we give on this classLoader when calling defineClass(). This > is done when the VM has to load a class on which depends the class we > wanted to load. This results in a compromised situation: I think Tom Tromey has already explained what you need to do. You shouldn't be delegating to your new class loader, but calling its getResource from the bootstrap class loader. In that case, the Protection Domain isn't going to be any different. When loading bootstrap classes you shouldn't be calling defineClass from your own class loader, but from the bootstrap class loader. Your own class loader should just provide the bytes to the bootstrap loader. Andrew.
Re: Compilation Time | More questions ClassLoader
Martin Schlienger writes: > 2007/5/4, Tom Tromey <[EMAIL PROTECTED]>: > > Yup. I'd suggest that instead of thinking of the solution in terms of > > writing a class loader, think about it in terms of writing some code > > that simply returns a byte[] that is the class contents. > > This is already done. Although this downloading code is called from > loadClass. The byte[] array is then transformed to a Class object > thanks to defineClass methods. > > > > > > Then call this downloading code from VMClassLoader.loadClass. > > > > How you implement the downloading is up to you, but it turns out to be > > pretty simple to re-use URLClassLoader for this... just treat the > > .class file at the URL as a resource, not a class, and don't use the > > loadClass family of calls on your delegate URLClassLoader at all. > > Basically when a new() is performed and the class not already loaded, > doesn't the VM calls loadClass() family of calls? How to avoid > loadClass() from ClassLoader, download our byte[] (as explained above) > and create a Class object out of it without calling defineClass() ? > > > > > Martin> We have done some basic tests and this seemed to work. However, > > when > > Martin> trying with a more complete program, it appears that it throws > > Martin> IllegalAccessException making me think that some security is added > > Martin> when using the defineClass() on core API. > > > > For this sort of a thing more details are needed. At least a full > > stack trace. But if you follow the above this may be moot. > > > > I think we got more on this one and every problem seems to be related. > Whether the class code is loaded from bootstrap (thanks to native code > VMClassLoader.loadClass() ) or thanks to a Java ClassLoader, the Class > object is finally created with native function defineClass() -called > directly from native loadClass in the first case (bootstrap) or called > with VMClassLoader.defineClass() in the second case (Java ClassLoader) > but it is finally the same function. > > There are two differences when calling native defineClass() in the two cases: > - Protection Domain, which we could easely set to null without any difference > - The classloader itself > > Our debbuging shows that the VM is actually able to find a Class from > the ClassLoader without calling loadClass() but just using the > reference we give on this classLoader when calling defineClass(). This > is done when the VM has to load a class on which depends the class we > wanted to load. This results in a compromised situation: I think Tom Tromey has already explained what you need to do. You shouldn't be delegating to your new class loader, but calling its getResource from the bootstrap class loader. In that case, the Protection Domain isn't going to be any different. When loading bootstrap classes you shouldn't be calling defineClass from your own class loader, but from the bootstrap class loader. Your own class loader should just provide the bytes to the bootstrap loader. Andrew.
Re: Compilation Time | More questions ClassLoader
2007/5/4, Tom Tromey <[EMAIL PROTECTED]>: Yup. I'd suggest that instead of thinking of the solution in terms of writing a class loader, think about it in terms of writing some code that simply returns a byte[] that is the class contents. This is already done. Although this downloading code is called from loadClass. The byte[] array is then transformed to a Class object thanks to defineClass methods. Then call this downloading code from VMClassLoader.loadClass. How you implement the downloading is up to you, but it turns out to be pretty simple to re-use URLClassLoader for this... just treat the .class file at the URL as a resource, not a class, and don't use the loadClass family of calls on your delegate URLClassLoader at all. Basically when a new() is performed and the class not already loaded, doesn't the VM calls loadClass() family of calls? How to avoid loadClass() from ClassLoader, download our byte[] (as explained above) and create a Class object out of it without calling defineClass() ? Martin> We have done some basic tests and this seemed to work. However, when Martin> trying with a more complete program, it appears that it throws Martin> IllegalAccessException making me think that some security is added Martin> when using the defineClass() on core API. For this sort of a thing more details are needed. At least a full stack trace. But if you follow the above this may be moot. I think we got more on this one and every problem seems to be related. Whether the class code is loaded from bootstrap (thanks to native code VMClassLoader.loadClass() ) or thanks to a Java ClassLoader, the Class object is finally created with native function defineClass() -called directly from native loadClass in the first case (bootstrap) or called with VMClassLoader.defineClass() in the second case (Java ClassLoader) but it is finally the same function. There are two differences when calling native defineClass() in the two cases: - Protection Domain, which we could easely set to null without any difference - The classloader itself Our debbuging shows that the VM is actually able to find a Class from the ClassLoader without calling loadClass() but just using the reference we give on this classLoader when calling defineClass(). This is done when the VM has to load a class on which depends the class we wanted to load. This results in a compromised situation: - If forcing ClassLoader to null when calling defineClass from our ClassLoader (to have a similar call as if we were the real bootstrap) the VM is unable to find the classes which should be automatically loaded (since it can't be loaded from bootstrap and a null pointer to ClassLoader means bootstrap). ClassNotFoundException. - If giving the right reference to our ClassLoader, IllegalAccessException are thrown, when constructor (and thus calls to super() ) is executed. java.lang.IllegalAccessException: method is not accessible at java.awt.Frame.(Frame.java:246) at java.awt.Frame.(Frame.java:234) at test2.Test1.(Test1.java:9) at test2.Test.(Test.java:8) at test2.Test.main(Test.java:11) It seems that whether the ClassLoader is set to null or not is the discriminating security parameter. And we can't set it to null because of the problem explained above, about dependencies. I think we have to find where this security is managed -at runtime and not when loading class- and modify its behavior... thanks for any help and sorry if I didn't understand your first solution, I'm all ears for more details. martin Tom
Re: Compilation Time | More questions ClassLoader
> "Martin" == Martin Schlienger <[EMAIL PROTECTED]> writes:
Martin> Current behavior acts like this, macro-algorithm:
[..]
Martin> Class loadClass(String className){
Martin> if (bootstrap) {
Martin> c= VMClassLoader.loadClass(className);
Martin> Basically all we have done is add a step at
Martin> bootstrap. Normally all classes to perform a
Martin> RemoteClassLoader.loadClass() can be loaded with the first
Martin> step VMClassLoader.loadClass(). In the end we want to have
Martin> only these classes in the local core API and the rest of the
Martin> API distant.
Yup. I'd suggest that instead of thinking of the solution in terms of
writing a class loader, think about it in terms of writing some code
that simply returns a byte[] that is the class contents.
Then call this downloading code from VMClassLoader.loadClass.
How you implement the downloading is up to you, but it turns out to be
pretty simple to re-use URLClassLoader for this... just treat the
.class file at the URL as a resource, not a class, and don't use the
loadClass family of calls on your delegate URLClassLoader at all.
Martin> We have done some basic tests and this seemed to work. However, when
Martin> trying with a more complete program, it appears that it throws
Martin> IllegalAccessException making me think that some security is added
Martin> when using the defineClass() on core API.
For this sort of a thing more details are needed. At least a full
stack trace. But if you follow the above this may be moot.
Tom
Re: Compilation Time | More questions ClassLoader
Hi,
thanks for your information that confirms what we have seen already.
We have noted that VMClassLoader was the root Class with native method
signature and almost understand the mechanisms/links with our VM
(jamvm).
However as a first constraint we do not want to code our thing in a
native method but rather in Java. So we cannot add features to our
native loadClass that links with VMClassLoader.loadClass().
I remind our goal: load classes from glibj.zip remotely and only have
the minimum classes stored locally to do this.
As far as we understood, when the bootstrap is called (i.e. current
ClassLoader.parent==null), the method VMClassLoader.loadClass() is
called . When method is loaded from the classpath, i.e. by
SystemClassLoader, findClass() method is called, which does a lot of
things to get the Class bytes and finally call the root class
VM.defineClass() to set some parameters on the class, including name &
security etc., if I am right.
Current behavior acts like this, macro-algorithm:
Class loadClass(String className){
if (bootstrap) {
c= VMClassLoader.loadClass(className);
if (c!=null){
return c;
}
c= RemoteClassLoader.loadClass(className);
if (c!=null){
return c;
}
}
//not bootstrap
else {
c = parent.loadClass(name)
}
// then if this have failed tries to load from classpath as usual
Basically all we have done is add a step at bootstrap. Normally all
classes to perform a RemoteClassLoader.loadClass() can be loaded with
the first step VMClassLoader.loadClass(). In the end we want to have
only these classes in the local core API and the rest of the API
distant.
RemoteClassLoader extends ClassLoader. RemoteClassLoader.loadClass()
get bytes from a URL and then call the parent defineClass() (parent in
sense of Java inheritance, not ClassLoader delegation, this is also a
bit confusing, thus ClassLoader.defineClass()).
ClassLoader.defineClass() finally calls VMClassLoader.defineClass().
Keep in mind that this RemoteClassLoader.loadClass() is aimed at
loading the core API classes.
We have done some basic tests and this seemed to work. However, when
trying with a more complete program, it appears that it throws
IllegalAccessException making me think that some security is added
when using the defineClass() on core API. Normally when calling
VMClassLoader.loadClass(), which is the method that should be used for
bootstrap classes, defineClass() is never called.
I have read (Component Development for Java Platform, Stuart Dabbs
Halloway), that code is the core API or extension will have the
suppresAccessChecks permission whereas code loaded via URLClassLoader
will not have this permission. Another thing is the ProtectionDomain
used by defineClass(). We have investigated this without succes at the
moment...
Thanks for any help and hope I was clear enough in my explainations.
martin
2007/5/1, Tom Tromey <[EMAIL PROTECTED]>:
> "Martin" == Martin Schlienger <[EMAIL PROTECTED]> writes:
Martin> - Is there any place where I can find something like a UML sequence
Martin> diagram of classloading mechanisms.
Nope, sorry.
Martin> It seems there is a lot of back and forth between
Martin> URLClassLoader, ClassLoader, VMClassLoader and all is a little
Martin> bit confusing.
Most of this is delegation. ClassLoaders are arranged in a dynamic
hierarchy and each delegates loading to its parent before trying to
load itself. This chain is rooted in VMClassLoader, which is not a
standard class but rather the way we've chosen to represent the
bootstrap loader inn Classpath .
Martin> - Any suggestion to make a class loaded by another loader to have the
Martin> same rights as the ones loaded with the bootstrap (
Martin> VMClassLoader.loadClass() ). Since we are loading some of the classes
Martin> from glibj.zip with another loader that can access a remote file, but
Martin> ATM we have some IllegalAccessException with some of these. Since we
Martin> use the default ClassLoader's defineClass(), we may have to hack this
Martin> one (no success ATM).
The VM* classes in Classpath are a bit funny. They define an API but
they don't use inheritance for implementations -- instead a given VM
can replace one of the classes entirely and it is simply expected to
conform to the defined API.
So, you can change VMClassLoader to do whatever you like. For
instance, in libgcj we delegate some operations to a URLClassLoader
that we construct after VM startup.
I'm not really sure what you mean about IllegalAccessException and the
default defineClass. As I recall, ultimately all classes are defined
via a method on VMClassLoader.
Tom
Re: Compilation Time | More questions ClassLoader
> "Martin" == Martin Schlienger <[EMAIL PROTECTED]> writes: Martin> - Is there any place where I can find something like a UML sequence Martin> diagram of classloading mechanisms. Nope, sorry. Martin> It seems there is a lot of back and forth between Martin> URLClassLoader, ClassLoader, VMClassLoader and all is a little Martin> bit confusing. Most of this is delegation. ClassLoaders are arranged in a dynamic hierarchy and each delegates loading to its parent before trying to load itself. This chain is rooted in VMClassLoader, which is not a standard class but rather the way we've chosen to represent the bootstrap loader inn Classpath . Martin> - Any suggestion to make a class loaded by another loader to have the Martin> same rights as the ones loaded with the bootstrap ( Martin> VMClassLoader.loadClass() ). Since we are loading some of the classes Martin> from glibj.zip with another loader that can access a remote file, but Martin> ATM we have some IllegalAccessException with some of these. Since we Martin> use the default ClassLoader's defineClass(), we may have to hack this Martin> one (no success ATM). The VM* classes in Classpath are a bit funny. They define an API but they don't use inheritance for implementations -- instead a given VM can replace one of the classes entirely and it is simply expected to conform to the defined API. So, you can change VMClassLoader to do whatever you like. For instance, in libgcj we delegate some operations to a URLClassLoader that we construct after VM startup. I'm not really sure what you mean about IllegalAccessException and the default defineClass. As I recall, ultimately all classes are defined via a method on VMClassLoader. Tom
Re: Compilation Time | More questions ClassLoader
Hi again, Thanks for your answers. back to ClassLoader. Two questions: - Is there any place where I can find something like a UML sequence diagram of classloading mechanisms. It seems there is a lot of back and forth between URLClassLoader, ClassLoader, VMClassLoader and all is a little bit confusing. - Any suggestion to make a class loaded by another loader to have the same rights as the ones loaded with the bootstrap ( VMClassLoader.loadClass() ). Since we are loading some of the classes from glibj.zip with another loader that can access a remote file, but ATM we have some IllegalAccessException with some of these. Since we use the default ClassLoader's defineClass(), we may have to hack this one (no success ATM). Thanks for any information. Martin. 2007/4/26, Robert Schuster <[EMAIL PROTECTED]>: Hi, > There is no rule here as far as I know. We just register all the > charsets we have. Adding a charset is done either because it is > specified by the standard, or because someone needed it. I know I should have answered earlier. The standard charsets are chosen according to Sun's specification: http://java.sun.com/j2se/1.5.0/docs/api/java/nio/charset/Charset.html There is a set of standard charsets which must be available. For everything else we just made available what we had and what others requested (I remember that certain asian charsets where requested). Regards Robert
Re: Compilation Time | More questions -Charsets
Hi, > There is no rule here as far as I know. We just register all the > charsets we have. Adding a charset is done either because it is > specified by the standard, or because someone needed it. I know I should have answered earlier. The standard charsets are chosen according to Sun's specification: http://java.sun.com/j2se/1.5.0/docs/api/java/nio/charset/Charset.html There is a set of standard charsets which must be available. For everything else we just made available what we had and what others requested (I remember that certain asian charsets where requested). Regards Robert signature.asc Description: OpenPGP digital signature
Re: Compilation Time | More questions -Charsets
> "Martin" == Martin Schlienger <[EMAIL PROTECTED]> writes: Martin> Actually since we saw that support for different charsets were Martin> constantly added to GNU/Classpath, this may have been linked with Martin> GNU/Classpath and not jamvm. Martin> Indeed, gnu.java.nio.charset.Provider loads multiple default charsets. Martin> We modify this one as well. It seems that UTF8 , 8859_1 and US_ASCII Martin> are enough to boot. What makes a charset considered mandatory in Martin> GNU/Classpath? Oh, sorry, I thought you meant charsets being searched for at startup or something like that. But instead you seem to be talking about registration of the NIO charsets... is that right? There is no rule here as far as I know. We just register all the charsets we have. Adding a charset is done either because it is specified by the standard, or because someone needed it. I suppose registration could be done more lazily somehow. I haven't looked into it. Tom
Re: Compilation Time | More questions -Charsets
Actually since we saw that support for different charsets were constantly added to GNU/Classpath, this may have been linked with GNU/Classpath and not jamvm. Indeed, gnu.java.nio.charset.Provider loads multiple default charsets. We modify this one as well. It seems that UTF8 , 8859_1 and US_ASCII are enough to boot. What makes a charset considered mandatory in GNU/Classpath? 24 Apr 2007 12:05:04 -0600, Tom Tromey <[EMAIL PROTECTED]>: > "Martin" == Martin Schlienger <[EMAIL PROTECTED]> writes: Martin> Now we are investigating the charsets classes since for sure we don't Martin> need them all on our minimal system. JamVM tries to load a bunch of Martin> them when initializing and we would stick to one (8859 or UTF8 for Martin> example). You may give us a hint again about how to make jamvm/gnu Martin> classpath work with only one charset. I don't know about jamvm in particular. In most places, though, something like 'file.encoding' chooses the encoding to use. The locale may also affect this. So as a first attempt I would make sure that these things are set properly. Tom
Re: Compilation Time | More questions -Charsets
> "Martin" == Martin Schlienger <[EMAIL PROTECTED]> writes: Martin> Now we are investigating the charsets classes since for sure we don't Martin> need them all on our minimal system. JamVM tries to load a bunch of Martin> them when initializing and we would stick to one (8859 or UTF8 for Martin> example). You may give us a hint again about how to make jamvm/gnu Martin> classpath work with only one charset. I don't know about jamvm in particular. In most places, though, something like 'file.encoding' chooses the encoding to use. The locale may also affect this. So as a first attempt I would make sure that these things are set properly. Tom
Re: Compilation Time | More questions -Charsets
ok, thanks for that, I may move to Eclipse if needed. Actually we are hacking the ClassLoader in a way we break Java Spec, so this may not be really interested for main maintainers. The idea is to split the glibj.zip: the very useful class to boot the VM (jamvm) and our custom ClassLoader are kept in a local file while all other are stored in a distant file that is accessed by our ClassLoader. The goal is to have a minimal Java environment to embed on minimal systems and that is still able to run Java apps. This hack was quite easy and seems to work, although we have to check for perfs. Now we are investigating the charsets classes since for sure we don't need them all on our minimal system. JamVM tries to load a bunch of them when initializing and we would stick to one (8859 or UTF8 for example). You may give us a hint again about how to make jamvm/gnu classpath work with only one charset. In a general way, we want to get rid off everything that would be platform specific and keep only what we need. Thanks for any info. 2007/4/23, Mark Wielaard <[EMAIL PROTECTED]>: On Mon, 2007-04-23 at 09:33 -0600, Tom Tromey wrote: > > "Martin" == Martin Schlienger <[EMAIL PROTECTED]> writes: > > Martin> Each time we change the ClassLoader.java class, compilation > Martin> lasts quite a long and I am not really convinced that > Martin> everything need to be recompiled. I am quite a beginner with > Martin> Makefile but it seems that no real check is done and > Martin> everything is compiled by default. > > Martin> Maybe you can give me some tricks on that or tell me what method you > Martin> use when hacking Classpath. > > IMNSHO, the best way to do incremental development on Classpath is to > use Eclipse. It will only recompile the minimum necessary when you > make a change. There a HOWTO for setting this up on the wiki -- it > isn't as trivial as we would like, but it is doable, and once you have > it set up you rarely have to tweak it. That page is here btw: http://developer.classpath.org/mediation/ClasspathHackingWithEclipse It is slightly easier than described there now that the CDT is usually available in most distros out of the box. It sounds like an interesting hack to the ClassLoader. Please do let us know how it works out. Cheers, Mark
Re: Compilation Time
On Mon, 2007-04-23 at 09:33 -0600, Tom Tromey wrote: > > "Martin" == Martin Schlienger <[EMAIL PROTECTED]> writes: > > Martin> Each time we change the ClassLoader.java class, compilation > Martin> lasts quite a long and I am not really convinced that > Martin> everything need to be recompiled. I am quite a beginner with > Martin> Makefile but it seems that no real check is done and > Martin> everything is compiled by default. > > Martin> Maybe you can give me some tricks on that or tell me what method you > Martin> use when hacking Classpath. > > IMNSHO, the best way to do incremental development on Classpath is to > use Eclipse. It will only recompile the minimum necessary when you > make a change. There a HOWTO for setting this up on the wiki -- it > isn't as trivial as we would like, but it is doable, and once you have > it set up you rarely have to tweak it. That page is here btw: http://developer.classpath.org/mediation/ClasspathHackingWithEclipse It is slightly easier than described there now that the CDT is usually available in most distros out of the box. It sounds like an interesting hack to the ClassLoader. Please do let us know how it works out. Cheers, Mark signature.asc Description: This is a digitally signed message part
Re: Compilation Time
> "Martin" == Martin Schlienger <[EMAIL PROTECTED]> writes: Martin> Each time we change the ClassLoader.java class, compilation Martin> lasts quite a long and I am not really convinced that Martin> everything need to be recompiled. I am quite a beginner with Martin> Makefile but it seems that no real check is done and Martin> everything is compiled by default. Martin> Maybe you can give me some tricks on that or tell me what method you Martin> use when hacking Classpath. IMNSHO, the best way to do incremental development on Classpath is to use Eclipse. It will only recompile the minimum necessary when you make a change. There a HOWTO for setting this up on the wiki -- it isn't as trivial as we would like, but it is doable, and once you have it set up you rarely have to tweak it. Tom
