RE: [arch] VM/Classlibrary Interface ( VM Accessors )
Hi Tim, Thanks for your thoughtful comments on the architecture of VM accessors. Ellison, Tim wrote: These are a helper set of singleton Java classes to support Classlibrary implementation. We can instantiate them through an Accessor factory interface ( Modular JVM diagram ). They will provide access to VM functionality not exposed through the public Java api. So this is a classlibrary developer's toolkit ? I assume the goal is to allow implementation of classlibrary functionality in pure Java that currently has to be written using native code (assuming a C-based VM)? We could look at them as a VM access toolkit( beyond access provided by the Java public api ). The requirement may be broader than Classlibraries. One can eg., think of JVMTI agents inserting byte code instrumentation that uses accessors to reach the VM. Your list appears to be a blend of operations that are functional enhancements and optimization enhancements. I expect that the functional enhancements would be discovered by need of the library implementors. The optimization enhancements would likey vary by VM/JIT implementor, with some requiring more 'hints' than others to generate good code. I would expect such functionality to be VM-specific, and therefore part of the kernel classes provided by the VM-implementor. The question is how many accessors it is reasonable to expect such an implementor to be required to support in order to run the Harmony classlibraries. Yes, accessors for functional requirements would easier to standardize on than those for optimization. However, is it worthwhile to define an initial set that can be candidates for standardization? eg., Object Accessor( get/set of fields, method invocation, create instance, serialization support, monitor enter/exit etc. ) Array Accessor( locking, get/set elements bypassing bounds check, etc. ) String Accessor( to support creation, of string objects from native buffers, string access, comparison ) ThreadStack Accessor( to access call stack trace ) Memory Accessor( to support low level operations for memory allocation/deallocation/access ) Etc. A Harmony compliant VM would then expose, at a minimum, this core set of accessor classes with a default implementation of these accessors using JNI, but compliant VM's could choose to provide more optimized implementations also. We want to restrict access to the accessor functionality to the Classlibrary, etc. One way to enforce this policy would be for the accessor factory to check that the requesting class is on some white list of classes before returning an accessor reference. Thoughts on alternative schemes ? The problem with this scheme is that it requires everyone to follow the pattern. There is a risk that somebody may forget to check the list, or that a 'white' intermediary may pass-out a reference to someone who does not have rights to it directly. The componentization model discussed earlier would allow accessors to be in a separate component (which at least would prevent non-white code from ever referencing the accessors). My initial reaction is that we should use such functionality judiciously. Classlibrary code should be using public APIs wherever and whenever possible. The VMs' and JITs' optimization of public API will benefit both the classlibrary implementation and application code; and we don't want to be extending the language by the back door. While optimization of the public api does benefit all Java code, it may not be realistic to hold privileged VM access code to the same standard as application code. The public api has broader architectural targets which require artifacts like standard call frames, access security checks on each invocation, etc. These might become unnecessary if we intend to limit the access to these classes to non-public code only. Using VMAccessors will be predicated on a more robust security story and a fast Java-VM interface. I absolutely agree. Thanks, Rana Dasgupta Intel Managed Runtime Development
Re: [arch] VM/Classlibrary Interface (take 2)
usman bashir wrote: i am looking the same sort of things from IBM guys, as if i am not wrong they claim to do same sort of things before :) and it will really help full if we can have two baselines to work on. This would work better with a diagram, ... IBM has a set of class libraries (only a subset of SE) that have been independently developed, i.e. based on the spec. These are designed to be portable across VM implementations but have been principally written against J9. Other VMs can run the class library by implementing the JNI + VMI interface. Here the VM is defined as the interpreter etc. _plus_ a core set of kernel classes. This structure has already been described here [1]. The class library natives and J9 VM in turn go through the port layer to the OS. I described the port lib briefly here [2]. Apart from 'portability', the port lib structure gives a degree of isolation and control over VM instances by associating resource use with a particular VM instance. Regards, Tim [1] http://mail-archives.apache.org/mod_mbox/incubator-harmony-dev/200507.mbox/[EMAIL PROTECTED] [2] http://mail-archives.apache.org/mod_mbox/incubator-harmony-dev/200509.mbox/[EMAIL PROTECTED] On 8/29/05, Geir Magnusson Jr. [EMAIL PROTECTED] wrote: And on the wiki after posting here, please? :) geir On Aug 19, 2005, at 12:33 PM, Tim Ellison wrote: Weldon Washburn wrote: On 7/11/05, Tim Ellison [EMAIL PROTECTED] wrote: Recently, within IBM, we have been defining the interface between IBM's class library and the J9 VM. We deliberately haven't looked at the GNU Classpath/VM interface specification. The principal goals are to enable the class libraries to be hosted on different versions of a virtual machine, and potentially different virtual machines, without sacrificing performance or introducing complexity. In our design, this results in a number of class types being (architecturally) labeled as 'kernel classes'. Kernel classes can be thought of as part of the VM and have to be written by the VM-provider. With a thoughtful set of kernel classes the API from class library to the VM, and from VM to class libraries, can be kept remarkably small. Our complete VM/Classlibrary interface comprises a short C header (vmi.h), about 18 classes defined by 1.4 public API (java.lang, java.lang.reflect, ...), and two classes that are specifically to support the interface. We are working on necessary extensions to this interface for 1.5. If there is an interest, we can share the interface we are using and evolve it as part of harmony. Tim, It would be good if you would go ahead and post the VM/Classlibrary interface you describe above on harmony wiki. Thanks Weldon I'm just about to leave for a week's vacation, so rather than post and then disappear, I'll wait until I get back and can engage in proper discussion. Regards, Tim It would be great if we could share experiences with the GNU Classpath VM interface in such a way that the Harmony interface was suitable for the widest variety of VMs and class libraries. Regards, Tim -- Tim Ellison ([EMAIL PROTECTED]) IBM Java technology centre, UK. -- Tim Ellison ([EMAIL PROTECTED]) IBM Java technology centre, UK. -- Geir Magnusson Jr +1-203-665-6437 [EMAIL PROTECTED] -- Tim Ellison ([EMAIL PROTECTED]) IBM Java technology centre, UK.
Re: [arch] VM/Classlibrary Interface ( VM Accessors )
Hi Rana, Dasgupta, Rana wrote: I would like to invite discussion and comments from the community on the VM Accessor component in the Harmony Modular JVM diagram, on its functionality, design and implementation. Here are some initial thoughts. These are a helper set of singleton Java classes to support Classlibrary implementation. We can instantiate them through an Accessor factory interface ( Modular JVM diagram ). They will provide access to VM functionality not exposed through the public Java api. So this is a classlibrary developer's toolkit ? I assume the goal is to allow implementation of classlibrary functionality in pure Java that currently has to be written using native code (assuming a C-based VM)? Examples would be object accessors( for direct get/set of object fields, object allocation bypassing constructors, etc. ), array accessors ( to access arrays bypassing bounds checks, manipulating GC on array types ), thread stack accessors etc. Thoughts on how we define a complete list of VM accessor classes? Your list appears to be a blend of operations that are functional enhancements and optimization enhancements. I expect that the functional enhancements would be discovered by need of the library implementors. The optimization enhancements would likey vary by VM/JIT implementor, with some requiring more 'hints' than others to generate good code. How will we implement accessors in Harmony? An option could be using JNI, and distributing/packaging VM accessors with the class library, but there are performance problems associated with this. A proprietory implementation tightly coupled to the VM could expose functionality in the JIT or GC engine. This could potentially perform much better( be inlinable by the JIT ), but how does that impact portability of the accessor classes? I would expect such functionality to be VM-specific, and therefore part of the kernel classes provided by the VM-implementor. The question is how many accessors it is reasonable to expect such an implementor to be required to support in order to run the Harmony classlibraries. We want to restrict access to the accessor functionality to the Classlibrary, etc. One way to enforce this policy would be for the accessor factory to check that the requesting class is on some white list of classes before returning an accessor reference. Thoughts on alternative schemes ? The problem with this scheme is that it requires everyone to follow the pattern. There is a risk that somebody may forget to check the list, or that a 'white' intermediary may pass-out a reference to someone who does not have rights to it directly. The componentization model discussed earlier would allow accessors to be in a separate component (which at least would prevent non-white code from ever referencing the accessors). There seems to be an opportunity for an OpenSource programming model and standardization of portions of the accessor api. My initial reaction is that we should use such functionality judiciously. Classlibrary code should be using public APIs wherever and whenever possible. The VMs' and JITs' optimization of public API will benefit both the classlibrary implementation and application code; and we don't want to be extending the language by the back door. Using VMAccessors will be predicated on a more robust security story and a fast Java-VM interface. Regards, Tim Thanks, Rana Dasgupta Intel Managed Runtime Development -- Tim Ellison ([EMAIL PROTECTED]) IBM Java technology centre, UK.
Re: [arch] VM/Classlibrary Interface (take 2)
Hi Mladen,
I am curious about 'light-weight' native calls for primitive array
type you mentioned below. In the general case, a GC might move the
primitive array while the native method is operating on the array.
Can you tell us how the light-weight interface would deal with this
situation? Does the array need to be pinned before calling the
light-weight native?
Another approach is to disallow GC for the duration of the native
method. I think this works as long as there are guarantees on the
maximum duration of native method execution and the native method is
not allowed to call another java method. The java method called from
the native method might execute an object new which, in turn, might
cause the GC to collect. Of course the GC can't collect because there
is a light-weight native method on the stack that won't allow it.
There might also be similar sorts of issues with a light-weight native
method throwing a java exception.
It woudl be great if you could provide us with more details on your
native method interface ideas.
Thanks
Weldon
On 7/11/05, Mladen Turk [EMAIL PROTECTED] wrote:
The major problem is the way how the native OS abstraction layer is
called. JNI is used as a single native interface from the ground up
and didn't change much for all those years.
For example all JVM's when calling the simple C function:
class Foo { native int add(int x, int y); }
...
int add(int x, int y) { return x + y; }
Are actually calling the:
jint Java_Foo_add(JNIEnv *env, jclass cls, jint x, jint y)
For that native 'add' you have to write a wrapper function, etc...
I propose that we add 'light-weight' native calls that will
enable to directly call the native method that doesn't need to
interfere with the JVM because they return and consume the atomic
parameters.
Further more, the same can be done for the primitive array type.
All this would require additional markup for native functions that can
be set by adding the 'RegisterNativesEx' to the JNI interface and
thus preserving the existing language spec.
Mladen.
--
Weldon Washburn
Intel Managed Runtime Division
[arch] VM/Classlibrary Interface ( VM Accessors )
I would like to invite discussion and comments from the community on the VM Accessor component in the Harmony Modular JVM diagram, on its functionality, design and implementation. Here are some initial thoughts. These are a helper set of singleton Java classes to support Classlibrary implementation. We can instantiate them through an Accessor factory interface ( Modular JVM diagram ). They will provide access to VM functionality not exposed through the public Java api. Examples would be object accessors( for direct get/set of object fields, object allocation bypassing constructors, etc. ), array accessors ( to access arrays bypassing bounds checks, manipulating GC on array types ), thread stack accessors etc. Thoughts on how we define a complete list of VM accessor classes? How will we implement accessors in Harmony? An option could be using JNI, and distributing/packaging VM accessors with the class library, but there are performance problems associated with this. A proprietory implementation tightly coupled to the VM could expose functionality in the JIT or GC engine. This could potentially perform much better( be inlinable by the JIT ), but how does that impact portability of the accessor classes? We want to restrict access to the accessor functionality to the Classlibrary, etc. One way to enforce this policy would be for the accessor factory to check that the requesting class is on some white list of classes before returning an accessor reference. Thoughts on alternative schemes ? There seems to be an opportunity for an OpenSource programming model and standardization of portions of the accessor api. Thanks, Rana Dasgupta Intel Managed Runtime Development
Re: [arch] VM/Classlibrary Interface (take 2)
i am looking the same sort of things from IBM guys, as if i am not wrong they claim to do same sort of things before :) and it will really help full if we can have two baselines to work on. On 8/29/05, Geir Magnusson Jr. [EMAIL PROTECTED] wrote: And on the wiki after posting here, please? :) geir On Aug 19, 2005, at 12:33 PM, Tim Ellison wrote: Weldon Washburn wrote: On 7/11/05, Tim Ellison [EMAIL PROTECTED] wrote: Recently, within IBM, we have been defining the interface between IBM's class library and the J9 VM. We deliberately haven't looked at the GNU Classpath/VM interface specification. The principal goals are to enable the class libraries to be hosted on different versions of a virtual machine, and potentially different virtual machines, without sacrificing performance or introducing complexity. In our design, this results in a number of class types being (architecturally) labeled as 'kernel classes'. Kernel classes can be thought of as part of the VM and have to be written by the VM-provider. With a thoughtful set of kernel classes the API from class library to the VM, and from VM to class libraries, can be kept remarkably small. Our complete VM/Classlibrary interface comprises a short C header (vmi.h), about 18 classes defined by 1.4 public API (java.lang, java.lang.reflect, ...), and two classes that are specifically to support the interface. We are working on necessary extensions to this interface for 1.5. If there is an interest, we can share the interface we are using and evolve it as part of harmony. Tim, It would be good if you would go ahead and post the VM/Classlibrary interface you describe above on harmony wiki. Thanks Weldon I'm just about to leave for a week's vacation, so rather than post and then disappear, I'll wait until I get back and can engage in proper discussion. Regards, Tim It would be great if we could share experiences with the GNU Classpath VM interface in such a way that the Harmony interface was suitable for the widest variety of VMs and class libraries. Regards, Tim -- Tim Ellison ([EMAIL PROTECTED]) IBM Java technology centre, UK. -- Tim Ellison ([EMAIL PROTECTED]) IBM Java technology centre, UK. -- Geir Magnusson Jr +1-203-665-6437 [EMAIL PROTECTED] -- Usman Bashir Certified IBM XML Solution Developer Certified UML Developer Brainbench Certified Internet Perfessional[advance](BCIP) Brainbench Certified Java Perfessional (BCJP) Brainbench Certified .NET Perfessional Brainbench Ceritified C++ Perfessional (BCCP) Software engineer IT24 Faculty Member Operation Badar Lahore
Re: [arch] VM/Classlibrary Interface (take 2)
And on the wiki after posting here, please? :) geir On Aug 19, 2005, at 12:33 PM, Tim Ellison wrote: Weldon Washburn wrote: On 7/11/05, Tim Ellison [EMAIL PROTECTED] wrote: Recently, within IBM, we have been defining the interface between IBM's class library and the J9 VM. We deliberately haven't looked at the GNU Classpath/VM interface specification. The principal goals are to enable the class libraries to be hosted on different versions of a virtual machine, and potentially different virtual machines, without sacrificing performance or introducing complexity. In our design, this results in a number of class types being (architecturally) labeled as 'kernel classes'. Kernel classes can be thought of as part of the VM and have to be written by the VM-provider. With a thoughtful set of kernel classes the API from class library to the VM, and from VM to class libraries, can be kept remarkably small. Our complete VM/Classlibrary interface comprises a short C header (vmi.h), about 18 classes defined by 1.4 public API (java.lang, java.lang.reflect, ...), and two classes that are specifically to support the interface. We are working on necessary extensions to this interface for 1.5. If there is an interest, we can share the interface we are using and evolve it as part of harmony. Tim, It would be good if you would go ahead and post the VM/Classlibrary interface you describe above on harmony wiki. Thanks Weldon I'm just about to leave for a week's vacation, so rather than post and then disappear, I'll wait until I get back and can engage in proper discussion. Regards, Tim It would be great if we could share experiences with the GNU Classpath VM interface in such a way that the Harmony interface was suitable for the widest variety of VMs and class libraries. Regards, Tim -- Tim Ellison ([EMAIL PROTECTED]) IBM Java technology centre, UK. -- Tim Ellison ([EMAIL PROTECTED]) IBM Java technology centre, UK. -- Geir Magnusson Jr +1-203-665-6437 [EMAIL PROTECTED]
Re: [arch] VM/Classlibrary Interface (take 2)
Weldon Washburn wrote: On 7/11/05, Tim Ellison [EMAIL PROTECTED] wrote: Recently, within IBM, we have been defining the interface between IBM's class library and the J9 VM. We deliberately haven't looked at the GNU Classpath/VM interface specification. The principal goals are to enable the class libraries to be hosted on different versions of a virtual machine, and potentially different virtual machines, without sacrificing performance or introducing complexity. In our design, this results in a number of class types being (architecturally) labeled as 'kernel classes'. Kernel classes can be thought of as part of the VM and have to be written by the VM-provider. With a thoughtful set of kernel classes the API from class library to the VM, and from VM to class libraries, can be kept remarkably small. Our complete VM/Classlibrary interface comprises a short C header (vmi.h), about 18 classes defined by 1.4 public API (java.lang, java.lang.reflect, ...), and two classes that are specifically to support the interface. We are working on necessary extensions to this interface for 1.5. If there is an interest, we can share the interface we are using and evolve it as part of harmony. Tim, It would be good if you would go ahead and post the VM/Classlibrary interface you describe above on harmony wiki. Thanks Weldon I'm just about to leave for a week's vacation, so rather than post and then disappear, I'll wait until I get back and can engage in proper discussion. Regards, Tim It would be great if we could share experiences with the GNU Classpath VM interface in such a way that the Harmony interface was suitable for the widest variety of VMs and class libraries. Regards, Tim -- Tim Ellison ([EMAIL PROTECTED]) IBM Java technology centre, UK. -- Tim Ellison ([EMAIL PROTECTED]) IBM Java technology centre, UK.
Re: [arch] VM/Classlibrary Interface (take 2)
On Aug 15, 2005, at 11:48 AM, Weldon Washburn wrote: On 7/12/05, Tim Ellison [EMAIL PROTECTED] wrote: Geir Magnusson Jr. wrote: On Jul 11, 2005, at 12:14 PM, Tim Ellison wrote: Recently, within IBM, we have been defining the interface between IBM's class library and the J9 VM. We deliberately haven't looked at the GNU Classpath/VM interface specification. A few of us at Intel have also done something similar. That is, we created a simple, easy to understand Classpath/VM interface. We also did not look at GNU Classpath because of its license. Hopefully Harmony licensing issues will get resolved soon so that actual code containing interfaces can be donated to Apache. This will make pulling the best ideas from each version of VM/Classlib interface into Apache Harmony much easier Indeed. I have another rev to propose - it will invert the questionnaire to a) make it clear that people will be able to work on *anything* for which there isn't a problem due to previous exposure, rather than only be able to work on things they initially declared b) ensure there's an included resolution procedure for cases where previous exposure is resolvable c) Make the compontent list a bit finer grained so previous exposure to X doesn't keep you from working on things unrelated solely because of our bucketing. geir -- Geir Magnusson Jr +1-203-665-6437 [EMAIL PROTECTED]
Re: [arch] VM/Classlibrary Interface (take 2)
Hi Weldon, Weldon Washburn wrote: On 7/12/05, Tim Ellison [EMAIL PROTECTED] wrote: Geir Magnusson Jr. wrote: On Jul 11, 2005, at 12:14 PM, Tim Ellison wrote: Recently, within IBM, we have been defining the interface between IBM's class library and the J9 VM. We deliberately haven't looked at the GNU Classpath/VM interface specification. A few of us at Intel have also done something similar. That is, we created a simple, easy to understand Classpath/VM interface. We also did not look at GNU Classpath because of its license. Hopefully Harmony licensing issues will get resolved soon so that actual code containing interfaces can be donated to Apache. This will make pulling the best ideas from each version of VM/Classlib interface into Apache Harmony much easier Agreed. Geir has been refining the Authorized Contributor Questionnaire [1] document and it looks pretty good to me. I'd just like to see the list of components in Part II tweaked to maximize participation in Harmony while still ensuring the protection from previous exposure. But that's a different subject. [1] http://incubator.apache.org/harmony/auth_cont_quest.html The principal goals are to enable the class libraries to be hosted on different versions of a virtual machine, and potentially different virtual machines, without sacrificing performance or introducing complexity. Yes these are the correct goals and it is fairly straight forward to achieve them. I connected GNU Classpath to ORP several years back. My experience indicates that moving between Harmony Class Classpath and GNU Classpath should not be that much engineering effort. In other words, it's not really worth debating. Good -- hopefully we can be in a position where simple bridging code maps between the interfaces; then we can mix and match. As I said before, there are other VM-specific classes that are entirely the responsibility of the VM developer (e.g. Class, Thread, etc.) and these are covered by regular Java API specifications. A few classes are predominantly common (i.e. reusable) code. VM developers can choose to either implement the entire class, or pick up the common code and implement one or two methods to complete them (e.g. AccessController, and String). Regards, Tim This basically matches our experience in the interface between the VM and the classpath. Again, this interface is one probably one of the easiest interfaces to change in a modular JVM. The idea of a generic common code implementation that can be replaced by a vm specific implementation is classic software engineering practice. It's a good idea, not new and should be used in the Classpath/VM interface. Probably best if we don't call the Harmony code classpath though ;-) Let's simply call it the ClassLibrary/VM interface. -- Tim Ellison ([EMAIL PROTECTED]) IBM Java technology centre, UK.
Re: [arch] VM/Classlibrary Interface (take 2)
On 7/12/05, Tim Ellison [EMAIL PROTECTED] wrote: Geir Magnusson Jr. wrote: On Jul 11, 2005, at 12:14 PM, Tim Ellison wrote: Recently, within IBM, we have been defining the interface between IBM's class library and the J9 VM. We deliberately haven't looked at the GNU Classpath/VM interface specification. A few of us at Intel have also done something similar. That is, we created a simple, easy to understand Classpath/VM interface. We also did not look at GNU Classpath because of its license. Hopefully Harmony licensing issues will get resolved soon so that actual code containing interfaces can be donated to Apache. This will make pulling the best ideas from each version of VM/Classlib interface into Apache Harmony much easier The principal goals are to enable the class libraries to be hosted on different versions of a virtual machine, and potentially different virtual machines, without sacrificing performance or introducing complexity. Yes these are the correct goals and it is fairly straight forward to achieve them. I connected GNU Classpath to ORP several years back. My experience indicates that moving between Harmony Class Classpath and GNU Classpath should not be that much engineering effort. In other words, it's not really worth debating. As I said before, there are other VM-specific classes that are entirely the responsibility of the VM developer (e.g. Class, Thread, etc.) and these are covered by regular Java API specifications. A few classes are predominantly common (i.e. reusable) code. VM developers can choose to either implement the entire class, or pick up the common code and implement one or two methods to complete them (e.g. AccessController, and String). Regards, Tim This basically matches our experience in the interface between the VM and the classpath. Again, this interface is one probably one of the easiest interfaces to change in a modular JVM. The idea of a generic common code implementation that can be replaced by a vm specific implementation is classic software engineering practice. It's a good idea, not new and should be used in the Classpath/VM interface.
Re: [arch] VM/Classlibrary Interface (take 2)
On Jul 20, 2005, at 7:02 PM, Mark Wielaard wrote: Hi, On Tue, 2005-07-19 at 12:38 -0400, Geir Magnusson Jr. wrote: The reason nobody answered this question is because we are still debating how to accept code that is both GPLv2 and ASLv2 compatible. I'm not sure that's the issue exactly - I think it's about dual licensing. No it is not. That would create all kinds of trouble for us to track who contributed what under which terms. This really is just about coming up with a simple default contribution policy that is acceptable to all. As Leo explained so graphically people don't want lists of options or difficult decision diagrams. We want to provide a simple contribution policy that gets us in a situation that all contributed code is ASLv2 and GPLv2 compatible by default. Then there is no confusion or need to track things (except of course the FSF/ASF individual/company contribution/disclaimer paperwork). But we don't host code at the ASF that's under other licenses except in extreme situations, and I'm not convinced that we're there yet. We could do that by asking every contribution to be dual licensed GPL/ASL, but that might be too confusing to some. So I propose we just pick MIT/X for now and be done with it. (Or any other option I mentioned in a couple of mails if you really don't like MIT/X.) I believe that your only problem with ALv2 are the patent termination clauses, right? I have an idea, but need to do a bit of homework - back in a few hours with it... geir -- Geir Magnusson Jr +1-203-665-6437 [EMAIL PROTECTED]
Re: [arch] VM/Classlibrary Interface (take 2)
Hi Geir, On Tue, 2005-07-19 at 12:32 -0400, Geir Magnusson Jr. wrote: If there are non-trivial pieces of code, they can be posted here if or should be submitted into a JIRA, choosing to contribute under the apache license, and then post a message here. On Mon, 2005-07-18 at 19:40 +0200, [EMAIL PROTECTED] wrote: The reason nobody answered this question is because we are still debating how to accept code that is both GPLv2 and ASLv2 compatible. As long as that isn't worked out the best thing to do with such code is to post them on some website with a simple MIT/X disclaimer at the top and post on the mailinglist at which URL people can check it out. (And having a individual/company assignment/disclaimer on file with either the FSF or ASF is recommended.) We seem to be giving mixed signals here. Can we try to coordinate a bit more on the policies we want to have for this project? Even if we cannot agree on some things it would be good to setup some resolution mechanism. Thanks, Mark signature.asc Description: This is a digitally signed message part
Re: [arch] VM/Classlibrary Interface (take 2)
Hi, On Tue, 2005-07-19 at 12:38 -0400, Geir Magnusson Jr. wrote: The reason nobody answered this question is because we are still debating how to accept code that is both GPLv2 and ASLv2 compatible. I'm not sure that's the issue exactly - I think it's about dual licensing. No it is not. That would create all kinds of trouble for us to track who contributed what under which terms. This really is just about coming up with a simple default contribution policy that is acceptable to all. As Leo explained so graphically people don't want lists of options or difficult decision diagrams. We want to provide a simple contribution policy that gets us in a situation that all contributed code is ASLv2 and GPLv2 compatible by default. Then there is no confusion or need to track things (except of course the FSF/ASF individual/company contribution/disclaimer paperwork). We could do that by asking every contribution to be dual licensed GPL/ASL, but that might be too confusing to some. So I propose we just pick MIT/X for now and be done with it. (Or any other option I mentioned in a couple of mails if you really don't like MIT/X.) Cheers, Mark signature.asc Description: This is a digitally signed message part
Re: [arch] VM/Classlibrary Interface (take 2)
Hi Johnson! it is really good, some what like COM (querying IUknown ,sorry for having some term from nonjava ;) ), yeah i think so this way not only promote the interop as well as we can replace the each trianger side (VM, OS layer and class library). though i suppose the efficieny can be a constraint, but supporting interfaces between every layer could allow us to merge and replace each componenet effectively (surely another dependency creep up that the componenet must follow the rules i.e. interfaces included in the proposed harmony.) but i think so it will still not costly because , we are suppose to create harmony in this universe. And what i m thinking about the harmony is kind of thing like Eclipse (excellent thing i should cong to them as well) so i want plug ins (and here plug ins will be class library and os layer etc). what u think ? On 7/18/05, [EMAIL PROTECTED] [EMAIL PROTECTED] wrote: The mailing list is probably not the right forum for posting non-trivial pieces of code. Geir: What's the right forum for this sort of discussion? I am checking every day to see if the answer to this has popped up, but no luck so far. In case I missed something, could someone point me into the right direction? The reason nobody answered this question is because we are still debating how to accept code that is both GPLv2 and ASLv2 compatible. As long as that isn't worked out the best thing to do with such code is to post them on some website with a simple MIT/X disclaimer at the top and post on the mailinglist at which URL people can check it out. (And having a individual/company assignment/disclaimer on file with either the FSF or ASF is recommended.) Cheers, Mark P.S. I am currently with poor connectivity. I have written replies to some of the discussion already, but have no way to actually get those off my laptop. Back in a day or two I hope. -- Usman Bashir Certified IBM XML Solution Developer Certified UML Developer Brainbench Certified Internet Perfessional[advance](BCIP) Brainbench Certified Java Perfessional (BCJP) Brainbench Certified .NET Perfessional Brainbench Ceritified C++ Perfessional (BCCP) Software engineer IT24 Faculty Member Operation Badar Lahore
Re: [arch] VM/Classlibrary Interface (take 2)
On Jul 18, 2005, at 12:48 PM, Zsejki Sorin Miklós wrote: The mailing list is probably not the right forum for posting non- trivial pieces of code. Geir: What's the right forum for this sort of discussion? I am checking every day to see if the answer to this has popped up, but no luck so far. In case I missed something, could someone point me into the right direction? Oh - I thought it was answered. If there are non-trivial pieces of code, they can be posted here if or should be submitted into a JIRA, choosing to contribute under the apache license, and then post a message here. geir -- Geir Magnusson Jr +1-203-665-6437 [EMAIL PROTECTED]
Re: [arch] VM/Classlibrary Interface (take 2)
On Jul 18, 2005, at 1:40 PM, [EMAIL PROTECTED] wrote: The mailing list is probably not the right forum for posting non- trivial pieces of code. Geir: What's the right forum for this sort of discussion? I am checking every day to see if the answer to this has popped up, but no luck so far. In case I missed something, could someone point me into the right direction? The reason nobody answered this question is because we are still debating how to accept code that is both GPLv2 and ASLv2 compatible. I'm not sure that's the issue exactly - I think it's about dual licensing. I'm all for any contributor also offering their code under other licenses, but that's not going to be something we track here. We can encourage, but for this list and our JIRA, it's AL at the moment. As long as that isn't worked out the best thing to do with such code is to post them on some website with a simple MIT/X disclaimer at the top and post on the mailinglist at which URL people can check it out. (And having a individual/company assignment/disclaimer on file with either the FSF or ASF is recommended.) Well, what you could do is post here, and then also note it's available somewhere else under another license. geir Cheers, Mark P.S. I am currently with poor connectivity. I have written replies to some of the discussion already, but have no way to actually get those off my laptop. Back in a day or two I hope. -- Geir Magnusson Jr +1-203-665-6437 [EMAIL PROTECTED]
Re: [arch] VM/Classlibrary Interface (take 2)
The mailing list is probably not the right forum for posting non-trivial pieces of code. Geir: What's the right forum for this sort of discussion? I am checking every day to see if the answer to this has popped up, but no luck so far. In case I missed something, could someone point me into the right direction?
Re: [arch] VM/Classlibrary Interface (take 2)
The mailing list is probably not the right forum for posting non-trivial pieces of code. Geir: What's the right forum for this sort of discussion? I am checking every day to see if the answer to this has popped up, but no luck so far. In case I missed something, could someone point me into the right direction? The reason nobody answered this question is because we are still debating how to accept code that is both GPLv2 and ASLv2 compatible. As long as that isn't worked out the best thing to do with such code is to post them on some website with a simple MIT/X disclaimer at the top and post on the mailinglist at which URL people can check it out. (And having a individual/company assignment/disclaimer on file with either the FSF or ASF is recommended.) Cheers, Mark P.S. I am currently with poor connectivity. I have written replies to some of the discussion already, but have no way to actually get those off my laptop. Back in a day or two I hope.
Re: [arch] VM/Classlibrary Interface (take 2)
Akhilesh Shirbhate [EMAIL PROTECTED] wrote on 07/12/2005
05:18:20 AM:
Can we have a look at the vmi.h and the list of 18 classes, and
specially the two classes required for integration ?
As a follow-up to Tim Ellison's response let me provide more detail on
J9's VM Interface (VMI) and Kernel classes. Here is the list of
VM-dependent ?Kernel classes?. Tim provided some additional details about
com.ibm.oti.lang.SystemProcess and com.ibm.oti.vm.VM in a follow-up to his
original post.
java.lang
Class
ClassLoader
Compiler
Object
Package
Runtime
StackTraceElement
String
Thread
ThreadGroup
Throwable
java.lang.ref
Reference
ReferenceQueue
java.lang.reflect
AccessibleObject
Array
Constructor
Field
Method
java.security
AccessControlContext
AccessController
com.ibm.oti.lang
SystemProcess
com.ibm.oti.vm
VM
As Tim stated, the role of the VMI is to insulate the Java class library
implementation from internal knowledge of the VM. Recall that the VMI is
used by the portion of the class library that is not tightly tied to the
virtual machine (i.e: non-Kernel classes). The vast majority of
non-Kernel classes can be implemented using a combination of Java and
'pure' JNI code.
The VMI is a C-based interface, similar to JNI, which provides access to
VM features that are outside the scope of JNI. The meat of vmi.h is the
following structure declaration (somewhat simplified) which defines the
API between the class library and VM:
typedef struct VMInterfaceFunctions_* VMInterface;
struct VMInterfaceFunctions_
{
vmiError (JNICALL * CheckVersion)(VMInterface* vmi, vmiVersion*
version);
JavaVM* (JNICALL * GetJavaVM) (VMInterface* vmi);
J9PortLibrary* (JNICALL * GetPortLibrary) (VMInterface* vmi);
JavaVMInitArgs* (JNICALL * GetInitArgs) (VMInterface* vmi);
vmiError (JNICALL * GetSystemProperty) (VMInterface* vmi, char*
key, char** valuePtr);
vmiError (JNICALL * SetSystemProperty) (VMInterface* vmi, char*
key, char* value);
vmiError (JNICALL * CountSystemProperties) (VMInterface* vmi, int*
countPtr);
vmiError (JNICALL * IterateSystemProperties) (VMInterface* vmi,
vmiSystemPropertyIterator iterator, void* userData);
};
Like the JNIEnv the VMInterface is a structure that contains function
pointers and is associated 1:1 with a JavaVM.
Structurally, the VMI is built as a shared library (.dll/.so) that exports
two functions which return a VMInterface given either a JavaVM or a
JNIEnv. The class library JNI code is dynamically linked to this shared
library, and is VM-neutral. The VMI shared library can be replaced to
support different VM implementations without modifying the JNI natives
themselves.
The following sample illustrates how a JNI native could obtain a
VMInterface and make a simple VMI query. Note the intentional similarity
between the JNI and VMI usage.
jstring JNICALL Java_example_Example_getJavaHome(JNIEnv *env, jclass
clazz)
{
/* Obtain the VM Interface from the JNIEnv */
VMInterface * vmi = VMI_GetVMIFromJNIEnv(env);
char* javaHome;
/* Query the value of java.home using VMI */
if ((*vmi)-GetSystemProperty(vmi, java.home,javaHome) !=
VMI_ERROR_NONE)
return NULL;
return (*env)-NewStringUTF(env,javaHome);
}
The GetPortLibrary() member of the VMInterface is particularly
interesting. This function returns a handle to a J9PortLibrary structure,
which provides a platform-independent abstraction of operating system
functionality. The portability layer includes those features required by
both the virtual machine and core class library, including:
Basic I/O (files, sockets, tty)
Synchronization and threading
Memory allocation
Signal handling
Shared library support
Operating system and environmental queries
During the development of J9 we found that centralizing VM and core class
library interfaces to the operating system (and keeping these interfaces
thin) reduces the porting effort. If there is sufficient interest I can
follow up with a more detailed description of the J9 portability library.
The mailing list is probably not the right forum for posting non-trivial
pieces of code. Geir: What's the right forum for this sort of discussion?
Graeme Johnson
J9 VM Team, IBM Canada.
Re: [arch] VM/Classlibrary Interface (take 2)
Can we have a look at the vmi.h and the list of 18 classes, and specially the two classes required for integration ? Besides, I would also like to know the changes/extensions you have thought till date for 1.5 spec. To me, it seems that there should be a lot of extensions given the fact that 1.5 has put in more efforts to give closer insider look in the VM. -Akhilesh. On Jul 11, 2005, at 12:14 PM, Tim Ellison wrote: The principal goals are to enable the class libraries to be hosted on different versions of a virtual machine, and potentially different virtual machines, without sacrificing performance or introducing complexity. In our design, this results in a number of class types being (architecturally) labeled as 'kernel classes'. Kernel classes can be thought of as part of the VM and have to be written by the VM-provider. With a thoughtful set of kernel classes the API from class library to the VM, and from VM to class libraries, can be kept remarkably small. Our complete VM/Classlibrary interface comprises a short C header (vmi.h), about 18 classes defined by 1.4 public API ^^ (java.lang, java.lang.reflect, ...), and two classes that are specifically to support the interface. We are working on necessary extensions to this interface for 1.5.
Re: [arch] VM/Classlibrary Interface (take 2)
Geir Magnusson Jr. wrote: On Jul 11, 2005, at 12:14 PM, Tim Ellison wrote: Recently, within IBM, we have been defining the interface between IBM's class library and the J9 VM. We deliberately haven't looked at the GNU Classpath/VM interface specification. The principal goals are to enable the class libraries to be hosted on different versions of a virtual machine, and potentially different virtual machines, without sacrificing performance or introducing complexity. In our design, this results in a number of class types being (architecturally) labeled as 'kernel classes'. Kernel classes can be thought of as part of the VM and have to be written by the VM-provider. With a thoughtful set of kernel classes the API from class library to the VM, and from VM to class libraries, can be kept remarkably small. Our complete VM/Classlibrary interface comprises a short C header (vmi.h), about 18 classes defined by 1.4 public API (java.lang, java.lang.reflect, ...), and two classes that are specifically to support the interface. We are working on necessary extensions to this interface for 1.5. what is in the two classes that are needed to support the interface? IOW, what functionality is needed? Some parts of the class library implementation require functionality that is not part of the regular Java API and is (typically) VM-specific. In our interface, these are pulled out into the VM and SystemProcess classes, and it becomes the responsibility of the VM-author to provide an implementation. All the methods on the VM class are static. The javadoc for these classes has a full specification, but briefly, the VM class has: static ClassLoader getStackClassLoader(int) Answers the ClassLoader for the method at the given stack depth. static ClassLoader getNonBootstrapClassLoader() Answers the most recent non-null/non-bootstrap class loader on the current stack. static ClassLoader callerClassLoader() Answers the class loader of the method that called the caller. static void shutdown() The vm must invoke this method when all other threads have been shutdown. This is the final shutdown sequence. static void closeJars() Informs the VM that the class libs opened a JAR cache, and a call to close the cache should be made during VM shutdown sequence. static void deleteOnExit() Informs the VM that the class libs have created some 'delete on exit' tasks, and a call to invoke the tasks should be made during the VM shutdown sequence. The SystemProcess class is simply the concrete implementation of the java.lang.Process abstract class. It is used in the implementation of Runtime#exec() methods. As I said before, there are other VM-specific classes that are entirely the responsibility of the VM developer (e.g. Class, Thread, etc.) and these are covered by regular Java API specifications. A few classes are predominantly common (i.e. reusable) code. VM developers can choose to either implement the entire class, or pick up the common code and implement one or two methods to complete them (e.g. AccessController, and String). Regards, Tim If there is an interest, we can share the interface we are using and evolve it as part of harmony. It would be great if we could share experiences with the GNU Classpath VM interface in such a way that the Harmony interface was suitable for the widest variety of VMs and class libraries. Regards, Tim -- Tim Ellison ([EMAIL PROTECTED]) IBM Java technology centre, UK. Archie Cobbs wrote: Geir Magnusson Jr. wrote: Ok, from the school of Storming the Gates! Take 2, lets again examine the question of VM/classlib interface as this is an important aspect to address and our first run at it wasn't so successful. The questions I have are all around the different ways has this been done. So far we know about GNU Classpath, but I assume there are others. What I'm specifically interested in is : - what were the architectural goals - what mistakes made in the past did you try to avoid - what are the known limitations - does the interface support our target version of 1.5 Just my opinion: I don't think the VM/classlib API matters that much. Seriously... I've never done VM work, but system-level work that I have done can live or die based on the API, for all sorts of reasons, like performance, the ability to elegantly evolve over time, etc. I think you're right -- in general -- that API design can have a major impact on things. I'm just saying that in this specific case it doesn't look like it's so critical (to me). The API is private to the VM implementation, so the only effect it can have on application code is how efficient it is. The API isn't private to the VM implementation, is it? The _implementation_ of the API is, but not the API itself - that's a contract between the classlib and VM... You're confusing two things: the API that applications
Re: [arch] VM/Classlibrary Interface (take 2)
Tim Ellison wrote: Recently, within IBM, we have been defining the interface between IBM's class library and the J9 VM. We deliberately haven't looked at the GNU Classpath/VM interface specification. snip/ If there is an interest, we can share the interface we are using and evolve it as part of harmony. It would be great if we could share experiences with the GNU Classpath VM interface in such a way that the Harmony interface was suitable for the widest variety of VMs and class libraries. Cool! It'd be real interesting to try and get n-way interop (eg use classpath and the J9 VM together, use the J9 classlib and gcj together, ...). I can see how much good stuff could come from that. cheers, Leo
Re: [arch] VM/Classlibrary Interface (take 2)
On Jul 10, 2005, at 5:21 PM, Archie Cobbs wrote: Geir Magnusson Jr. wrote: Ok, from the school of Storming the Gates! Take 2, lets again examine the question of VM/classlib interface as this is an important aspect to address and our first run at it wasn't so successful. The questions I have are all around the different ways has this been done. So far we know about GNU Classpath, but I assume there are others. What I'm specifically interested in is : - what were the architectural goals - what mistakes made in the past did you try to avoid - what are the known limitations - does the interface support our target version of 1.5 Just my opinion: I don't think the VM/classlib API matters that much. Rather, what's more important is making the runtime transitions into and out of the JVM (i.e., through this API) seamless. This is mostly a property of the VM implementation rather than the API design. Hm. I'm not sure if I agree, as I can think of some really awful APIs between the VM and the classlibrary. XML, for example... public String exec(String xmlInput) (wait, that's the semantic description of webservices, isn't it...) Seriously... I've never done VM work, but system-level work that I have done can live or die based on the API, for all sorts of reasons, like performance, the ability to elegantly evolve over time, etc. The API is private to the VM implementation, so the only effect it can have on application code is how efficient it is. The API isn't private to the VM implementation, is it? The _implementation_ of the API is, but not the API itself - that's a contract between the classlib and VM... Classpath has defined a reasonable API. I can't imagine how rearranging the API dramatically would make things noticably better, either in performance or code clarity/maintainability. That's just the question I'm interested in exploring. I'm not starting out with an predetermined answer, but rather know that there are others in the universe that have thought about this, and am interested in what they think. For example, when people were porting their platforms to use GNU Classpath's API, what did they think? Did they have to redo anything internally? Can't comment on the 1.5 aspects, but the design goal of Classpath seems to have been a combination of: 1. Make the set of VM native calls not too far from the minimal possible set (i.e., don't add native methods for things that can be done fairly easily in pure Java (plus other native methods)). yep - that makes sense. 2. Keep the API reasonably easy to understand. yep 3. Don't worry about inlining Java code: assume the VM can do 'easy' inlining like invoking static methods. How does that aspect matter to the VM/classlib interface? geir Again, just my opinion. -Archie __ Archie Cobbs *CTO, Awarix* http:// www.awarix.com -- Geir Magnusson Jr +1-203-665-6437 [EMAIL PROTECTED]
Re: [arch] VM/Classlibrary Interface (take 2)
3. Don't worry about inlining Java code: assume the VM can do 'easy' inlining like invoking static methods. How does that aspect matter to the VM/classlib interface? It matters in that when defining the VM/classlib interface you should assume that adding a level of easy to inline wrapper functions will have zero performance impact. Eg, Object.clone isn't directly declared to be a native method, it simply delegates to VMObject.clone, which may or may not be a native method depending on the VM implementation. Trust that the JIT will inline away the trivial wrapper method (java.lang.Object.clone) so that delagating the real work to a method of VMObject won't cost you anything. --dave
Re: [arch] VM/Classlibrary Interface (take 2)
Geir Magnusson Jr. wrote: Ok, from the school of Storming the Gates! Take 2, lets again examine the question of VM/classlib interface as this is an important aspect to address and our first run at it wasn't so successful. The questions I have are all around the different ways has this been done. So far we know about GNU Classpath, but I assume there are others. What I'm specifically interested in is : - what were the architectural goals - what mistakes made in the past did you try to avoid - what are the known limitations - does the interface support our target version of 1.5 Just my opinion: I don't think the VM/classlib API matters that much. Seriously... I've never done VM work, but system-level work that I have done can live or die based on the API, for all sorts of reasons, like performance, the ability to elegantly evolve over time, etc. I think you're right -- in general -- that API design can have a major impact on things. I'm just saying that in this specific case it doesn't look like it's so critical (to me). The API is private to the VM implementation, so the only effect it can have on application code is how efficient it is. The API isn't private to the VM implementation, is it? The _implementation_ of the API is, but not the API itself - that's a contract between the classlib and VM... You're confusing two things: the API that applications use (which is strictly a Java API you can read about via Javadoc) and the API that the class library uses, via native method calls, to talk directly to the VM. The latter is not publicly visible (and usually VM specific). Now, there may be public methods such as Thread.start() that just happen to be implemented via native method calls directly into the VM, but those are degenerate cases. The application doesn't know that things happen to be implemented that way and that it might be making a direct native call into the VM. By the way, Classpath avoids any such degenerate cases: all methods in class Foo that could be native instead are called through a liason class VMFoo. So all of the Classpath/VM API native methods are non-public. (This is why I said Classpath assumes the VM can do easy inlining, because of the Foo - VMFoo extra hops). The API that applications use is defined by the public classes and methods in the class library. How those classes are implemented is not known to the application (and the application shouldn't care). Which ones actually involve calls into the VM is irrelevant. E.g., have you ever seen the Javadoc for a method contain the native keyword? No -- that keyword is stripped out because it's irrelevant from an API point of view (so is synchronized). Instead, those are implementation details. The application just sees a big Java class library. The class library sees a handful of native methods into the VM. The VM sees an operating system of some sort. Etc. For example, when people were porting their platforms to use GNU Classpath's API, what did they think? Did they have to redo anything internally? In my experience, things mostly made sense, but there were definitely some problems. However, they were there more because nobody had gotten around to fixing them, rather than some conflict of API design vision or anything like that. For example, the API call to load native libraries didn't have a ClassLoader parameter. But that is required to properly implement Java 2 style class loading (and unloading). So we added one. There was nothing controversial about it. Similarly for lots of other stuff. In other words, there hasn't been much debate on the Classpath lists about APIs. It's pretty straightforward, and in for the most part the little details don't matter that much anyway. This is why many of us who work on Classpath are wondering why anyone would bother wondering how to reinvent the wheel, when the wheel already rolls just fine :-) Cheers, -Archie __ Archie Cobbs *CTO, Awarix* http://www.awarix.com
Re: [arch] VM/Classlibrary Interface (take 2)
Recently, within IBM, we have been defining the interface between IBM's class library and the J9 VM. We deliberately haven't looked at the GNU Classpath/VM interface specification. The principal goals are to enable the class libraries to be hosted on different versions of a virtual machine, and potentially different virtual machines, without sacrificing performance or introducing complexity. In our design, this results in a number of class types being (architecturally) labeled as 'kernel classes'. Kernel classes can be thought of as part of the VM and have to be written by the VM-provider. With a thoughtful set of kernel classes the API from class library to the VM, and from VM to class libraries, can be kept remarkably small. Our complete VM/Classlibrary interface comprises a short C header (vmi.h), about 18 classes defined by 1.4 public API (java.lang, java.lang.reflect, ...), and two classes that are specifically to support the interface. We are working on necessary extensions to this interface for 1.5. If there is an interest, we can share the interface we are using and evolve it as part of harmony. It would be great if we could share experiences with the GNU Classpath VM interface in such a way that the Harmony interface was suitable for the widest variety of VMs and class libraries. Regards, Tim -- Tim Ellison ([EMAIL PROTECTED]) IBM Java technology centre, UK. Archie Cobbs wrote: Geir Magnusson Jr. wrote: Ok, from the school of Storming the Gates! Take 2, lets again examine the question of VM/classlib interface as this is an important aspect to address and our first run at it wasn't so successful. The questions I have are all around the different ways has this been done. So far we know about GNU Classpath, but I assume there are others. What I'm specifically interested in is : - what were the architectural goals - what mistakes made in the past did you try to avoid - what are the known limitations - does the interface support our target version of 1.5 Just my opinion: I don't think the VM/classlib API matters that much. Seriously... I've never done VM work, but system-level work that I have done can live or die based on the API, for all sorts of reasons, like performance, the ability to elegantly evolve over time, etc. I think you're right -- in general -- that API design can have a major impact on things. I'm just saying that in this specific case it doesn't look like it's so critical (to me). The API is private to the VM implementation, so the only effect it can have on application code is how efficient it is. The API isn't private to the VM implementation, is it? The _implementation_ of the API is, but not the API itself - that's a contract between the classlib and VM... You're confusing two things: the API that applications use (which is strictly a Java API you can read about via Javadoc) and the API that the class library uses, via native method calls, to talk directly to the VM. The latter is not publicly visible (and usually VM specific). Now, there may be public methods such as Thread.start() that just happen to be implemented via native method calls directly into the VM, but those are degenerate cases. The application doesn't know that things happen to be implemented that way and that it might be making a direct native call into the VM. By the way, Classpath avoids any such degenerate cases: all methods in class Foo that could be native instead are called through a liason class VMFoo. So all of the Classpath/VM API native methods are non-public. (This is why I said Classpath assumes the VM can do easy inlining, because of the Foo - VMFoo extra hops). The API that applications use is defined by the public classes and methods in the class library. How those classes are implemented is not known to the application (and the application shouldn't care). Which ones actually involve calls into the VM is irrelevant. E.g., have you ever seen the Javadoc for a method contain the native keyword? No -- that keyword is stripped out because it's irrelevant from an API point of view (so is synchronized). Instead, those are implementation details. The application just sees a big Java class library. The class library sees a handful of native methods into the VM. The VM sees an operating system of some sort. Etc. For example, when people were porting their platforms to use GNU Classpath's API, what did they think? Did they have to redo anything internally? In my experience, things mostly made sense, but there were definitely some problems. However, they were there more because nobody had gotten around to fixing them, rather than some conflict of API design vision or anything like that. For example, the API call to load native libraries didn't have a ClassLoader parameter. But that is required to properly implement Java 2 style class loading (and unloading). So we added
Re: [arch] VM/Classlibrary Interface (take 2)
Geir Magnusson Jr. wrote:
- what were the architectural goals
- what mistakes made in the past did you try to avoid
- what are the known limitations
- does the interface support our target version of 1.5
Anyone who has experience, please post it here, describing the
particulars rather than just point someone to docs or code that may be
under a license not acceptable by everyone. That said, please include
any references too...
The major problem is the way how the native OS abstraction layer is
called. JNI is used as a single native interface from the ground up
and didn't change much for all those years.
For example all JVM's when calling the simple C function:
class Foo { native int add(int x, int y); }
...
int add(int x, int y) { return x + y; }
Are actually calling the:
jint Java_Foo_add(JNIEnv *env, jclass cls, jint x, jint y)
For that native 'add' you have to write a wrapper function, etc...
I propose that we add 'light-weight' native calls that will
enable to directly call the native method that doesn't need to
interfere with the JVM because they return and consume the atomic
parameters.
Further more, the same can be done for the primitive array type.
All this would require additional markup for native functions that can
be set by adding the 'RegisterNativesEx' to the JNI interface and
thus preserving the existing language spec.
Mladen.
Re: [arch] VM/Classlibrary Interface (take 2)
Mladen Turk wrote: The major problem is the way how the native OS abstraction layer is called. JNI is used as a single native interface from the ground up and didn't change much for all those years. Almost all VMs have their own proprietary non-JNI native method interface that is much more efficient. E.g., JVCM has the JCNI which actually has zero overhead (this is possible because in JC both executing Java code and JCNI native code are are just compiled C code). On the other hand, optimizations (such as method inlining) involving native code are often difficult or impossible, so they still can have an indirect cost. -Archie __ Archie Cobbs *CTO, Awarix* http://www.awarix.com
Re: [arch] VM/Classlibrary Interface (take 2)
Archie Cobbs wrote: Mladen Turk wrote: The major problem is the way how the native OS abstraction layer is called. JNI is used as a single native interface from the ground up and didn't change much for all those years. Almost all VMs have their own proprietary non-JNI native method interface that is much more efficient. I didn't speak on non-JNI, but rather making sense to the JNI (JVM internally). For calling a simple native methods that don't need the complete JVM runtime status, the call itself should not have higher cost the calling any non-inlinable method. So just by marking that the native method as 'simple' will make classpath interface much faster. The majority of native methods don't need the exception mechanism or new object creation. All that can be done in the JVM itself by properly writing the classpath itself (of course if the atomic access to the native functions is low). Regards, Mladen.
Re: [arch] VM/Classlibrary Interface (take 2)
On Jul 11, 2005, at 11:36 AM, Archie Cobbs wrote: Geir Magnusson Jr. wrote: The API is private to the VM implementation, so the only effect it can have on application code is how efficient it is. The API isn't private to the VM implementation, is it? The _implementation_ of the API is, but not the API itself - that's a contract between the classlib and VM... You're confusing two things: the API that applications use (which is strictly a Java API you can read about via Javadoc) and the API that the class library uses, via native method calls, to talk directly to the VM. The latter is not publicly visible (and usually VM specific). I'm not confusing them. It's very clear to me. Now, there may be public methods such as Thread.start() that just happen to be implemented via native method calls directly into the VM, but those are degenerate cases. The application doesn't know that things happen to be implemented that way and that it might be making a direct native call into the VM. Right. By the way, Classpath avoids any such degenerate cases: all methods in class Foo that could be native instead are called through a liason class VMFoo. So all of the Classpath/VM API native methods are non-public. (This is why I said Classpath assumes the VM can do easy inlining, because of the Foo - VMFoo extra hops). Right. The API that applications use is defined by the public classes and methods in the class library. How those classes are implemented is not known to the application (and the application shouldn't care). Which ones actually involve calls into the VM is irrelevant. Right - and I'd argue that how the VM/classlib interface is implemented by the VM is not known to the class library and the class library shouldn't care. E.g., have you ever seen the Javadoc for a method contain the native keyword? No -- that keyword is stripped out because it's irrelevant from an API point of view (so is synchronized). Instead, those are implementation details. The application just sees a big Java class library. The class library sees a handful of native methods into the VM. The VM sees an operating system of some sort. Etc. Yes. I understand all this. And what I'm looking for is a way that we can define the VM-Classlib interface (where I use interface in the 'set of contracts and specifications' sense rather than the Java sense) that reflects everything known to date in the community on how to do this. For example, when people were porting their platforms to use GNU Classpath's API, what did they think? Did they have to redo anything internally? In my experience, things mostly made sense, but there were definitely some problems. However, they were there more because nobody had gotten around to fixing them, rather than some conflict of API design vision or anything like that. For example, the API call to load native libraries didn't have a ClassLoader parameter. But that is required to properly implement Java 2 style class loading (and unloading). So we added one. There was nothing controversial about it. Similarly for lots of other stuff. Sure In other words, there hasn't been much debate on the Classpath lists about APIs. It's pretty straightforward, and in for the most part the little details don't matter that much anyway. This is why many of us who work on Classpath are wondering why anyone would bother wondering how to reinvent the wheel, when the wheel already rolls just fine :-) :) Cheers, -Archie __ Archie Cobbs *CTO, Awarix* http:// www.awarix.com -- Geir Magnusson Jr +1-203-665-6437 [EMAIL PROTECTED]
Re: [arch] VM/Classlibrary Interface (take 2)
Geir Magnusson Jr. wrote: The API is private to the VM implementation, so the only effect it can have on application code is how efficient it is. The API isn't private to the VM implementation, is it? The _implementation_ of the API is, but not the API itself - that's a contract between the classlib and VM... You're confusing two things: the API that applications use (which is strictly a Java API you can read about via Javadoc) and the API that the class library uses, via native method calls, to talk directly to the VM. The latter is not publicly visible (and usually VM specific). I'm not confusing them. It's very clear to me. Sorry, I see now that I misspoke.. the API is private to the VM implementation should have been the API is private to the classlib/VM implementation, i.e., invisible to the application, so it can't directly effect the application, so it's a private issue to the JVM creators (where JVM includes the classlib), etc. The application just sees a big Java class library. The class library sees a handful of native methods into the VM. The VM sees an operating system of some sort. Etc. Yes. I understand all this. And what I'm looking for is a way that we can define the VM-Classlib interface (where I use interface in the 'set of contracts and specifications' sense rather than the Java sense) that reflects everything known to date in the community on how to do this. Right... and I'm still saying that although there may be different ways to organize the API, I'm not aware of anyone claiming that it makes very much difference in the end... would like to hear if anyone does. -Archie __ Archie Cobbs *CTO, Awarix* http://www.awarix.com
[arch] VM/Classlibrary Interface (take 2)
Ok, from the school of Storming the Gates! Take 2, lets again examine the question of VM/classlib interface as this is an important aspect to address and our first run at it wasn't so successful. The questions I have are all around the different ways has this been done. So far we know about GNU Classpath, but I assume there are others. What I'm specifically interested in is : - what were the architectural goals - what mistakes made in the past did you try to avoid - what are the known limitations - does the interface support our target version of 1.5 Anyone who has experience, please post it here, describing the particulars rather than just point someone to docs or code that may be under a license not acceptable by everyone. That said, please include any references too... geir -- Geir Magnusson Jr +1-203-665-6437 [EMAIL PROTECTED]
Re: [arch] VM/Classlibrary interface
Hi, I also agree, but I'm trying to find the time to read through all the ClassPath VM object interfaces and the stub objects to see if I can workout what is happening currently, especially what the package private methods are doing. I'm hoping this will make it clear why implementing the methods in separately packaged classes that provide a JVM interface implementation would not be possible, or preferable. I don't like changes in Java.Lang, although I know they should be invisible it just doesn't feel like a very clean interface design, overlaying extra functionality without extending the class or interface. This could, of cause, all be prejudice and hence having to now go off and judge what benefits it gives over a more clear separation. I should have a more clear idea after the weekend, or be so confused I'll stop suggesting change ;-} Apparently, only you and I agree. ;-) Dalibor Topic wrote: Richard S. Hall wrote: To me, this is the point. I would like to see all of the libraries built on to of the JVM to be packaged in a more module-like fashion, so that their exports and imports are explicit. There would be many benefits if this were done, rather than relying on the current approach of assuming that everything is accessible. +1 So, from my point of view, it is definitely going in the right direction to make libraries understand which class loader they should use to get to their own module's classes, as opposed to just assuming they can get them from any application class loader. +1 to that, too. Thanks Pete
RE: [arch] VM/Classlibrary interface
Nor do I disagree... I /love/ modularity too. RB Qui ne dit mot consent (who tells nothing agrees) -Original Message- From: Geir Magnusson Jr. [mailto:[EMAIL PROTECTED] Sent: Thursday, June 09, 2005 7:13 AM To: harmony-dev@incubator.apache.org Subject: Re: [arch] VM/Classlibrary interface Heh. I agree. I just was too busy in the VM/class library fire- fight :) So, given that my foray into architecture discussion was such a stunning success, would you like to start the discussion of such a thing might be approached? (Please change the subject, of course...) geir On Jun 8, 2005, at 3:01 PM, Richard S. Hall wrote: Apparently, only you and I agree. ;-) Dalibor Topic wrote: Richard S. Hall wrote: To me, this is the point. I would like to see all of the libraries built on to of the JVM to be packaged in a more module- like fashion, so that their exports and imports are explicit. There would be many benefits if this were done, rather than relying on the current approach of assuming that everything is accessible. +1 So, from my point of view, it is definitely going in the right direction to make libraries understand which class loader they should use to get to their own module's classes, as opposed to just assuming they can get them from any application class loader. +1 to that, too. cheers, dalibor topic -- Geir Magnusson Jr +1-203-665-6437 [EMAIL PROTECTED]
Re: [arch] VM/Classlibrary interface
Richard S. Hall wrote: To me, this is the point. I would like to see all of the libraries built on to of the JVM to be packaged in a more module-like fashion, so that their exports and imports are explicit. There would be many benefits if this were done, rather than relying on the current approach of assuming that everything is accessible. +1 So, from my point of view, it is definitely going in the right direction to make libraries understand which class loader they should use to get to their own module's classes, as opposed to just assuming they can get them from any application class loader. +1 to that, too. cheers, dalibor topic
Re: [arch] VM/Classlibrary interface
Apparently, only you and I agree. ;-) Dalibor Topic wrote: Richard S. Hall wrote: To me, this is the point. I would like to see all of the libraries built on to of the JVM to be packaged in a more module-like fashion, so that their exports and imports are explicit. There would be many benefits if this were done, rather than relying on the current approach of assuming that everything is accessible. +1 So, from my point of view, it is definitely going in the right direction to make libraries understand which class loader they should use to get to their own module's classes, as opposed to just assuming they can get them from any application class loader. +1 to that, too. cheers, dalibor topic
Re: [arch] VM/Classlibrary interface
On Jun 3, 2005, at 7:18 AM, Peter Edworthy wrote: Hello, And you can circumvent the language protection (package private...) if you work hard enough too, I believe... Keeping out of java.lang seems wise if we can arrange it... I agree, but ClassPath has its interface classes in Java.lang and while we can and probably should implement these as an adapter level to our internal system abstraction I can see why the JVM providing # java.lang.Class # java.lang.Runtime # java.lang.Thread # java.lang.reflect.Constructor # java.lang.reflect.Method # java.lang.reflect.Field makes sense, as these are completely dependant on information only available to the JVM. ClassPath could implement these objects and then have them call across to, say, jvm.implementation.* objects but this adds an extra level of indirection with no increase in flexibility of the implementation. Oops. I see that I wasn't clear. I didn't meant that the VM shouldn't implement things from java.lang - it makes perfect sense that we do so. What I was trying to say [sloppily] was that we shouldn't extend java.lang w/ new classes or interfaces. Sorry. The other java.lang.VM* objects I agree should really not be in Java.lang.* but having references to these use an explicit ClassLoader mitigates this into a cosmetic issue. Yes It seems to me that control of 'java.lang' and 'java.lang.*' would be the simplest way to deal with issues of control of the implementation so that it is a Harmony implementation, not one from Sun, IBM, FSF, Sourceforge.* or anyone else. From the Sun JDK 1.5.0 src.zip, there are 49 distinct 'sun.*' imports in the 'java.lang' package, none of which are found in this archive: I wasn't aware that we were trying to restrict the possible ClassLibs that could be used. In an ideal world we would want to make switching between ClassPath and Jikes's implementations easy, encouraging the user to use the most efficient implementation. (see next email for suggestions of mix and match implementations). yes, and it would be cool if we could concoct an adapter for Sun's implementation as well for fun :) Making the interfaces between VM and class library standard should be something we do now, starting with GNU Classpath's interface, and figuring out what else we might need based on experiences of others coughIBM/Intel/BEA/cough doing the same thing. ...snip... By writing _only_ java.lang and java.lang.*, we can truly speak of a separate implementation. Adding only _parts_ of libraries like GNU ClassPath would mean that users would implement Harmony library policies, not Sun's, FSF's, or anybody elses. This implemetation of java.lang and java.lang.* is only 165 classes in 53K lines of code, not including native code. Doable by this group without a doubt. Even better than that the ClassPath project has already done this. As I understand it, we are discussing methods of interfacing with the ClassPath project in such a way that; other implementations can be plugged in with minimal coding, changes to the Java API do not require changes to the JVM, only the ClassLibs. Well, two comments : 1) we clearly want to interface w/ GNU Classpath, but we want to generalize thought whatever we can learn from others. 2) I don't know for sure, but guess that as time goes on and the Java API evolves, it might require additions to the JVM/Classlib API geir Thanks, Peter -- Geir Magnusson Jr +1-203-665-6437 [EMAIL PROTECTED]
Re: [arch] VM/Classlibrary interface
On Jun 3, 2005, at 1:06 PM, Sven de Marothy wrote: Hello, And you can circumvent the language protection (package private...) if you work hard enough too, I believe... Keeping out of java.lang seems wise if we can arrange it... By writing _only_ java.lang and java.lang.*, we can truly speak of a separate implementation. Adding only _parts_ of libraries like GNU ClassPath would mean that users would implement Harmony library policies, not Sun's, FSF's, or anybody elses. Blatant Not-Invented-Hereism. This isn't about Control over the implementation, this was about language protection features. I won't comment on NIH (as I didn't quite grok what Dan was saying - I'm at 37k feet coming back on a redeye...) but we need to figure out if the language protection features are enough of a benefit to outweigh he risks of extending java.lang, and if the needs of 1.5 require so much more that the language protection is moot. The FSF doesn't have any library policy. What are you talking about? By writing _only_ java.lang and java.lang.*, we can truly speak of a separate implementation. Why do you need a seperate implementation? Separate from GNU Classpath or something else? geir /Sven -- Geir Magnusson Jr +1-203-665-6437 [EMAIL PROTECTED]
Re: [arch] VM/Classlibrary interface
Hello, And you can circumvent the language protection (package private...) if you work hard enough too, I believe... Keeping out of java.lang seems wise if we can arrange it... I agree, but ClassPath has its interface classes in Java.lang and while we can and probably should implement these as an adapter level to our internal system abstraction I can see why the JVM providing # java.lang.Class # java.lang.Runtime # java.lang.Thread # java.lang.reflect.Constructor # java.lang.reflect.Method # java.lang.reflect.Field makes sense, as these are completely dependant on information only available to the JVM. ClassPath could implement these objects and then have them call across to, say, jvm.implementation.* objects but this adds an extra level of indirection with no increase in flexibility of the implementation. The other java.lang.VM* objects I agree should really not be in Java.lang.* but having references to these use an explicit ClassLoader mitigates this into a cosmetic issue. It seems to me that control of 'java.lang' and 'java.lang.*' would be the simplest way to deal with issues of control of the implementation so that it is a Harmony implementation, not one from Sun, IBM, FSF, Sourceforge.* or anyone else. From the Sun JDK 1.5.0 src.zip, there are 49 distinct 'sun.*' imports in the 'java.lang' package, none of which are found in this archive: I wasn't aware that we were trying to restrict the possible ClassLibs that could be used. In an ideal world we would want to make switching between ClassPath and Jikes's implementations easy, encouraging the user to use the most efficient implementation. (see next email for suggestions of mix and match implementations). ...snip... By writing _only_ java.lang and java.lang.*, we can truly speak of a separate implementation. Adding only _parts_ of libraries like GNU ClassPath would mean that users would implement Harmony library policies, not Sun's, FSF's, or anybody elses. This implemetation of java.lang and java.lang.* is only 165 classes in 53K lines of code, not including native code. Doable by this group without a doubt. Even better than that the ClassPath project has already done this. As I understand it, we are discussing methods of interfacing with the ClassPath project in such a way that; other implementations can be plugged in with minimal coding, changes to the Java API do not require changes to the JVM, only the ClassLibs. Thanks, Peter
Re: [arch] VM/Classlibrary interface
Peter, Some good thinking here. In your _next_ e-mail you wrote, How clean are the separations between packages/groups of packages in ClassPath? i.e. how possible would it be to use say java.sql.* with another implementation of java.lang? I think we are thinking along similar lines. This is why I suggest doing 'java.lang' and descendents. Of all the packages in the entire corpus of Java, this family of packages is absolutely vital to the health and personality of the machine. In _this_ e-mail, you wrote, [Original Message] From: Peter Edworthy [EMAIL PROTECTED] To: harmony-dev@incubator.apache.org Date: 6/3/05 5:18:47 AM Subject: Re: [arch] VM/Classlibrary interface Hello, And you can circumvent the language protection (package private...) if you work hard enough too, I believe... Keeping out of java.lang seems wise if we can arrange it... I agree, but ClassPath has its interface classes in Java.lang and while we can and probably should implement these as an adapter level to our internal system abstraction I can see why the JVM providing # java.lang.Class # java.lang.Runtime # java.lang.Thread # java.lang.reflect.Constructor # java.lang.reflect.Method # java.lang.reflect.Field You might add to this list others the methods that use native implementations because you will have to implement them anyway, and probably as native methods because, depending on the JVM design, there may not be any other place they can be implemented. Here is the Sun JDK 1.5.0 native method list for java.lang.*: java.lang.reflect.Array java.lang.reflect.Proxy java.lang.Class java.lang.ClassLoader java.lang.Compiler java.lang.Double java.lang.Float java.lang.Object java.lang.Package java.lang.ProcessEnvironment java.lang.ProcessImpl java.lang.Runtime java.lang.SecurityManager java.lang.Shutdown java.lang.StrictMath java.lang.String java.lang.System java.lang.Thread java.lang.Throwable This list might vary from one implementation to another and from one release to another, but you get the idea. There appear to be a few differences from JDK 1.4.2 here. ...snip... Dan Lydick
Re: [arch] VM/Classlibrary interface
Hello, And you can circumvent the language protection (package private...) if you work hard enough too, I believe... Keeping out of java.lang seems wise if we can arrange it... By writing _only_ java.lang and java.lang.*, we can truly speak of a separate implementation. Adding only _parts_ of libraries like GNU ClassPath would mean that users would implement Harmony library policies, not Sun's, FSF's, or anybody elses. Blatant Not-Invented-Hereism. This isn't about Control over the implementation, this was about language protection features. The FSF doesn't have any library policy. What are you talking about? By writing _only_ java.lang and java.lang.*, we can truly speak of a separate implementation. Why do you need a seperate implementation? /Sven
Re: [arch] VM/Classlibrary interface
Peter, Dan, Sorry to jump into the middle of the conversation :-) On Fri, Jun 03, 2005 at 10:40:13AM -0500, Dan Lydick wrote: Peter, Some good thinking here. In your _next_ e-mail you wrote, How clean are the separations between packages/groups of packages in ClassPath? i.e. how possible would it be to use say java.sql.* with another implementation of java.lang? I'd like to say that I'm currently doing this. And not just with java.sql.*. In a (insert NDA-covered tape) project I'm on at the moment, the vendor has either removed or crippled a few core packages in their proprietary VM. So far, I have had to source the following packages from GNU Classpath to get 3rd party libs working (and to add missing functionality such as JDBC): java.sql.* java.util.logging.* javax.sql.* javax.transaction.* javax.transaction.xa.* Now, granted there are a lof of interfaces in there, but as things stand these replacements have behaved as 1:1 drop in replacements. (I know that the way Sun has done things, a lot of Java packages aren't really isolated; there is substantial amount of cross package dependency in the core Java API) Thanks, Steph -- Stephane Meslin-Weber Email: [EMAIL PROTECTED] Software Engineer Web: http://odonata.tangency.co.uk signature.asc Description: Digital signature
Re: [arch] VM/Classlibrary interface
On May 28, 2005, at 11:26 AM, Richard S. Hall wrote: Ulrich Kunitz wrote: On Fri, 27 May 2005, Geir Magnusson Jr. wrote: (Tomcat : I'd bet they fixed that (or will fix...)) Well, can't the VM just prevent non-kernel code from using them? Maybe overhead too high? You could play class loader games, however those could be circumvented by just another customized class loader. Doug Lea discussed the general issue in a message to this mailing list already. IMHO the problem is, that you can make a class only public, package-private or visible for a single class (e.g. private static). Some finer grained controls might be usefil like exporting a class to a particular package. Doug referenced the paper http://www.research.ibm.com/people/d/dgrove/papers/oopsla03.html that discuesses a number of the issues involved and proposes a solution based on a module concept. He referenced this page http://www.cs.utah.edu/flux too. You can definitely get these types of features from class loader games, but somehow that sounds like a pejorative description. I don't see why using class loaders for this purpose is not a good idea, it is possible to create a pretty decent module system just using class loaders and JAR files. I have just made available an alpha release of Oscar 2.0, my OSGi framework implementation, with some extensions to the R3 version of the specification: http://oscar.objectweb.org/oscar-alpha.html There are basic capabilities, like a having a JAR declare what it allows to be exported (and imported), but there are also more advanced capabilities, like being able to include/exclude certain classes from specific exported package. This latter capability makes it possible to get around some of the limitations of Java's access modifiers as they pertain to packages. In these situations, the classes in the JAR file itself have complete access to their contents, but external classes do not. This is all fairly easy and just requires that the classes be packaged into JAR files with some metadata. As far as circumventing these capabilities using another customized class loader, yes, they could create a URLClassLoader directly to a module JAR file, which would ignore the metadata completely and then they might be able to load classes directly from it. However, this is going to a bit of effort to get at implementation details... And you can circumvent the language protection (package private...) if you work hard enough too, I believe... Keeping out of java.lang seems wise if we can arrange it... geir - richard -- Geir Magnusson Jr +1-203-665-6437 [EMAIL PROTECTED]
Re: [arch] VM/Classlibrary interface
On May 29, 2005, at 6:35 PM, Peter Edworthy wrote: Sadly I've left my copy of the Java 2 security model book at work, but basically my understanding is ClassLoaders restricting access is the correct solution. It provides the interface independence we are looking for, i.e. changes to the VM interface and changes to the Java(x).* interface are entirely independent. Yes it can be circumvented, but only with access to the security settings and then anything could be done. This does require support from 'ClassPath' type projects though, but any solution requires an agreement of the interface between the JVM and the java(x).* class libraries. Yes - that would be one of the goals here at Apache Harmony - to drive to consensus on what that interface should indeed be, and all work to make that so. The implementation would be whenever a 'VM implementation' class is used in 'ClassPath' or similar projects the 'VM implementation ClassLoader' would have to be used. When a Java(x).* class was used the default loader would be used, this explicitly seperates the namespace for JVM classes and java(x).* classes. The obvious way to get this loader would be through Object.getClassLoader() and ensuring the 'VM implementation ClassLoader' was on the tree below the java(x).* ClassLoader. I'm open to suggestions of how best to mark the ClassLoader as the 'VM implementation ClassLoader' as of cause an interface marking it would have to be at the ClassPath( java(x).*) level. I don't like relying on it being the bootstrap ClassLoader as this may be being used in a separate ClassLoader space within another JVM. Not that I want to punt the question, but it is an implementation question for the VM, right? I mean, we can decide on the interface, declare that our VMs and class library implementations will support said interface, and then the VM impls can innovate on how they deal with it. Clearly they deal with it in a few ways - Classpath has their scheme of language protection via package private, but there must be other solution from other VMs. geir -- Geir Magnusson Jr +1-203-665-6437 [EMAIL PROTECTED]
Re: [arch] VM/Classlibrary interface
Rodrigo Kumpera wrote: There is a long track of incompatibilities between JVMs and most of then are subtle, poor specified behavior. A good example is java.util.ConcurrentModificationException, this exception is thrown in a best-effort way by the Collection classes. This means that JVMs will throw this exception as they see fit. I've seen software that did not depent on this behavior fail because of such problems. Developers are not supposed to depend on that exception for correctness of their code. It's just a tool to diagnose deeper problems. See API specification for details. That being said, of course there are differences in behaviour between different VMs. That is understandable as they use different ways to do things internally, and perfectly fine as long as such behaviour is within the scope of the specifications. If their behaviour is outside the scope of the specifications, than it's a bug in the VM and needs to be fixed. There may be cases where the specifications are not that great. In such cases, the proper way to fix the problem is to fix the specifications. Or to use third-party software with equivalent features, which can be relied upon at deployment, rather than hoping that some unspecified behaviour is present on all versions of runtimes one wishes to deploy on. Other good example is the bugtracker for Sun's JVM, there are many bugs that differ from specified behavior and there might be software out there that depends on these bugs but never quite realized that. Then it will be fixed when someone realizes that it is broken. That happens all the time when any environment is upgraded, be it free or non-free: buggy software breaks eventually, developers find out about the bugs, it gets fixed, and life goes on. Broken software is a reality and sometimes the cause is the blur line between the spec and the RI. You can even find some public methods somewhere on AWT that takes sun.* classes. In that case the specifications need to be fixed. Could you please file a bug at the spec bug tracker? I am sure that an independant effort like this one can help spot and prevent such mistakes to happen in the future, and that can only be good for the quality of the platform. cheers, dalibor topic
Re: [arch] VM/Classlibrary interface
You could play class loader games, however those could be circumvented by just another customized class loader. Doug Lea discussed the general issue in a message to this mailing list already. IMHO the problem is, that you can make a class only public, package-private or visible for a single class (e.g. private static). Some finer grained controls might be usefil like exporting a class to a particular package. Doug referenced the paper http://www.research.ibm.com/people/d/dgrove/papers/oopsla03.html that discuesses a number of the issues involved and proposes a solution based on a module concept. He referenced this page http://www.cs.utah.edu/flux too. You can definitely get these types of features from class loader games, but somehow that sounds like a pejorative description. I don't see why using class loaders for this purpose is not a good idea, it is possible to create a pretty decent module system just using class loaders and JAR files. I agree. Sadly I've left my copy of the Java 2 security model book at work, but basically my understanding is ClassLoaders restricting access is the correct solution. It provides the interface independence we are looking for, i.e. changes to the VM interface and changes to the Java(x).* interface are entirely independent. Yes it can be circumvented, but only with access to the security settings and then anything could be done. This does require support from 'ClassPath' type projects though, but any solution requires an agreement of the interface between the JVM and the java(x).* class libraries. The implementation would be whenever a 'VM implementation' class is used in 'ClassPath' or similar projects the 'VM implementation ClassLoader' would have to be used. When a Java(x).* class was used the default loader would be used, this explicitly seperates the namespace for JVM classes and java(x).* classes. The obvious way to get this loader would be through Object.getClassLoader() and ensuring the 'VM implementation ClassLoader' was on the tree below the java(x).* ClassLoader. I'm open to suggestions of how best to mark the ClassLoader as the 'VM implementation ClassLoader' as of cause an interface marking it would have to be at the ClassPath( java(x).*) level. I don't like relying on it being the bootstrap ClassLoader as this may be being used in a separate ClassLoader space within another JVM. I hope that all make sense, Pete
Re: [arch] VM/Classlibrary interface
Rodrigo Kumpera wrote: Last time I checked, no one, nether me or you, is developing code agains the TCK, but to a real JVM. And as hard as we may try, sometimes we end with software that depends on unspecified behavior. So it's better try to be bug compatible too. If you end up with software that depends on unspecified behaviour, then it is either a) your deliberate decision, then you probably have a very, very good reason to tie yourself to the particular revision of the particular platform, or b) an accidental mistake, then you fix the small bug in your code, feel better about the quality of your code, and move on. Neither a nor b requires anyone to be bug compatible as a) is not a bug in anyone's code, and b) is something you'll want to fix in your code, if you care about the quality of your code, rather then working around your bugs in everyone's class library code, and breaking other people's applications. Regarding usage of implementation specific classes, I believe that most developers using the Java programming language are familar with the warnings not to use implementation-specific classes or to rely on their behaviour, names, presence or anything esle. cheers, dalibor topic
Re: [arch] VM/Classlibrary interface
Rodrigo Kumpera wrote: Last time I checked, no one, nether me or you, is developing code agains the TCK, but to a real JVM. And as hard as we may try, sometimes we end with software that depends on unspecified behavior. So it's better try to be bug compatible too. No, I don't agree on this either. Dalibor already mentioned several good reasons why Harmony should not try to be implementation compatible with any other VM and another good reason is that the usage of com.sun-classes is also version or release dependent of Sun's VM. If Sun decides to rename, repackage or somehow change the internal classes in a new VM release, e.g. 1.5.0_04, code relying on these classes will break as well. The implementors of such code should fix their part and no other VM vendor seem to find a reason to implement their VMs in such a fashion that broken code will run on them. Tor
Re: [arch] VM/Classlibrary interface
Tor-Einar Jarnbjo wrote: Rodrigo Kumpera wrote: Last time I checked, no one, nether me or you, is developing code agains the TCK, but to a real JVM. And as hard as we may try, sometimes we end with software that depends on unspecified behavior. So it's better try to be bug compatible too. No, I don't agree on this either. Dalibor already mentioned several good reasons why Harmony should not try to be implementation compatible with any other VM and another good reason is that the usage of com.sun-classes is also version or release dependent of Sun's VM. Well, it needs to be compatible and pass the official test suites. That's possible and will happen. Being compatible with something for which no specification or documentation exists is impossible, though, so it can't happen. And in practice, it is not necessary, either. Evolution sorts out portable from unportable code over time. I have not seen an actively maintained application using the com.sun.java.swing or java.awt.swing migrational, runtime-specific API[1] in the last 3 years at least. ;) cheers, dalibor topic [1] http://java.sun.com/products/jfc/package.html
Re: [arch] VM/Classlibrary interface
Ulrich Kunitz wrote: On Fri, 27 May 2005, Geir Magnusson Jr. wrote: (Tomcat : I'd bet they fixed that (or will fix...)) Well, can't the VM just prevent non-kernel code from using them? Maybe overhead too high? You could play class loader games, however those could be circumvented by just another customized class loader. Doug Lea discussed the general issue in a message to this mailing list already. IMHO the problem is, that you can make a class only public, package-private or visible for a single class (e.g. private static). Some finer grained controls might be usefil like exporting a class to a particular package. Doug referenced the paper http://www.research.ibm.com/people/d/dgrove/papers/oopsla03.html that discuesses a number of the issues involved and proposes a solution based on a module concept. He referenced this page http://www.cs.utah.edu/flux too. You can definitely get these types of features from class loader games, but somehow that sounds like a pejorative description. I don't see why using class loaders for this purpose is not a good idea, it is possible to create a pretty decent module system just using class loaders and JAR files. I have just made available an alpha release of Oscar 2.0, my OSGi framework implementation, with some extensions to the R3 version of the specification: http://oscar.objectweb.org/oscar-alpha.html There are basic capabilities, like a having a JAR declare what it allows to be exported (and imported), but there are also more advanced capabilities, like being able to include/exclude certain classes from specific exported package. This latter capability makes it possible to get around some of the limitations of Java's access modifiers as they pertain to packages. In these situations, the classes in the JAR file itself have complete access to their contents, but external classes do not. This is all fairly easy and just requires that the classes be packaged into JAR files with some metadata. As far as circumventing these capabilities using another customized class loader, yes, they could create a URLClassLoader directly to a module JAR file, which would ignore the metadata completely and then they might be able to load classes directly from it. However, this is going to a bit of effort to get at implementation details... - richard
Re: [arch] VM/Classlibrary interface
On 5/28/05, Dalibor Topic [EMAIL PROTECTED] wrote: Rodrigo Kumpera wrote: Last time I checked, no one, nether me or you, is developing code agains the TCK, but to a real JVM. And as hard as we may try, sometimes we end with software that depends on unspecified behavior. So it's better try to be bug compatible too. If you end up with software that depends on unspecified behaviour, then it is either a) your deliberate decision, then you probably have a very, very good reason to tie yourself to the particular revision of the particular platform, or b) an accidental mistake, then you fix the small bug in your code, feel better about the quality of your code, and move on. I agree with you about the first one, but the second is where the fine line between pragmatic and retoric solutions line. It's easy to say 'just fix it then', but I hope that Harmony gets more users that a few hackers. The TCK is not the silver bullet for compatibility, A software I wrote for 1.4.0 suddenly got broken on 1.4.1 because of, I don't know, bug fixes or subtle changes on behavior of java.nio. My point is, testing against just the TCK is just not enouth. Testing against real applications is where the real value of Harmony can be asserted. Most free JVMs already do that and nobody seens to be complaining. Rodrigo
[arch] VM/Classlibrary interface
I think that this is a good topic that many around here have experience with and can get involved in. I think that it's not breaking new ground to say that we'll need a standard interface between the VM and the classlibrary, to let people integrate the various VMs and class libraries that we'll probably see. The GNU Classpath project has documented what they need http://www.gnu.org/software/classpath/docs/vmintegration.html and this seems to break down to : 1) Some sort of initialization protocol (currently seems to be defined in code, and only up to v1.1 - that may be stale docs...) 2) Core classes implemented by the VM for class-library usage - standard - things that you expect in java.lang (java.lang.Object) - non-standard - extension to java.lang (java.lang.VMObject) 3) VM hooks into the classlib 4) RMI stuff Clearly this is one model, and one that works because it's been used by many integrations. Now, questions : 0) Standing back from this specific model, can the model be generalized? 1) Are there other models? How do some of the commercial VMs do it? 2) Are there things that the GNU Classpath model is missing due to the version of the API it's implementing? (I.e. they don't realize they need it yet...) 3) I was uncomfortable with extending java.lang. I understand the argument - that as they are package private, the language can be depended upon to keep them safe from user code using them rather than some security infrastructure. However, isn't this a bit dangerous in terms of standard java.lang changes colliding? I'd like to drive to a standard interface that we can all agree on, and hopefully GNU Classpath will support it. This would insulate us from the ongoing legal discussion surrounding the license, and let us get on with using it directly. geir -- Geir Magnusson Jr +1-203-665-6437 [EMAIL PROTECTED]
Re: [arch] VM/Classlibrary interface
Geir == Geir Magnusson [EMAIL PROTECTED] writes: Geir 4) RMI stuff On irc Geir said that he meant JNI here. Basically what this means, I think, is that Classpath has an implementation of jni.h, but the VM has to provide certain typedefs, e.g., jint. Geir 2) Are there things that the GNU Classpath model is missing due to Geirthe version of the API it's implementing? (I.e. they don't realize Geirthey need it yet...) Yeah, there are some needed additions for 1.5. E.g., annotations require new reflection info, and in the Classpath model this comes from the VM layer. This will all show up eventually, perhaps on the generics branch. Geir 3) I was uncomfortable with extending java.lang. I understand the Geirargument - that as they are package private, the language can be Geirdepended upon to keep them safe from user code using them rather Geirthan some security infrastructure. However, isn't this a bit Geirdangerous in terms of standard java.lang changes colliding? I don't think there is a killer failure mode. First, note that it is both the language and the VM that implement this protection. This is crucial, as otherwise someone could easily circumvent the protection at runtime. Second, suppose Sun adds java.lang.VMObject to their public API. What then? * A program written against Sun's new release that uses this feature will not work against our old release. But that is already the case -- a program using a new API won't work against an older Classpath that doesn't implement it. The choice of name doesn't matter. * A program written against Sun's new release that uses this feature also won't compile against our old release. But this won't work anyway either. * To implement this new feature we would have to rename our VMObject and thus change the VM interface. But that is also not a huge problem. The VM interface can't remain fixed in general, because sometimes new features (see the 1.5 example) require additions to it. (This does mean some churn for VM implementors, and it is inconvenient. But it only breaks the VM, not anything else.) * Finally, there's no way to compile a program against Classpath that uses the Classpath VMObject, due to the access protection. So failures the other way aren't possible. Geir I'd like to drive to a standard interface that we can all agree on, Geir and hopefully GNU Classpath will support it. Experience has shown that the Classpath approach is pretty flexible. A wide range of VMs already use it. Also we're pretty open to specific needed changes. Tom
Re: [arch] VM/Classlibrary interface
Geir Magnusson Jr. wrote: 2) Core classes implemented by the VM for class-library usage - standard - things that you expect in java.lang (java.lang.Object) - non-standard - extension to java.lang (java.lang.VMObject) 3) I was uncomfortable with extending java.lang. I understand the argument - that as they are package private, the language can be depended upon to keep them safe from user code using them rather than some security infrastructure. However, isn't this a bit dangerous in terms of standard java.lang changes colliding? In general a VMInterface class resides as a package private class in the package that the class they help implement resides in. Using package private, not-user-visible classes seems to be pefectly valid way to do get similar things done in other, officially certified as compatible implementations[1], so I don't think that having package private classes means someone is extending the package's namespace, given that the VMInterface classes are not exported to user code to compile against. Therefore I think the term 'extension' is misplaced in this context. The VMInterface classes are an internal (and very useful in practice) detail of this particular class library implementation that does not extend the namespace available to user code to link against. Extending the name spaces would not be clever for all the obvious binary compatibility reasons, after all. The JCP may eventually specify classes in the future with the same names. That could happen, and would be just fine. The JCP is obviously free to call the specified classes any way they want, and if for some reason the JCP decides to call some future 1.6 Object class VMObject, so be it. :) The internal GNU Classpath VM interface for Object would have to change at that point, and things would move on just as they do all the time someone changes the VM interface for a different reason. It currently changes rather frequently in some areas, as people figure out new ways how to implement the respective classes in a better fashion. cheers, dalibor topic [1] http://forum.java.sun.com/thread.jspa?threadID=591979messageID=3091404 see java.io.UnixFileSystem class in the stack trace.
Re: [arch] VM/Classlibrary interface
Geir Magnusson Jr. wrote: (Tomcat : I'd bet they fixed that (or will fix...)) It doesn't seem so. The SSL code has been in Tomcat versions 4.1.x to 5.5.9 and I just saw that also the LDAP code in Tomcat 5.5.9 uses classes in com.sun. Well, can't the VM just prevent non-kernel code from using them? Maybe overhead too high? How do you distinguish kernel code and non-kernel code? From the VM point of view, the classes in the java(x).* packages do not differ from user code classes and it is also possible to bootstrap the VM and replace java(x).* classes with own implementations. Been there, done that. Tor
Re: [arch] VM/Classlibrary interface
On May 27, 2005, at 6:25 PM, Tor-Einar Jarnbjo wrote: Geir Magnusson Jr. wrote: (Tomcat : I'd bet they fixed that (or will fix...)) It doesn't seem so. The SSL code has been in Tomcat versions 4.1.x to 5.5.9 and I just saw that also the LDAP code in Tomcat 5.5.9 uses classes in com.sun. Amazing. Jetty was doing the same thing until we asked them to fix it. They did, and all is well now. Well, can't the VM just prevent non-kernel code from using them? Maybe overhead too high? How do you distinguish kernel code and non-kernel code? From the VM point of view, the classes in the java(x).* packages do not differ from user code classes and it is also possible to bootstrap the VM and replace java(x).* classes with own implementations. Been there, done that. I meant execution context. Is there a clear boundary between code thats executing in the context of the VM and code that's executing in the context of the 'user' app? geir -- Geir Magnusson Jr +1-203-665-6437 [EMAIL PROTECTED]
Re: [arch] VM/Classlibrary interface
On Fri, 27 May 2005, Geir Magnusson Jr. wrote:
(Tomcat : I'd bet they fixed that (or will fix...))
Well, can't the VM just prevent non-kernel code from using them? Maybe
overhead too high?
You could play class loader games, however those could be
circumvented by just another customized class loader.
Doug Lea discussed the general issue in a message to this mailing
list already. IMHO the problem is, that you can make a class only
public, package-private or visible for a single class (e.g.
private static). Some finer grained controls might be usefil like
exporting a class to a particular package.
Doug referenced the paper
http://www.research.ibm.com/people/d/dgrove/papers/oopsla03.html
that discuesses a number of the issues involved and proposes a
solution based on a module concept. He referenced this page
http://www.cs.utah.edu/flux too.
He mentioned also that such features are planned for Java version 7.
Another approach right now might be to use metadata in the following way:
@Export(java.lang.*) public class VMObject {...
The bytecode linker could interpret the @Export annotation as an
override of public during resolution and prevent classes outside
of java.lang.* from linking VMObject by throwing a subclass
instance of IllegalAccessError. The compiler used should also be aware
of the @Export annotation.
If the @Export annotation is not supported by the VM or the
compiler a public class can still be linked from outside the
package, a package-private class will not. This way the author of
the class has even control over the failure mode.
Assuming that Harmony will support annotations, it shouldn't be to
complex to implement the feature, which doesn't require some
hard-coded magic or some new kind of configuration files.
Uli
Re: [arch] VM/Classlibrary interface
Geir Magnusson Jr. wrote: I meant execution context. Is there a clear boundary between code thats executing in the context of the VM and code that's executing in the context of the 'user' app? Usually not, but it might be possible to emulate something similar using several classloaders or implement the necessary mechanisms in the default classloader. One similar example can be taken from Java applets. They are of course not allowed to load and execute native code, as the VM can't enforce any privilege checks on what the native code is doing, still though, the applet must of course be allowed to indirectly execute native code through the standard API, e.g. to gain allowed network access. In this case, the applet classes are loaded by a restricted classloader, which does not allow direct access to native code, while the standard API classes are loaded by a privileged classloader, which grants all privileges on kernel level and relies on the implementation of the classes to enforce the necessary security checks. All this magic is however implemented using the mechanisms in the security manager and since an application must be allowed to use its own security manager, I don't see how it could be possible to prevent an application to break through such a protection either. Tor
Re: [arch] VM/Classlibrary interface
On 5/27/05, Geir Magnusson Jr. [EMAIL PROTECTED] wrote: On May 27, 2005, at 4:34 PM, Rodrigo Kumpera wrote: We should provide wrappers to classes for the sake of compatibility. Are there any legal problems with doing so? Why would we do that? We don't want to encourage such misbehavior...? geir There are two kinds of compatibility: by specification, the TCK buy us that; and pragmatic, and this is having software just work. Look around for all FOSS that are made to be an alternative to proprietary software, they all have kludges to be implementation compatible to the non-free counterpart. Last time I checked, no one, nether me or you, is developing code agains the TCK, but to a real JVM. And as hard as we may try, sometimes we end with software that depends on unspecified behavior. So it's better try to be bug compatible too.
