Native layer
Hi, I have a working native layer branch on jamvm (though I have not really counted if there was a regression I know the one from kaffe and the important ones in mauve are passing on standard linux). I will propose before the end of the week a patch against HEAD for a merge. I suggest people to take a look at the state of the NATIVE-LAYER branch and tell me if they see huge trouble with it. There are still a lot to do but the current state get rid of the target layer. I would like to remove the other remaining direct syscalls from the rest of the JNI code for clarity. It's a pain to merge HEAD to NATIVE-LAYER by the way (too many patches ;) )... Cheers, Guilhem.
Target native layer
h are not used (dead-code). Thus if at least one function is needed from a object file, all other function from that object file are linked to the application, too. There is only one automatic way to remove dead-code (-ffunction-sections), but this have other disadvantages. Even the man-page do not recommend it. Thus to remove dead-code of a function some #ifdef-endif around a function is needed. On the other side: A non-used macro will not produce any dead code. Some personal view: By the way: I like "long" names and I hate uncommon abbreviations e. g. like "fnctn" instead "function". I also like some prefix which indicate the location of, e. g. "file_open()" instead "open()". I usually have no problems with long names if the naming is consistent and useful. I also have no problems with lines longer then 80 characters, because my editor does not have some "optimal" line length. These are my thoughts to this topic. I hope all developers who are interrested in some target native layer will reconsider the current discussion. And I hope we will find a solution which can satisfy everybody. Sincerely, Torsten ___ Classpath mailing list [EMAIL PROTECTED] http://lists.gnu.org/mailman/listinfo/classpath
New native layer
Hi classpath ! I would like to mention that I am developping/fine tuning the new native layer for classpath in a separate branch called "NATIVE-LAYER". If you have some time to give your impressions then fetch it and look into native/jni/native-lib, native/jni/java-io, ... the new layer itself is developped in native-lib. The other libraries are being adjusted currently... Regards, Guilhem.
Re: Native layer
Hi Guilhem, On Tue, 2006-06-06 at 22:34 +0200, Guilhem Lavaux wrote: > I have a working native layer branch on jamvm (though I have not really > counted if there was a regression I know the one from kaffe and the > important ones in mauve are passing on standard linux). I will propose > before the end of the week a patch against HEAD for a merge. I suggest > people to take a look at the state of the NATIVE-LAYER branch and tell > me if they see huge trouble with it. There are still a lot to do but the > current state get rid of the target layer. I would like to remove the > other remaining direct syscalls from the rest of the JNI code for clarity. Yeah, it is probably a good idea to merge it back now otherwise you will keep having to merge things back and forth. I haven't reviewed the current branch code yet, but I did follow your patches. It is essential that if this is merged there are no mauve regressions for runtimes that use the classpath JNI code. If you need a help verifying that please do post a patch NATIVE-LAYER -> trunk somewhere so others can help you test that. > It's a pain to merge HEAD to NATIVE-LAYER by the way (too many patches > ;) )... Sorry about that. Lets ask everybody to stop working so hard :) It is indeed kind of amazing how many patches are written each day now. I have trouble keeping on top of it all myself at times. Cheers, Mark -- Escape the Java Trap with GNU Classpath! http://www.gnu.org/philosophy/java-trap.html Join the community at http://planet.classpath.org/ signature.asc Description: This is a digitally signed message part
Native layer, part II
Dear Classpath members, I yust check-in the second part of the native layer in: native/jni/nio native/jni/java-net native/jni/java-util Now most of the modifications are done now, but some fine tuning and of course bug-fixes (if there are bugs) still have to be done. I will do this as soon as possible. If you detect some bug or if there is something strange please send me a email. Sincerely, Torsten ___ Classpath mailing list [EMAIL PROTECTED] http://mail.gnu.org/mailman/listinfo/classpath
Re: Target native layer
se.h are > needed for checking chsize() (truncate) or mkdir(). For embOS even some > header-files can not be included, because they are wrong (but cannot be > changed/fixed by aicas). These things make autoconf very complicate to > use, because if there are a lot of possible functions which can be used > to implement some feature, it is not clear which function is detected by > autoconf for some specific system. That is handled in libgcj by having a hybrid system. autoconf for normal Posix like systems and an os-layer for systems that are completely different. > There even could be very bad > side-effects if more than one function is available (e. g. f1() and > f2()) and at some time f2() is used instead f1() (with different > behaviors or limitations), if there is some change for another target > system (e. g. you add some changes for RTEMS, but this will also have > effects on e. g. embOS. You will not detect this problem until testing > again all targets for any change in autoconf). It is a little bit > "undeterministic" which features are detected and if they are usable. But on the other hand that is autoconf its strength. You put in tests for features of different systems. If a platform has those features you get your port for free. > - some features are not detectable by autoconf, e. g. the ordering of > parameters for functions like inb() and outb() (we had that problem) or > additional parameters (which usually only produce a warning which is > discarded), e. g. gethostbyname_r() under Solaris. Have you looked at The GNU Autoconf Macro Archive? http://www.gnu.org/software/ac-archive/ It has a macro specifically for this gethostbyname_r() parameter detection for example. And there is also the ac-archive on sf. http://ac-archive.sourceforge.net/ > There are much more difficult things which occur with autoconf. To > replace some target layer (e. g. TARGET_*) by autoconf only will imho > make implementations for non-Unix-like systems very difficult and will > only shift the so called "complex" C-macro-implementation into "complex" > autoconf-macros-implementations (imho M4 is not much better then a > C-preprocessor and difficult to debug). But using autoconf has the advantage that "everybody" else is using it. At least the two most important projects to GNU Classpath gcj and kaffe are build around it. So solving something with autoconf helps everybody. Having our own target porting layer will make it more difficult to work together with others that do use autoconf like systems. And autoconf is very well documented out own layer isn't really well documented and actually has no examples how to use it properly. > 4. Multiple code - some statistics: > > In the current implementation we use at aicas we have the following > systems. The numbers below count the number of macros at all (functions > and constants) which are different from the standard (generic) > implementation: > > generic macros: 220 > > Linux: 0 > Solaris: 11 > RTEMS: 3 > MinGW: 46 > embOS: 10 (only partially implemented) I think this sums up pretty well why most people don't really see the point of the target layer. We only have the "empty implementation" for GNU/Linux systems. It would be interesting to compare the MinGW implementation with the way libgcj does it (mostly also through a os-layer). Also you say "the current implementation we use at aicas". To really consider the strength of the target-layer we must make it our implementation (for GNU Classpath as group). Otherwise it is only this abstraction that we don't really use. > Efficient code: wrapper-functions are nice, but in some cases an > overkill, e. g. when calling simple function like sin(). In general C > compilers do not optimize this (imho that is some reason why "inline" > was introduced). If "inline" can be used, macros are almost not needed > anymore. autoconf comes with AC_C_INLINE which handles this case. > dead-code: the standard GNU linker does not remove functions which are > not used (dead-code). Thus if at least one function is needed from a > object file, all other function from that object file are linked to the > application, too. There is only one automatic way to remove dead-code > (-ffunction-sections), but this have other disadvantages. Even the > man-page do not recommend it. Thus to remove dead-code of a function > some #ifdef-endif around a function is needed. On the other side: A > non-used macro will not produce any dead code. If this matters wouldn't you just put these functions that might not be used into their own source file so you have multiple object files? > These are my thoughts to this topic. I hope all developers who are > interrested in some target n
Re: Target native layer
Hi all, may I once more comment on that. > Also you say "the current implementation we use at aicas". To really > consider the strength of the target-layer we must make it our > implementation (for GNU Classpath as group). Otherwise it is only this > abstraction that we don't really use. This is a real argument, IMO. I can see some ways to solve the problem: 1. We get rid of the TARGET_* layer (because we don't use it), and the folks at AICAS could keep it on their own (much like some specialities in kaffe, gcj and sablevm are handled). Of course, this would make synchronizing their sources with the sources of GNU CP more difficult. 2. AICAS could contribute all their TARGET_* layer using code to GNU CP. This way this layer would kinde make sense to GNU CP hackers. Maybe we all can help to get rid of it nevertheless, but keep CP portable on those rare systems. 3. I dunno about the details, but we could go with a hybrid system: Most of the portable stuff goes into the generic layer and is autoconfiscated, like it was discussed to great lengths now. In really extreme cases the TARGET_* layer could be used. BUT: We should think about if we need all of the indirections in this layer and how we could make it more readable and debuggable. This means: - get rid of (multiline) macros where feasible, - shorten macro names, - group files together more intelligently and - reduce (better: completely avoid) redundant code (bugs multiplication!!). If this can be fullfilled somehow, then I could go with the TARGET layer. just my 2 cents again, /Roman ___ Classpath mailing list [EMAIL PROTECTED] http://lists.gnu.org/mailman/listinfo/classpath
Re: Target native layer
Dear Roman, 1. We get rid of the TARGET_* layer (because we don't use it), and the folks at AICAS could keep it on their own (much like some specialities in kaffe, gcj and sablevm are handled). Of course, this would make synchronizing their sources with the sources of GNU CP more difficult. I assume if TARGET_* is removed, aicas have to do their own development of the native code. This would be possible, but of course this would not be very useful. For aicas it would be difficult, because we are almost "kicked-out" of the project (at least for the native code). 2. AICAS could contribute all their TARGET_* layer using code to GNU CP. This way this layer would kinde make sense to GNU CP hackers. Maybe we all can help to get rid of it nevertheless, but keep CP portable on those rare systems. You are right, we are late with contributing more layers. Because of limited manpower we still did not contributed some more layers, e.g . Solaris. We like to do this. 3. I dunno about the details, but we could go with a hybrid system: Most of the portable stuff goes into the generic layer and is autoconfiscated, like it was discussed to great lengths now. The generic target is already doing this (at least partially). It is designed to use autoconf results to select the right OS-function or the right constants. Thus most of the code is located there. And that is also the reason why Linux seems to be "empty". It is not empty, it includes the generic cases of the target layer, thus there is not special implementation in Linux (please have a look at the final "include"-statement at the end of the files). If needed we can rename "generic" into "Linux" and discard the current "Linux". I do not want to do this, because this would look like the layer is limited to "Linux-like" systems. In really extreme cases the TARGET_* layer could be used. The extrem cases are the problem. I spend a lot of time to handle such cases and it is difficult to do this with autoconf only. If you only use Unix-like systems, probably you will not run into this problem, thus autoconf only is nice. But imho soon you like to use other systems, too, e. g. Windows or embedded systems it becomes a challange. I can imagine it is difficult for someone to understand my viewpoint, if that person is not developing software for embedded systems. - get rid of (multiline) macros where feasible, Of course it would be possible to remove the "linefeeds", but that would not help (imho it would make it even more difficult to read). I have the following suggestion: Because the multiline-macros are to difficult to handle, we can replace that macros e. g. by direct function-calls. Thus a macro would become some "alias" only. We also can change (shorten) the names. For this we have to change the generic implementation. The advantages would be: - debugging is possible - type-safety - easier to read and for aicas - we can keep most of the code we already have written - we can continue to participate in Classpath For aicas this would be some compromise, because we can still use the target-layer. What would be still important is, that in the native _no_ OS-specific feature is used directly, like constants or datatypes or OS-functions calls. With such a implementation we can survive. What do you think? - shorten macro names, Please make some suggestion. If TARGET_NATIVE is to long, what about TARGET or NATIVE or NC or ...? Please take care that there is some risk for naming-collisions if the name becomes to short (thus "OS" would probably be a bad choice). - group files together more intelligently and I do not understand what is wrong with the current grouping. The grouping is done by function-types, e. g. file, network, memory and so on. How to make it better/more clear? - reduce (better: completely avoid) redundant code (bugs multiplication!!). Ok. There are some parts of the layer which could be improved. If we decide to keep the target layer in the code, then we should do these improvements. Sincerely, Torsten ___ Classpath mailing list [EMAIL PROTECTED] http://lists.gnu.org/mailman/listinfo/classpath
Re: Target native layer
Am Mittwoch, 11. August 2004 12:37 schrieb Dr. Torsten Rupp: > Dear Roman, > > > 1. We get rid of the TARGET_* layer (because we don't use it), > > and the folks at AICAS could keep it on their own (much like some > > specialities in kaffe, gcj and sablevm are handled). Of course, > > this would make synchronizing their sources with the sources of > > GNU CP more difficult. > > I assume if TARGET_* is removed, aicas have to do their own > development of the native code. This would be possible, but of > course this would not be very useful. For aicas it would be > difficult, because we are almost "kicked-out" of the project (at > least for the native code). I don't wanna kick somebody out of the project. I just want to improve our Free Software project. I hope you dont got me wrong from the start. > > 2. AICAS could contribute all their TARGET_* layer using code to > > GNU CP. This way this layer would kinde make sense to GNU CP > > hackers. Maybe we all can help to get rid of it nevertheless, but > > keep CP portable on those rare systems. > > You are right, we are late with contributing more layers. Because > of limited manpower we still did not contributed some more layers, > e.g . Solaris. We like to do this. > > > 3. I dunno about the details, but we could go with a hybrid > > system: Most of the portable stuff goes into the generic layer > > and is autoconfiscated, like it was discussed to great lengths > > now. > > The generic target is already doing this (at least partially). It > is designed to use autoconf results to select the right OS-function > or the right constants. Thus most of the code is located there. And > that is also the reason why Linux seems to be "empty". It is not > empty, it includes the generic cases of the target layer, thus > there is not special implementation in Linux (please have a look at > the final "include"-statement at the end of the files). If needed > we can rename "generic" into "Linux" and discard the current > "Linux". I do not want to do this, because this would look like the > layer is limited to "Linux-like" systems. Wouldnt it be better to call it "posix" (note: no captial letter) as it currently supports Linux, *BSD, AIX ... We can and should add Solaris too. With autoconf its really simple. Then AICAS can remove their own Solaris port totally and we have a clean non-redundant codebase for posix-like OSes. > > In really > > extreme cases the TARGET_* layer could be used. > > The extrem cases are the problem. I spend a lot of time to handle > such cases and it is difficult to do this with autoconf only. If > you only use Unix-like systems, probably you will not run into this > problem, thus autoconf only is nice. But imho soon you like to use > other systems, too, e. g. Windows or embedded systems it becomes a > challange. I can imagine it is difficult for someone to understand > my viewpoint, if that person is not developing software for > embedded systems. > > > - get rid of (multiline) macros where feasible, > > Of course it would be possible to remove the "linefeeds", but that > would not help (imho it would make it even more difficult to read). > > I have the following suggestion: > > Because the multiline-macros are to difficult to handle, we can > replace that macros e. g. by direct function-calls. Thus a macro > would become some "alias" only. We also can change (shorten) the > names. For this we have to change the generic implementation. The > advantages would be: > > - debugging is possible > - type-safety > - easier to read > > and for aicas > > - we can keep most of the code we already have written > - we can continue to participate in Classpath > > For aicas this would be some compromise, because we can still use > the target-layer. What would be still important is, that in the > native _no_ OS-specific feature is used directly, like constants or > datatypes or OS-functions calls. With such a implementation we can > survive. I like this idea. In fact I had the same in mind. > What do you think? > > > - shorten macro names, > > Please make some suggestion. If TARGET_NATIVE is to long, what > about > > TARGET or > NATIVE or > NC or > ...? Perhaps a "CP" prefix (for ClassPath). > Please take care that there is some risk for naming-collisions if > the name becomes to short (thus "OS" would probably be a bad > choice). > > > - group files together more intelligently and > > I do not understand what is wrong with the current grouping. The > grouping is done by function-types, e. g. file, network, memory and > so on. How to make it better/more clear? There are several ways to deal with the files. Either do it the AICAS way and group one OS in one dir or do it the libgcj way and put all into one dir, havean OS suffix/prefix in the filename and choose the files are configure time. > > - reduce (better: completely avoid) redundant code (bugs > > multiplication!!). > > Ok. There are some parts of the layer which could be improved. If > we
Re: Target native layer
Dear Michael, > Wouldnt it be better to call it "posix" (note: no captial letter) as it currently supports Linux, *BSD, AIX ... We can and should add Solaris too. With autoconf its really simple. Then AICAS can remove their own Solaris port totally and we have a clean non-redundant codebase for posix-like OSes. It is also not redundant code now, because the special implementation also have to be used in a single-directory code-basis (currently there are separated files which are a "design"-feature; the existence of the files could be called some redundancy). >> I have the following suggestion: >> >> Because the multiline-macros are to difficult to handle, we can >> replace that macros e. g. by direct function-calls. Thus a macro >> would become some "alias" only. We also can change (shorten) the >> names. For this we have to change the generic implementation. The ... > I like this idea. In fact I had the same in mind. I'm very happy to hear this. >>> - shorten macro names, ... > Perhaps a "CP" prefix (for ClassPath). I would prefer some prefix which indicates the "class" of the feature instead, e. g. "NC" (native code) or "TN" (target native), not some relation to the project name (everthing in Classpath is "Classpath", thus should probably have the prefix "CP"). But if "CP" would make you and other members happy, it is ok for me, too. > There are several ways to deal with the files. Either do it the AICAS way and group one OS in one dir or do it the libgcj way and put all into one dir, havean OS suffix/prefix in the filename and choose the files are configure time. I would like to keep the files in separated directories to keep it more clear what files belong to which OS. Usually files are stored in a hierarchical order by using sub-directories. For the "hierachical" include-tree in the target-layer this was also a good idea to avoid confusion in names of files with the same function-group (e. g. "network"), but for different systems (different implementations). > I'm really glad that we now on the way to tackle the issues with the TARGE_NATIVE layer. I'm sorry for this hard discussion. But I'm happy we are one a way to keep going on together. If everybody becomes satisfied with the following new idea for a target layer, I will prepare the changes. New target native layer: - transform code inside macros into functions (this will make debugging possible) - the current macros become an "alias" to the functions (this is important to be able to replace a function by something special if needed) - combine posix-like systems - Linux, BSD, AIX, Solaris - into a single set of files in one directory (usage of autoconf for posix-like systems) - add some more layers from aicas Sincerely, Torsten ___ Classpath mailing list [EMAIL PROTECTED] http://lists.gnu.org/mailman/listinfo/classpath
Re: Target native layer
Am Donnerstag, 12. August 2004 12:16 schrieb Dr. Torsten Rupp: > Dear Michael, > > > Wouldnt it be better to call it "posix" (note: no captial > > letter) as > > it currently supports Linux, *BSD, AIX ... We can and should add > Solaris too. With autoconf its really simple. Then AICAS can remove > their own Solaris port totally and we have a clean non-redundant > codebase for posix-like OSes. > > > It is also not redundant code now, because the special > implementation also have to be used in a single-directory > code-basis (currently there are separated files which are a > "design"-feature; the existence of the files could be called some > redundancy). But a autoconf implementation would be much more understandable by Free Software developers. Its stuff they act with all day long. Thre currently TRAGET_NATIVE solution is ... well ... uncommon and hard to understand. What you call a "design" feature will be called "hard to understand" because its distributed over the whole (not really true) directory tree. Persoanlly I think this is a mis-feature. > > Perhaps a "CP" prefix (for ClassPath). > > I would prefer some prefix which indicates the "class" of the > feature instead, e. g. "NC" (native code) or "TN" (target native), > not some relation to the project name (everthing in Classpath is > "Classpath", thus should probably have the prefix "CP"). But if > "CP" would make you and other members happy, it is ok for me, too. NC (native code): We know its native code, its written in C. TN (target native): We know its for a special target. Its in a special directory when using your solution. CP (ClassPath): We know its classpath BUT this makes it easy to dinstiguish code defined in classpath or outside of classpath, e.g. system headers. > > There are several ways to deal with the files. Either do it the > > AICAS > > way and group one OS in one dir or do it the libgcj way and put all > into one dir, havean OS suffix/prefix in the filename and choose > the files are configure time. > > > I would like to keep the files in separated directories to keep it > more clear what files belong to which OS. Usually files are stored > in a hierarchical order by using sub-directories. For the > "hierachical" include-tree in the target-layer this was also a good > idea to avoid confusion in names of files with the same > function-group (e. g. "network"), but for different systems > (different implementations). But we should put ALL possible OSes into ONE posix layer. Solaris needs no special layer. Its easy to support in a posix layer. That is the way Free Software developers handle it since a long time. No need to have it Solaris in an extra directory. > > I'm really glad that we now on the way to tackle the issues with > > the > > TARGE_NATIVE layer. > > > I'm sorry for this hard discussion. But I'm happy we are one a way > to keep going on together. If everybody becomes satisfied with the > following new idea for a target layer, I will prepare the changes. > > New target native layer: > > - transform code inside macros into functions (this will make > debugging possible) > - the current macros become an "alias" to the functions (this is > important to be able to replace a function by something special if > needed) - combine posix-like systems - Linux, BSD, AIX, Solaris - > into a single set of files in one directory (usage of autoconf for > posix-like systems) - add some more layers from aicas I still have one question: what layers do you want to add and when ? I still saw no other layer then the mis-named "Linux" one. I ask this because layers for embedded OSes that we can never get in touch with or somehow nonsense to all outside of AICAS. We could look at the code but nothing else. This doesnt really help either. Michael ___ Classpath mailing list [EMAIL PROTECTED] http://lists.gnu.org/mailman/listinfo/classpath
Re: Target native layer
Dear Michael, NC (native code): We know its native code, its written in C. TN (target native): We know its for a special target. Its in a special directory when using your solution. CP (ClassPath): We know its classpath BUT this makes it easy to dinstiguish code defined in classpath or outside of classpath, e.g. system headers. I would like to do one more step: we know it is native code, but I would also like to know _which_ part it belongs to in the native code. I see two big parts: target-independent code and target-dependent code (one is above the target native layer, the other is inside the target native laxer). It would be nice to distinguish these two parts by a suitable prefix. Of course "CP" would do this (I can make an "alias" in my brain to map all "CP" to "this is target dependent code"), but I think some prefix which is indicates this more clearly would be better. BUT: this detail is not really important. "CP" is ok for me, if it is ok for other people, too. But we should put ALL possible OSes into ONE posix layer. Solaris needs no special layer. Its easy to support in a posix layer. That is the way Free Software developers handle it since a long time. No need to have it Solaris in an extra directory. Yes, I following you in this point. That was it, I wanted to say with: - combine posix-like systems - Linux, BSD, AIX, Solaris - into a single set of files in one directory (usage of autoconf for posix-like systems) I still have one question: what layers do you want to add and when ? I still saw no other layer then the mis-named "Linux" one. I ask this because layers for embedded OSes that we can never get in touch with or somehow nonsense to all outside of AICAS. We could look at the code but nothing else. This doesnt really help either. I first wanted to avoid to offer here things in advance (otherwise I fear somebody will become upset if I can not do it in time, because of limitated manpower). Currently on the list is: Solaris, RTEMS, embOS and MinGW (Windows). Please do not ask me exactly _when_ I will supply this, but I promise I will do my best to add these as soon as possible. By the way: I think "Linux" is not mis-named, because it is "Linux". It is only a "coincidence" that is 100% identical to the generic case. The background is: we are developing basically with Linux. Sincerly, Torsten ___ Classpath mailing list [EMAIL PROTECTED] http://lists.gnu.org/mailman/listinfo/classpath
Re: Target native layer
Am Donnerstag, 12. August 2004 14:10 schrieb Dr. Torsten Rupp: > I first wanted to avoid to offer here things in advance (otherwise > I fear somebody will become upset if I can not do it in time, > because of limitated manpower). Currently on the list is: Solaris, > RTEMS, embOS and MinGW (Windows). Please do not ask me exactly > _when_ I will supply this, but I promise I will do my best to add > these as soon as possible. That sounds great. Can you just post the code without added to CVS ? This way we can at least look at it. We should discuss all before commiting to CVS anyway. > By the way: I think "Linux" is not mis-named, because it is > "Linux". It is only a "coincidence" that is 100% identical to the > generic case. The background is: we are developing basically with > Linux. Its a difference if code is target for Linux only or if its developed on Linux only. You use only Linux for it. Roman use it on FreeBSD. I use this code on NetBSD too. I have access to Solaris machines and they want to use this code too. So in fact is not Linux anymore, its Posix. Michael > Sincerly, > > Torsten ___ Classpath mailing list [EMAIL PROTECTED] http://lists.gnu.org/mailman/listinfo/classpath
Re: New native layer
Hi Guilhem, On Sat, 2006-01-28 at 14:43 +0100, Guilhem Lavaux wrote: > I would like to mention that I am developping/fine tuning the new native > layer for classpath in a separate branch called "NATIVE-LAYER". If you > have some time to give your impressions then fetch it and look into > native/jni/native-lib, native/jni/java-io, ... > > the new layer itself is developped in native-lib. The other libraries > are being adjusted currently... Thanks for doing this on a branch. That makes it less disruptive and easier for people to choose to test it out before we decide whether to incorporate the work on the trunk. But even for CVS branches you need to send patches to the patches mailinglist. That way people can follow your work and already read up and comment on the things they see that they like or dislike. And it prevents you from having worked on something very hard and then "suddenly" dropping a big patch. You also need to update the ChangeLog file on the branch to show the steps taken in creating the new code. Please use [native] in the subject when sending patches to the list. (Sorry to be pedantic about this, but we want CVS to be our shared development space, not someone personal playground.) And please do send a proper patch for the things you already checked in If you didn't create a branch-point cvs tag then the following should give you the initial diff: cvs diff -Nr classpath-0_20-release native What are the objectives for this branch? How will it be similar/different from the work that Roman was doing? Cheers, Mark signature.asc Description: This is a digitally signed message part
Re: New native layer
Hi Mark, Mark Wielaard wrote: Hi Guilhem, On Sat, 2006-01-28 at 14:43 +0100, Guilhem Lavaux wrote: I would like to mention that I am developping/fine tuning the new native layer for classpath in a separate branch called "NATIVE-LAYER". If you have some time to give your impressions then fetch it and look into native/jni/native-lib, native/jni/java-io, ... the new layer itself is developped in native-lib. The other libraries are being adjusted currently... Thanks for doing this on a branch. That makes it less disruptive and easier for people to choose to test it out before we decide whether to incorporate the work on the trunk. But even for CVS branches you need to send patches to the patches mailinglist. That way people can follow your work and already read up and comment on the things they see that they like or dislike. And it prevents you from having worked on something very hard and then "suddenly" dropping a big patch. You also need to update the ChangeLog file on the branch to show the steps taken in creating the new code. Please use [native] in the subject when sending patches to the list. (Sorry to be pedantic about this, but we want CVS to be our shared development space, not someone personal playground.) Ok. There was no patch at the moment so I could get to a real point. Now is nearly the case. Let's begin the patch flood... And please do send a proper patch for the things you already checked in If you didn't create a branch-point cvs tag then the following should give you the initial diff: cvs diff -Nr classpath-0_20-release native Ok. I'll send three sets of patches to classpath-patches: one to remove target layer (obviously), one to add the foundations of the new layer, and another to show the adaptations to the already existing classpath's code. What are the objectives for this branch? How will it be similar/different from the work that Roman was doing? I thought to have been already clear about that in the past (and with no answers !). Let's try to summarise my goal: * if we want something which is quite portable (but maybe not as portable as aicas portability layer) we must ensure some level of abstraction to hide how syscalls are really used. That way if we have to call different things in the OS (e.g. for more exotic platforms like windows) or to use the syscalls differently we will not be completely stuck by tons of autoconf code in the common VM layer. * some VMs (like us in kaffe) do like to be able to intercept most syscalls to ensure that syscall operations are atomic from the point of view of the threading system. Sometimes you even need to know when a syscall may block to acknowledge the VM to take wise decisions for the GC (I don't remember if it was about wait4 at that time). * autoconf code is a lot clearer if we can isolate the portions of code to be replaced. The native layer I am writing would like to give an answer these problems. The default code will just be a rough encapsulation of the native syscalls (though maybe not for recv/send/... which needs some polling operations). I think that in future we'll need another configure option for the VM to be able to specify a new adaptative native layer. At the moment, I have separated this code in another directory of the classpath tree (native/jni/native-lib). Cheers, Guilhem.
Re: New native layer
On Jan 28, 2006, at 7:21 AM, Guilhem Lavaux wrote: Hi Mark, Mark Wielaard wrote: Hi Guilhem, On Sat, 2006-01-28 at 14:43 +0100, Guilhem Lavaux wrote: I would like to mention that I am developping/fine tuning the new native layer for classpath in a separate branch called "NATIVE- LAYER". If you have some time to give your impressions then fetch it and look into native/jni/native-lib, native/jni/java-io, ... the new layer itself is developped in native-lib. The other libraries are being adjusted currently... Thanks for doing this on a branch. That makes it less disruptive and easier for people to choose to test it out before we decide whether to incorporate the work on the trunk. But even for CVS branches you need to send patches to the patches mailinglist. That way people can follow your work and already read up and comment on the things they see that they like or dislike. And it prevents you from having worked on something very hard and then "suddenly" dropping a big patch. You also need to update the ChangeLog file on the branch to show the steps taken in creating the new code. Please use [native] in the subject when sending patches to the list. (Sorry to be pedantic about this, but we want CVS to be our shared development space, not someone personal playground.) Ok. There was no patch at the moment so I could get to a real point. Now is nearly the case. Let's begin the patch flood... And please do send a proper patch for the things you already checked in If you didn't create a branch-point cvs tag then the following should give you the initial diff: cvs diff -Nr classpath-0_20-release native Ok. I'll send three sets of patches to classpath-patches: one to remove target layer (obviously), one to add the foundations of the new layer, and another to show the adaptations to the already existing classpath's code. What are the objectives for this branch? How will it be similar/different from the work that Roman was doing? I thought to have been already clear about that in the past (and with no answers !). Sorry, I meant to reply about what you were proposing, but forgot :-) What you're working on seems a bit better than the target layer stuff, in that you have real functions that the compiler/debugger can tell you about, if there's an error. I don't, however, entirely see the point of another funcall to wrap the syscalls in. It seems to me as though efforts like this are trying to re-use the bits of JNI code that wrap syscalls in the native parts of VM* classes, and not much else. In my opinion, time would be better spent: - Writing utility functions that make JNI easier to deal with. - Writing concise, portable (in the realm of POSIXy systems, at least) native implementations of VM* classes that require native support. - Optimizing some functions for target systems (e.g., using epoll on Linux, and kqueue on FreeBSD, etc.). But on the whole, making library that is concise, easy-to-understand, and which works efficiently on free platforms. The point is, we want (at least I want) Classpath not just a *complete* class library, but also an extremely good one. I think trying to make system details generic like this accomplishes that. Let's try to summarise my goal: * if we want something which is quite portable (but maybe not as portable as aicas portability layer) we must ensure some level of abstraction to hide how syscalls are really used. That way if we have to call different things in the OS (e.g. for more exotic platforms like windows) or to use the syscalls differently we will not be completely stuck by tons of autoconf code in the common VM layer. It seems like a lot of the argument for this kind of native layer is "we can port to Windows/some exotic platform," but frankly, I don't see it. Windows (in my uninformed opinion ;-) is probably wildly different enough such that writing a win32 `cpio_read' isn't much less work than writing a win32 `Java_gnu_java_nio_channels_FileChannelImpl_read__.' And since any port like this is (AFAIK) merely hypothetical, I don't see the value. * some VMs (like us in kaffe) do like to be able to intercept most syscalls to ensure that syscall operations are atomic from the point of view of the threading system. Sometimes you even need to know when a syscall may block to acknowledge the VM to take wise decisions for the GC (I don't remember if it was about wait4 at that time). That's kind of interesting. Can you explain how this works a little more, or point me to the code where this happens? And why having the native-lib shim between JNI and the system helps this? * autoconf code is a lot clearer if we can isolate the portions of code to be replaced. This is true, to a degree. But in many places where APIs hardly diverge
Re: New native layer
Hi, On Sat, 2006-01-28 at 17:02 -0800, Casey Marshall wrote: > On Jan 28, 2006, at 7:21 AM, Guilhem Lavaux wrote: > > I thought to have been already clear about that in the past (and > > with no answers !). > > Sorry, I meant to reply about what you were proposing, but forgot :-) Yeah, sorry. We all need little reminders from time to time. But showing up with real code does help getting replies :) > What you're working on seems a bit better than the target layer > stuff, in that you have real functions that the compiler/debugger can > tell you about, if there's an error. Agreed, it is definitely better then what we have now with the macros. > I don't, however, entirely see > the point of another funcall to wrap the syscalls in. It seems to me > as though efforts like this are trying to re-use the bits of JNI code > that wrap syscalls in the native parts of VM* classes, and not much > else. I do see some value because it makes it clear where the JNI part ends and where the platform specific part starts. But I am not sure putting all platform specific parts in separate files is always an improvement since most of it is actually pretty small and most of it seems not to be that different between platforms. Adding separate functions for things that might be different or that need VM specific hooks/callbacks is good though. I would just hate to see it overdone for small things that could just as well be in the same file or even function. > In my opinion, time would be better spent: > >- Writing utility functions that make JNI easier to deal with. That is what JCL and NSA were for (native/jni/classpath). They were supposed to be "safe wrappers" for some common things like throwing exceptions, conversion of jstrings and cstrings and generic RawData support that hides whether the platform is 32 or 64 bits. But those functions were never really worked out much, nor are they used very consistently in all our JNI code at the moment. >- Writing concise, portable (in the realm of POSIXy systems, at > least) native implementations of VM* classes that require native > support. Through autoconf and replacement functions where needed. That is how libgcj (and kaffe) do it. > > * if we want something which is quite portable (but maybe not as > > portable as aicas portability layer) we must ensure some level of > > abstraction to hide how syscalls are really used. That way if we > > have to call different things in the OS (e.g. for more exotic > > platforms like windows) or to use the syscalls differently we will > > not be completely stuck by tons of autoconf code in the common VM > > layer. > > It seems like a lot of the argument for this kind of native layer is > "we can port to Windows/some exotic platform," but frankly, I don't > see it. Windows (in my uninformed opinion ;-) is probably wildly > different enough such that writing a win32 `cpio_read' isn't much > less work than writing a win32 > `Java_gnu_java_nio_channels_FileChannelImpl_read__.' And since any > port like this is (AFAIK) merely hypothetical, I don't see the value. libgcj has a somewhat separate Windows implementation of some platform native code (see the nat*Win32.cc files in gnu/java/nio, gnu/java/net, java/lang and java/net in gcc/libjava). And they also support cygwin and mingw to a point (although I don't know whether the Posix or Win32 native implementation is used by either the cygwin or mingw target). All this can/should of course be wrapped behind our VMClass interface layer since that also helps making our library JNI/CNI/.NET/Oberon/etc agnostic. > I don't mind this proposal, and I think you should go ahead with it. > I still have my own opinions about how to write Classpath's native > library, so I may try my own experiment in another branch (unless, of > course, no-one really agrees with me, in which case I will just stop > critiquing other people's code). I think I agree with Casey. Please go ahead with your work Guilhem. We will try to be constructive in our criticism. Actually doing the work and providing patches is a very good way to proof your point that this is needed and makes things cleaner. Thanks for working on this. Cheers, Mark signature.asc Description: This is a digitally signed message part
Re: New native layer
On Sun, 2006-01-29 at 15:17 +0100, Mark Wielaard wrote: > Hi, > >- Writing concise, portable (in the realm of POSIXy systems, at > > least) native implementations of VM* classes that require native > > support. > > Through autoconf and replacement functions where needed. That is how > libgcj (and kaffe) do it. I'd like to praise gnulib there a bit, since it makes some of that stuff really nice and convenient. It's a fairly new thing, though. cheers, dalibor topic
Re: New native layer
Guilhem Lavaux wrote: I thought to have been already clear about that in the past (and with no answers !). Let's try to summarise my goal: * if we want something which is quite portable (but maybe not as portable as aicas portability layer) we must ensure some level of abstraction to hide how syscalls are really used. That way if we have to call different things in the OS (e.g. for more exotic platforms like windows) or to use the syscalls differently we will not be completely stuck by tons of autoconf code in the common VM layer. * some VMs (like us in kaffe) do like to be able to intercept most syscalls to ensure that syscall operations are atomic from the point of view of the threading system. Sometimes you even need to know when a syscall may block to acknowledge the VM to take wise decisions for the GC (I don't remember if it was about wait4 at that time). * autoconf code is a lot clearer if we can isolate the portions of code to be replaced. The native layer I am writing would like to give an answer these problems. The default code will just be a rough encapsulation of the native syscalls (though maybe not for recv/send/... which needs some polling operations). I think that in future we'll need another configure option for the VM to be able to specify a new adaptative native layer. At the moment, I have separated this code in another directory of the classpath tree (native/jni/native-lib). It would be nice, I believe, to re-use libraries that have handled most of the "porting" and "wrapping" for you such as APR (http://apr.apache.org/), or NPR (http://www.mozilla.org/projects/nspr/) to platforms GNU Classpath might care to support. You might also find glib useful, http://developer.gnome.org/arch/gtk/glib.html. If Apache's license and GPLv3 will be compatible then this is probably the way to go. People that care about platform specific optimizations, or code size, etc. could then go beat up the external library provider or write a replacement themselves. Brian
Re: New native layer
On Mon, 2006-01-30 at 12:20 -0500, Brian Jones wrote: > It would be nice, I believe, to re-use libraries that have handled most > of the "porting" and "wrapping" for you such as APR > (http://apr.apache.org/), or NPR (http://www.mozilla.org/projects/nspr/) > to platforms GNU Classpath might care to support. You might also find > glib useful, http://developer.gnome.org/arch/gtk/glib.html. I think that's a nice idea in theory, but not so nice in practice. APR is not GPL compatible atm, though it may be next year. NSPR looks like it is GPL compatible right now, all files I peeked at are dual licensed with the GPL or tri-licensed with GPL and LGPL. For added fun, NSPR actually comes with its own tiny mark and sweep gc, a Java-like Monitor data type, and a bunch of other goodies. APR is being used by subversion and the Apache web server. NSPR is being used by evolution, openoffice & various mozilla projects. They seem to cover a similar range of platforms, thought NSPR also now covers RISC OS ;) Otoh, there is no lack of GPL-compatible utility libraries, from Glib, and GNU Common C++, to Qt's non-GUI classes & Boost. I'd have a hard time picking any of them, since they all seem to address problems larger than those we'd need addressing at the moment (we're expecting a largely POSIXy/glibc environment anyway, and don't have any ports to very weird environments yet in the CVS tree), while at the same time 'obfuscating' the code in ways similar to the current target layer. I believe that's the reason why neither APR nor NSPR (nor other similar projects) have seen wide spread acceptance, even though they have been around for many years now: they don't really pay off enough for people to rewrite their existing code bases to them. See also what the OpenBSD developers did as one of the first things, once they 'forked' Apache web server 1.3.x after the Apache license change: rip out APR and replace its use by plain old posix functions, since those were more maintainable for them. cheers, dalibor topic
Re: New native layer
Dalibor Topic wrote: On Mon, 2006-01-30 at 12:20 -0500, Brian Jones wrote: It would be nice, I believe, to re-use libraries that have handled most of the "porting" and "wrapping" for you such as APR (http://apr.apache.org/), or NPR (http://www.mozilla.org/projects/nspr/) to platforms GNU Classpath might care to support. You might also find glib useful, http://developer.gnome.org/arch/gtk/glib.html. I believe that's the reason why neither APR nor NSPR (nor other similar projects) have seen wide spread acceptance, even though they have been around for many years now: they don't really pay off enough for people to rewrite their existing code bases to them. Okay, but I brought it up since the work Guilhem Lavaux is doing sounds an awful lot like a rewrite. Although technically I think you can just run the macro processor by hand to get the current real source code for say Linux and autoconfiscate it from there. Or you could go back a couple of years and look in native/ for the original autoconf/posix version of the libraries and move forward from there. See also what the OpenBSD developers did as one of the first things, once they 'forked' Apache web server 1.3.x after the Apache license change: rip out APR and replace its use by plain old posix functions, since those were more maintainable for them. Yea, I think the point for me would be to keep Classpath's java hackers out of the business of writing native code, and especially out of the business of porting native code for such common idioms as generic file operations, network operations, etc. Anyway, that was my $0.02, Brian
Re: New native layer
Brian Jones wrote: Dalibor Topic wrote: On Mon, 2006-01-30 at 12:20 -0500, Brian Jones wrote: It would be nice, I believe, to re-use libraries that have handled most of the "porting" and "wrapping" for you such as APR (http://apr.apache.org/), or NPR (http://www.mozilla.org/projects/nspr/) to platforms GNU Classpath might care to support. You might also find glib useful, http://developer.gnome.org/arch/gtk/glib.html. I believe that's the reason why neither APR nor NSPR (nor other similar projects) have seen wide spread acceptance, even though they have been around for many years now: they don't really pay off enough for people to rewrite their existing code bases to them. Okay, but I brought it up since the work Guilhem Lavaux is doing sounds an awful lot like a rewrite. Although technically I think you can just run the macro processor by hand to get the current real source code for say Linux and autoconfiscate it from there. Or you could go back a couple of years and look in native/ for the original autoconf/posix version of the libraries and move forward from there. See also what the OpenBSD developers did as one of the first things, once they 'forked' Apache web server 1.3.x after the Apache license change: rip out APR and replace its use by plain old posix functions, since those were more maintainable for them. Yea, I think the point for me would be to keep Classpath's java hackers out of the business of writing native code, and especially out of the business of porting native code for such common idioms as generic file operations, network operations, etc. Hi Brian, The code is not really a rewrite. Actually I am just moving out TARGET layer and replacing it with native function call with an API not so different than posix. My idea is that these function calls should be purely POSIX (so that classpath's hackers does not have deal with subtleties). Anyway, as I have already said it, seeing the amount of code to do pure administrations in the JNI world, it is easier to read autoconfiscated code which is completely outside the JNI code (again look at javanet). The advantage of this method is that some VMs may want to reroute these syscalls and it is easy to do so using this semi-abstract interface without rewriting anything. Regards, Guilhem.
Re: New native layer
Hi Brian, hi list, > Yea, I think the point for me would be to keep Classpath's java hackers > out of the business of writing native code, and especially out of the > business of porting native code for such common idioms as generic file > operations, network operations, etc. BTW, Torsten, the man who first wrote the target native layer, mostly works on native code and porting of this to platforms you would not dream of. He is hardly a 'Classpath java hacker'. If the world was so nice that posix and autoconf is the solution to everything, then such work would hardly be necessary. But this seems to be a little behind the horizon of the brave GNU world. Man, all this narrow-mindedness sets me up, I think I better go back to java hacking ;-) Roman signature.asc Description: Dies ist ein digital signierter Nachrichtenteil
Re: New native layer
On Jan 31, 2006, at 11:32 AM, Roman Kennke wrote: Hi Brian, hi list, Yea, I think the point for me would be to keep Classpath's java hackers out of the business of writing native code, and especially out of the business of porting native code for such common idioms as generic file operations, network operations, etc. BTW, Torsten, the man who first wrote the target native layer, mostly works on native code and porting of this to platforms you would not dream of. That is arrogant. I'm sure everyone here who hacks on Classpath has worked with interesting technologies, and are all great engineers, each in his own right. So let's stop measuring cocks here, OK? He is hardly a 'Classpath java hacker'. If the world was so nice that posix and autoconf is the solution to everything, then such work would hardly be necessary. But this seems to be a little behind the horizon of the brave GNU world. Man, all this narrow-mindedness sets me up, I think I better go back to java hacking ;-) GNU Classpath is STILL a GNU project, and above all, we are trying to support free platforms, and are first of all concerned with supporting GNU (in it's GNU/Linux form). There is reams of evidence that targeting mostly POSIX systems, and handling the divergences with autoconf, produces a lot of software that is usable in a lot of places (witness, all the programs of the GNU project). Please, recognize that: - Macro-based portability layers like this are ugly and extremely hard to maintain and debug. - The benefits of such a thing never materialized anyway, because the only implementation available is for POSIX-like platforms. So it STILL relied on POSIX and autoconf. We have the responsibility, as contributors to a GNU project, to maintain the project for the GNU system. GNU is sorta-POSIX, as are a lot of other interesting platforms, and targeting them earns us, as free software contributors -- not necessarily other groups or companies that want to use Classpath -- a lot. I see these "native portability layers" as being counter to that goal, and they especially don't make sense given that there's no other free implementations for platforms other than what we are targeting. You can call that narrow-minded if you want, but we have to have goals and rules for the project, and they should be mandated by the larger project of which we are a part, which is GNU.
Re: New native layer
Casey Marshall wrote: We have the responsibility, as contributors to a GNU project, to maintain the project for the GNU system. GNU is sorta-POSIX, as are a lot of other interesting platforms, and targeting them earns us, as free software contributors -- not necessarily other groups or companies that want to use Classpath -- a lot. I see these "native portability layers" as being counter to that goal, and they especially don't make sense given that there's no other free implementations for platforms other than what we are targeting. An alternative take with similar conclusion: There is a standard "native portability layer". It is called Posix. There are multiple implementations of this layer available for MS-Windows. Other platforms we are likely to support (such as OS-X) already support Posix. I.e. there is no need for an extra portability layer. That is not to say we cannot add hooks and conditional compilation in modest doses to support Windows and other non-Posix platforms. But any extra indirection that hurts performance on Posix (even trivially), or anything that makes the code harder to maintain and understand is generally inappropriate. -- --Per Bothner [EMAIL PROTECTED] http://per.bothner.com/
Re: New native layer
Hi Casey, Am Dienstag, den 31.01.2006, 12:37 -0800 schrieb Casey Marshall: > On Jan 31, 2006, at 11:32 AM, Roman Kennke wrote: > > > Hi Brian, hi list, > > > >> Yea, I think the point for me would be to keep Classpath's java > >> hackers > >> out of the business of writing native code, and especially out of the > >> business of porting native code for such common idioms as generic > >> file > >> operations, network operations, etc. > > > > BTW, Torsten, the man who first wrote the target native layer, mostly > > works on native code and porting of this to platforms you would not > > dream of. > > That is arrogant. I'm sure everyone here who hacks on Classpath has > worked with interesting technologies, and are all great engineers, > each in his own right. > > So let's stop measuring cocks here, OK? Sorry, I didn't want to sound arrogant or whatnot. I am only a bit upset about this braindead discussion again. I get the impression that there is a general opposition against real portability concerns within the GNU (Classpath?) project. Correct me if I am wrong. There certainly are parties that have portability demands that go beyong posix. Granted, Aicas is a commercial party and cannot publish every port. There is also Kaffe, which is 100% GPL and now also suggest something similar like the target layer, only somewhat nicer, and again it seems to turn out to be a fight against windmills. But after all, that is what we have the VM interface for and I for myself don't want to discuss C issues anymore and instead concentrate on getting the VM interface into a good shape (it already is IMO, needs some tweaks though) and let people with different opinions on native code fork/implement their own stuff below the VM interface. Cheers, Roman signature.asc Description: Dies ist ein digital signierter Nachrichtenteil
Re: New native layer
On Jan 31, 2006, at 1:00 PM, Roman Kennke wrote: Hi Casey, Am Dienstag, den 31.01.2006, 12:37 -0800 schrieb Casey Marshall: On Jan 31, 2006, at 11:32 AM, Roman Kennke wrote: Hi Brian, hi list, Yea, I think the point for me would be to keep Classpath's java hackers out of the business of writing native code, and especially out of the business of porting native code for such common idioms as generic file operations, network operations, etc. BTW, Torsten, the man who first wrote the target native layer, mostly works on native code and porting of this to platforms you would not dream of. That is arrogant. I'm sure everyone here who hacks on Classpath has worked with interesting technologies, and are all great engineers, each in his own right. So let's stop measuring cocks here, OK? Sorry, I didn't want to sound arrogant or whatnot. I am only a bit upset about this braindead discussion again. I'm sure some of us do appreciate the discussion, and don't find it brain-dead in the least. You have your own goals, and have made up your mind about POSIX and autoconf, that is clear, so you can feel free to ignore this discussion. I get the impression that there is a general opposition against real portability concerns within the GNU (Classpath?) project. Correct me if I am wrong. I think you're mistaken, if you think it is mere opposition. We have, like Per said in another reply, a portability layer already, and it is POSIX. It's an established standard, and it is available on many free (and non-free) platforms today, all of which we desire our code to run on with the highest priority. To put it a little more bluntly, and to paraphrase David Wallace, we can really only love what we value. Portability for the sake of platforms we don't use, or even know the names of, is not something we find immediately valuable. There certainly are parties that have portability demands that go beyong posix. Granted, Aicas is a commercial party and cannot publish every port. There is also Kaffe, which is 100% GPL and now also suggest something similar like the target layer, only somewhat nicer, and again it seems to turn out to be a fight against windmills. I don't get the windmills reference, but I think I get your point. With Kaffe, it isn't completely clear to me what the issues are, and how a native portability layer solves them. But after all, that is what we have the VM interface for and I for myself don't want to discuss C issues anymore and instead concentrate on getting the VM interface into a good shape (it already is IMO, needs some tweaks though) and let people with different opinions on native code fork/implement their own stuff below the VM interface. Yes, that is the best idea, I think. I still want to see a nice POSIXy reference implementation, though, even if it is only for the selfish reason so I can run jamvm on my OS X and GNU/Linux machines to test out code.
Re: New native layer
Per Bothner wrote: Casey Marshall wrote: We have the responsibility, as contributors to a GNU project, to maintain the project for the GNU system. GNU is sorta-POSIX, as are a lot of other interesting platforms, and targeting them earns us, as free software contributors -- not necessarily other groups or companies that want to use Classpath -- a lot. I see these "native portability layers" as being counter to that goal, and they especially don't make sense given that there's no other free implementations for platforms other than what we are targeting. An alternative take with similar conclusion: There is a standard "native portability layer". It is called Posix. There are multiple implementations of this layer available for MS-Windows. Other platforms we are likely to support (such as OS-X) already support Posix. I.e. there is no need for an extra portability layer. Exactly my sentiments (as a bystander in this thread so far) The parts of POSIX that Classpath uses are pretty much exactly those parts that Classpath needs. So why not just implement that subset of POSIX on your obscure platform? As a side-benefit, lots of other stuff might compile and work on your platform then too. Another thing that bugs me is that it's extra work to create a new 'layer' for each platform. Instead, the autoconf system is much more efficient: once you implement a test for whatever non-POSIX thing, that test can work for all platforms that need it, now and in the future. The "layer" idea is too big a level of granularity to be porting at. We should be porting more like at the function call and macro level. -Archie __ Archie Cobbs *CTO, Awarix* http://www.awarix.com
Re: New native layer
On Tue, 2006-01-31 at 22:00 +0100, Roman Kennke wrote: > I get the impression that there > is a general opposition against real portability concerns within the GNU > (Classpath?) project. Correct me if I am wrong. There certainly are > parties that have portability demands that go beyong posix. Yes. Many parties probably do, or will do eventually. It's in the nature of free software to get ported, somehow, to anything people want to run it on. ;) But I doubt there are many developers inside GNU Classpath as a whole who'd be interested in maintaining non-POSIX code, since most VMs either support only POSIX, POSIX + something special or only something special, so that the set of stuff that's got a fair chance of being communally maintained since it is useful to more than a single VM boils down to POSIX alone. That's pretty much the stuff that has a fair chance of getting love by diffeent VMs in the common tree. So if, theoretically, Kaffe was ported to run on top of Ruby[0] it would make most sense to keep the maintenance & the associated busywork for the Ruby port of GNU Classpath's native libraries in KaffeOnRuby, rather than trying to maintain it inside GNU Classpath. Same if JamVM was ported to 16-bit Minix 2 on Amiga: the necessary changes to the native layer would be too invasive to bother the POSIX-only VMs with, as they'd add some burden without necessary immediate gratification for posix-only VMs. Immediate gratification good, maintenance burden bad. :) I think the hard question is how to make the native layer stuff such that a) most vms can profit from it, in particular when starting from ground up and b) it is easy enough to embrace and extend for those cases where it does not fit well, without forcing one to wholesome fork the native layer, or to maintain the odd, single-VM cases inside GNU Classpath. The answer to a) seems to be POSIX for the reasons given above, and to b) seems to be something like Guilhem's work on the native layer. > Granted, > Aicas is a commercial party and cannot publish every port. There is also > Kaffe, which is 100% GPL and now also suggest something similar like the > target layer, only somewhat nicer, and again it seems to turn out to be > a fight against windmills. A major issue is that for green threads (i.e. jthreads) in Kaffe, we'd have to make sure that we can disable interrupts before invoking a system method, and enable them before we exit again. If we don't do that, than bad things can (and do ;) happen. That's a pretty Kaffe-specific need, though, so it may not make sense to make room for it in GNU Classpath, unless it is accompanied by some other, instant gratification, like looking very much like plain old POSIX. In theory, if we're lucky, Kaffe may also be able to get by using AspectC++[1] or something equivalently weird to weave in the stuff we need for jthreads. In practice, I don't think anyone has done something like that yet, though. ;) cheers, dalibor topic [0] Just imagine the potential for book sales! ;) [1] http://www.aspectc.org/Publications.6.0.html
Re: New native layer
On Jan 31, 2006, at 3:51 PM, Dalibor Topic wrote: On Tue, 2006-01-31 at 22:00 +0100, Roman Kennke wrote: Granted, Aicas is a commercial party and cannot publish every port. There is also Kaffe, which is 100% GPL and now also suggest something similar like the target layer, only somewhat nicer, and again it seems to turn out to be a fight against windmills. A major issue is that for green threads (i.e. jthreads) in Kaffe, we'd have to make sure that we can disable interrupts before invoking a system method, and enable them before we exit again. If we don't do that, than bad things can (and do ;) happen. That's a pretty Kaffe-specific need, though, so it may not make sense to make room for it in GNU Classpath, unless it is accompanied by some other, instant gratification, like looking very much like plain old POSIX. In theory, if we're lucky, Kaffe may also be able to get by using AspectC++[1] or something equivalently weird to weave in the stuff we need for jthreads. In practice, I don't think anyone has done something like that yet, though. ;) Would it help to have a utility function, say as a private function in the JNIEnv, that told the VM that the code was entering or just exited a possibly-blocking system call? Which, in the default implementation for other VMs or threading systems, would be a no-op? E.g.: void EnterSystemCall (JNIEnv *env, void *address); void ExitSystemCall (JNIEnv *env, void *address); This would (I think) be a good place for Kaffe to disable whatever interrupts it had to, and would still be a generic-enough mechanism for a VM to keep track of where the native code is. We would need to find all the right places to call this, but I don't think that's much harder than rewriting everything with wrapper functions, and it would be extensible -- anything added that Kaffe needed to tweak on before calling would be made safe, as long as the usage was consistent.
Re: New native layer
Jikes RVM also does m-to-n threading, so it's there's more than 1 VM that's whacky in this regard. The things we need to do are most likely different than what Kaffe needs to do, but having a chance to inject a VM callback before the thread dives off into a blocking system call is something we would like to be able to do. We have some linux specific hacks (evil with dlopen to intercept poll, select, etc), but it's fragile and doesn't work on other platforms like AIX and OS X that Jikes RVM runs on. --dave > A major issue is that for green threads (i.e. jthreads) in Kaffe, we'd > have to make sure that we can disable interrupts before invoking a > system method, and enable them before we exit again. If we don't do > that, than bad things can (and do ;) happen. > > That's a pretty Kaffe-specific need, though, so it may not make sense to > make room for it in GNU Classpath, unless it is accompanied by some > other, instant gratification, like looking very much like plain old > POSIX. In theory, if we're lucky, Kaffe may also be able to get by using > AspectC++[1] or something equivalently weird to weave in the stuff we > need for jthreads. In practice, I don't think anyone has done something > like that yet, though. ;)
Re: New native layer
On Jan 31, 2006, at 6:10 PM, David P Grove wrote: Jikes RVM also does m-to-n threading, so it's there's more than 1 VM that's whacky in this regard. The things we need to do are most likely different than what Kaffe needs to do, but having a chance to inject a VM callback before the thread dives off into a blocking system call is something we would like to be able to do. We have some linux specific hacks (evil with dlopen to intercept poll, select, etc), but it's fragile and doesn't work on other platforms like AIX and OS X that Jikes RVM runs on. Would my suggestion for Enter/Exit callbacks help this?
Re: New native layer
Casey Marshall wrote: On Jan 31, 2006, at 6:10 PM, David P Grove wrote: Jikes RVM also does m-to-n threading, so it's there's more than 1 VM that's whacky in this regard. The things we need to do are most likely different than what Kaffe needs to do, but having a chance to inject a VM callback before the thread dives off into a blocking system call is something we would like to be able to do. We have some linux specific hacks (evil with dlopen to intercept poll, select, etc), but it's fragile and doesn't work on other platforms like AIX and OS X that Jikes RVM runs on. Would my suggestion for Enter/Exit callbacks help this? Though enter/exit callbacks may help for a few syscalls but it won't for some (like blocking read/write). For these new calls you need handle blocking queues in the VM thread scheduler. That needs to put the fd into non-blocking mode and when you do a read you first check if there is any data (in non-blocking mode) and if there is not data then the thread system queues the current thread and tells the scheduler to jump somewhere else until some datas are available on the specified fd. So we must really stick to rerouting a few IOs. I am sure that handling m-to-n threading system is as difficult as what we are doing (maybe more). Regards, Guilhem.
Re: New native layer
yep, that's pretty much the situation with m-n threading in Jikes RVM. For the blocking operations, we need to replace the entire call with something else. There might be some cases where the callbacks could be useful, but they wouldn't be a complete solution for us. --dave Guilhem Lavaux <[EMAIL PROTECTED]> wrote on 02/02/2006 01:27:28 PM: > > Though enter/exit callbacks may help for a few syscalls but it won't for > some (like blocking read/write). For these new calls you need handle > blocking queues in the VM thread scheduler. That needs to put the fd > into non-blocking mode and when you do a read you first check if there > is any data (in non-blocking mode) and if there is not data then the > thread system queues the current thread and tells the scheduler to jump > somewhere else until some datas are available on the specified fd. So we > must really stick to rerouting a few IOs. I am sure that handling m-to-n > threading system is as difficult as what we are doing (maybe more).
Re: New native layer
On Feb 2, 2006, at 10:27 AM, Guilhem Lavaux wrote: Casey Marshall wrote: On Jan 31, 2006, at 6:10 PM, David P Grove wrote: Jikes RVM also does m-to-n threading, so it's there's more than 1 VM that's whacky in this regard. The things we need to do are most likely different than what Kaffe needs to do, but having a chance to inject a VM callback before the thread dives off into a blocking system call is something we would like to be able to do. We have some linux specific hacks (evil with dlopen to intercept poll, select, etc), but it's fragile and doesn't work on other platforms like AIX and OS X that Jikes RVM runs on. Would my suggestion for Enter/Exit callbacks help this? Though enter/exit callbacks may help for a few syscalls but it won't for some (like blocking read/write). For these new calls you need handle blocking queues in the VM thread scheduler. That needs to put the fd into non-blocking mode and when you do a read you first check if there is any data (in non-blocking mode) and if there is not data then the thread system queues the current thread and tells the scheduler to jump somewhere else until some datas are available on the specified fd. So we must really stick to rerouting a few IOs. I am sure that handling m-to-n threading system is as difficult as what we are doing (maybe more). Aha, I grok the issue now. Would it make sense to do conditional compilation based on a "green threads" model, so that you can compile all blocking I/O calls to support such a model, if configured thusly? Or would file descriptor callbacks help -- where the FD is put into non-blocking mode before the call, then reset after, with support to catch the `EAGAIN' if it arrives? Also, I'm sure you could write a kernel-threads compatible library that also supported user-mode thread models at the same time (the restrictions on the former seem to be a subset of the latter), but don't know about what performance drain that would incur on kernel threaded systems. Because, "one POSIX library to rule them all," that supported both thread models, would still be an awesome thing.
Re: New native layer
Casey Marshall wrote: On Feb 2, 2006, at 10:27 AM, Guilhem Lavaux wrote: Casey Marshall wrote: On Jan 31, 2006, at 6:10 PM, David P Grove wrote: Jikes RVM also does m-to-n threading, so it's there's more than 1 VM that's whacky in this regard. The things we need to do are most likely different than what Kaffe needs to do, but having a chance to inject a VM callback before the thread dives off into a blocking system call is something we would like to be able to do. We have some linux specific hacks (evil with dlopen to intercept poll, select, etc), but it's fragile and doesn't work on other platforms like AIX and OS X that Jikes RVM runs on. Would my suggestion for Enter/Exit callbacks help this? Though enter/exit callbacks may help for a few syscalls but it won't for some (like blocking read/write). For these new calls you need handle blocking queues in the VM thread scheduler. That needs to put the fd into non-blocking mode and when you do a read you first check if there is any data (in non-blocking mode) and if there is not data then the thread system queues the current thread and tells the scheduler to jump somewhere else until some datas are available on the specified fd. So we must really stick to rerouting a few IOs. I am sure that handling m-to-n threading system is as difficult as what we are doing (maybe more). Aha, I grok the issue now. Would it make sense to do conditional compilation based on a "green threads" model, so that you can compile all blocking I/O calls to support such a model, if configured thusly? Or would file descriptor callbacks help -- where the FD is put into non-blocking mode before the call, then reset after, with support to catch the `EAGAIN' if it arrives? Hi Casey, You would continue to need some VM callbacks. Putting the FD in non-blocking mode is not sufficient though I could maybe imagine only rerouting select/poll in such occasions. You will also need the Enter/LeaveSyscall callback to ensure the thread safety of errors. To sum up, you can always invent a bunch of interfaces which can look more compact but they will give you a more obscure code. #if defined(GREEN_THREADS) #ifdef THIS_VARIANT_DOES_NOT_LIKE_THIS_SYSCALL #else #endif #ifdef... #endif #else /* ! GREEN_THREADS */ /* Do some other macro checking */ #endif Put in the perspective of what we already has around the code it will be a mess... Also, I'm sure you could write a kernel-threads compatible library that also supported user-mode thread models at the same time (the restrictions on the former seem to be a subset of the latter), but don't know about what performance drain that would incur on kernel threaded systems. Because, "one POSIX library to rule them all," that supported both thread models, would still be an awesome thing. Performance drain would a be one more indirection by syscall. I am not sure if it really matters as anyway syscalls are expected to be long compared to a usual function call. I can run some performance test on my computer though. The testcase would be: syscall.c: write some function to alias a syscall. I compile syscall.c with PIC to ensure the worst case. Then: main.c: Call this syscall an huge amount of time. Then compare to calling the real syscall. Regards, Guilhem.
Re: New native layer
Hi Casey, Following my previous mail, please try this small performance test. It illustrates what I want to do and the performance "loss" that is expected. Simply unpack and run make. Cheers, Guilhem. Casey Marshall wrote: On Feb 2, 2006, at 10:27 AM, Guilhem Lavaux wrote: Casey Marshall wrote: On Jan 31, 2006, at 6:10 PM, David P Grove wrote: Jikes RVM also does m-to-n threading, so it's there's more than 1 VM that's whacky in this regard. The things we need to do are most likely different than what Kaffe needs to do, but having a chance to inject a VM callback before the thread dives off into a blocking system call is something we would like to be able to do. We have some linux specific hacks (evil with dlopen to intercept poll, select, etc), but it's fragile and doesn't work on other platforms like AIX and OS X that Jikes RVM runs on. Would my suggestion for Enter/Exit callbacks help this? Though enter/exit callbacks may help for a few syscalls but it won't for some (like blocking read/write). For these new calls you need handle blocking queues in the VM thread scheduler. That needs to put the fd into non-blocking mode and when you do a read you first check if there is any data (in non-blocking mode) and if there is not data then the thread system queues the current thread and tells the scheduler to jump somewhere else until some datas are available on the specified fd. So we must really stick to rerouting a few IOs. I am sure that handling m-to-n threading system is as difficult as what we are doing (maybe more). Aha, I grok the issue now. Would it make sense to do conditional compilation based on a "green threads" model, so that you can compile all blocking I/O calls to support such a model, if configured thusly? Or would file descriptor callbacks help -- where the FD is put into non-blocking mode before the call, then reset after, with support to catch the `EAGAIN' if it arrives? Also, I'm sure you could write a kernel-threads compatible library that also supported user-mode thread models at the same time (the restrictions on the former seem to be a subset of the latter), but don't know about what performance drain that would incur on kernel threaded systems. Because, "one POSIX library to rule them all," that supported both thread models, would still be an awesome thing. perf.tar.gz Description: Unix tar archive
Re: New native layer
On Feb 4, 2006, at 12:53 PM, Guilhem Lavaux wrote: Hi Casey, Following my previous mail, please try this small performance test. It illustrates what I want to do and the performance "loss" that is expected. Simply unpack and run make. I wasn't saying that what you were doing would incur any performance penalty; I was just speculating about ways to write a native lib that supported user threads out of the box.
C native layer (pre-announcement)
Dear developers,
I'm a software engineer at aicas GmbH, Karlsruhe/Germany in the group
"operating systems". As formerly discusses with Brian Jones and my
college Andy Walter I adapted our 'C native layer' for the GNU
classpath project to be able to port the library to different operating
systems. The first version of this new layer for GNU classpath under
Linux is now complete. Because of too much other things to do we had a
delay in providing the C native layer to the GNU classpath project
earlier, but now we believe we are ready.
Currently there is a discussion to merge some or all of the native
functions with the GCJ project. Because this will probably have some
influence to the changes I made to the GNU classpath sources I want to
post the following read-me in this list before I check-in the new
sources. Please feel free to send me your comments. We use the native
layer already in our VM "Jamaica" and currently I'm working on native
layers for other operating systems, like vxWorks, Cygwin or QNX.
Maybe someone who is familiar with the GCJ project, too, would also be
able to check if the C native layer could also be used with the GCJ
project and how to make some "super merge". Any hints are welcome.
Sincerely,
Torsten
The GNU classpath native layer
~~
To enable GNU classpath to run on a diverse array of different hardware
platforms, a new native software layer has been added. This layer hide all
machine and hardware dependent issues except common available definitions,
which are ANSI C functions. For each targets system where the GNU classpath
library is to be used, a specific set of native layer functions have to be
provided. A generic set of functions is provided for Unix like systems
(currently tested only with Linux). For a new target system, some or
all native layer functions have to be rewritten. The following scheme
illustrate the native layer.
Java API
--
|Java classes |
| |
| C native functions |
| |
>> |C native layer| <<
| |
| operating system |
| |
| hardware |
--
The C native layer is implemented as a set of C pre-processor native macros.
These macros expand to the appropriated native code. Macros are used
instead function calls to give optimal performance and small code size.
Of course in special cases, a macro can also expand to a function call
if this is needed. This approach provide a flexible and efficient
implementation of the native layer.
The naming pattern for native macros is like follows:
TARGET_NATIVE__
where is a name of a module, e. g. FILE; is
the name of the specific native macro, e. g. OPEN_READ.
The parameters for the macro use in general (with a few exceptions) the
scheme
,,...,
where is input/output parameter and is the result
code TARGET_NATIVE_OK or TARGET_NATIVE_ERROR. Specific error codes
and error strings can be gotten with
TARGET_NATIVE_LAST_ERROR and
TARGET_NATIVE_LAST_ERROR_STRING
(see also file target_generic.h).
For a single target system there exists two sets of native macros in
the files
a) /target_native_.h
b) generic/target_generic_.h
The macros in "a" are target specific implementations of native
functions, the macros in "b" are generic implementations (for Unix) of
the same native functions. If a native macro is not defined in the file
"a", then the definition in file "b" is used (there is a check to see if
a native macros is already defined elsewhere). This technique enables
"a" to 'overwrite' single generic native macros with specific native
macros for a specific target. In the default case, where only the
generic implementation of the native macros is used, the files in the
directory '' are empty except for the mandatory include of the
generic header file in the directory 'generic' at the end. Please
look at the existing Linux target specific files.
The directory and file structure is as follows.
native
...
|
|--- target
| |
| |--- Linux
| | |--- target_native_.h
| | |--- ...
| | ...
| |--- ...
| |--- generic
| | |--- target_generic_.h
| | |--- ...
.........
Include hierarchy is as follows.
native file
--> include 'target_native_.h'
...
...
--> include 'target_generic_.h'
...
...
When writing native code, please take care with the following.
- Use _only_ ANSI C specific functions directly which are avai
Re: C native layer (pre-announcement)
"Dr. Torsten Rupp" <[EMAIL PROTECTED]> writes: > Currently there is a discussion to merge some or all of the native > functions with the GCJ project. It's come up in the past. The essential problem, for lack of a better term, is that because underlying code is not shared that native bugs must be fixed in two source trees. > Because this will probably have some influence to the changes I made > to the GNU classpath sources I want to post the following read-me in > this list before I check-in the new sources. It would be a good idea to post the "patch" to the mailing list first by using 'cvs diff -u' due to the size. Except for one minor directory name change I have no complaints. > The GNU classpath native layer > ~~ > > To enable GNU classpath to run on a diverse array of different hardware > platforms, a new native software layer has been added. This layer hide all > machine and hardware dependent issues except common available definitions, > which are ANSI C functions. For each targets system where the GNU classpath > library is to be used, a specific set of native layer functions have to be > provided. A generic set of functions is provided for Unix like systems > (currently tested only with Linux). For a new target system, some or > all native layer functions have to be rewritten. The following scheme > illustrate the native layer. > > Java API >-- > |Java classes | > | -------- | > | C native functions | > | | >>> |C native layer| << > | | > | operating system | > | | > | hardware | >-- > > The C native layer is implemented as a set of C pre-processor native macros. > These macros expand to the appropriated native code. Macros are used > instead function calls to give optimal performance and small code size. > Of course in special cases, a macro can also expand to a function call > if this is needed. This approach provide a flexible and efficient > implementation of the native layer. When this becomes part of Classpath would you mind adding this information to the hacker's guide or as a lesser option adding this README to CVS as well? > The directory and file structure is as follows. > >native > ... > | > |--- target Should this 'target' be called 'macros'? > | | > | |--- Linux > | | |--- target_native_.h > | | |--- ... > | | ... > | |--- ... > | |--- generic > | | |--- target_generic_.h > | | |--- ... > ......... > > > When writing native code, please take care with the following. > > - Use _only_ ANSI C specific functions directly which are available > on all target systems with the same parameters, e. g. strdup() is > not an ANSI C function, thus is is not available on all systems; mkdir() > expect on some systems different parameters. > > !!!Do NOT use this functions in your native code!!! I don't think I have a handy reference for what ANSI C functions are generally available on most systems with the same arguments. Where would I look? > - Avoid _all_ OS specific data types and constants, > e. g. structures or error numbers. Instead, wrap them in a native > macro and convert the values to basic scalar types like char, int, > double or long. I'd like to see how you deal with the 'tm' structure for time/date data. > - Avoid - if possible - non-reentrant functions. Non-reentrant functions > cause strange problems on some multitasking systems. I'm confused about when to worry about this. Does it matter for synchronized methods? Does it matter for all native code? Does ANSI define what is reentrant? That's all my thoughts for now, Brian -- Brian Jones <[EMAIL PROTECTED]> ___ Classpath mailing list [EMAIL PROTECTED] http://mail.gnu.org/mailman/listinfo/classpath
Re: C native layer (pre-announcement)
Hi Brian, I will attempt to answer some of your questions. Dr. Rupp will probably correct any error I make in my answers. I'm confused about when to worry about this. Does it matter for synchronized methods? Yes, as the system/library call you are doing might also be done by *another* method on *another* thread. Does it matter for all native code? Of course. Does ANSI define what is reentrant? Posix does, which is what most of Classpath's native code relies on. Etienne -- Etienne M. Gagnon, Ph.D. http://www.info.uqam.ca/~egagnon/ SableVM: http://www.sablevm.org/ SableCC: http://www.sablecc.org/ ___ Classpath mailing list [EMAIL PROTECTED] http://mail.gnu.org/mailman/listinfo/classpath
Some errors in native layer sources
Dear Classpath members, I checked-in some files with an abstraction layer for native OS functions from our Jamaica VM to be able to port the Classpath C sources to other OS than Unix. This is my first major change in Classpath and there could still be some errors or not fixed things. I will try to repair all broken things soon as possible. If you find something which is not working in your environment please let me know it. Sincerely, Torsten ___ Classpath mailing list [EMAIL PROTECTED] http://mail.gnu.org/mailman/listinfo/classpath
