Re: [drlvm][x86_64] status update
On 11/17/06, Geir Magnusson Jr. [EMAIL PROTECTED] wrote: Yah, I get failed status too. Because the VM segfaults. Try running StackTest via ./java StackTest and what do you see? Oh, reading more... It's output on my SuSEs befor my changes: $ (./build/lnx_em64t_gcc_debug/deploy/jre/bin/java -cp ./build/lnx_em64t_gcc_debug/semis/vm/em/_smoke.tests/c Segmentation fault $ (unset JAVA_HOME; ./build/lnx_em64t_gcc_debug/deploy/jre/bin/java -cp ./build/lnx_em64t_gcc_debug/semis/vm/ $ after em64t fix implementation. $ (./build/lnx_em64t_gcc_debug/deploy/jre/bin/java -cp ./build/lnx_em64t_gcc_debug/semis/vm/em/_smoke.tests/c Segmentation fault $ (unset JAVA_HOME; ./build/lnx_em64t_gcc_debug/deploy/jre/bin/java -cp ./build/lnx_em64t_gcc_debug/semis/vm/ PASS : java.lang.StackOverflowError $ Can you explain that fix? Yes, of cause... partially :) 1) mprotect can't work on main application thread without mmap. Rana found out this fact. I think I can explain it, but it is guess only. I can't find description of it in google... Also on different versions of Linux behavior is different. 2) Rana implemented call of mmap for protected page to call mprotect for it. But it's not enough on some Linux. On my machines sigsegv happened one page before guard page in this case. I suppose that OS check next page status before allocate requested page for the stack. And if next page is already mmapped - OS decides that stack can't grow and sigsegv is happened. Theoretically stack of the thread can grow infinitely especially on EM64T platform. 3) I checked mapped areas for the debugged VM and found, that for others threads, not main thread, whole stack mapped at once. So I tried to map whole stack for main thread in the beginning of the work, at once. And it works. Thanks Pavel Afremov. But gc.Force and others became fail. The source, as I understand, is in following: after mmap of the stack, java method Object.wait() can't works. SuSE 10 hangs up, SuSE 9 makes exit on it I'm just marveling over the fact you got gdb to work. Can anyone else w/ Ubunutu 32-bit or 64-bit debug drlvm in a reasonable way? Gdb shows sigsegv in #0 0x002a961d489d in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/tls/libpthread.so.0 #1 0x002a957a7501 in apr_thread_cond_wait (cond=Variable cond is not available.) at thread_cond.c:68 #2 0x002a957a3e85 in condvar_wait_impl (cond=0x2aaa309778, mutex=0x2aaa309728, ms=0, nano=0, interruptable=1) at /nfs/ims/proj/drl/mrt1/users/pnafremo/work/PBBC_64/drlvm/vm/thread/src/thread_native_condvar.c:69 #3 0x002a957a4463 in monitor_wait_impl (mon_ptr=0x2aaa3096c8, ms=0, nano=0, interruptable=1) at /nfs/ims/proj/drl/mrt1/users/pnafremo/work/PBBC_64/drlvm/vm/thread/src/thread_native_fat_monitor.c:208 #4 0x002a957a652b in thin_monitor_wait_impl (lockword_ptr=0x2a98c24e54, ms=0, nano=0, interruptable=1) at /nfs/ims/proj/drl/mrt1/users/pnafremo/work/PBBC_64/drlvm/vm/thread/src/thread_native_thin_monitor.c:430 #5 0x002a957a65b1 in hythread_thin_monitor_wait_interruptable (lockword_ptr=0x2a98c24e54, ms=0, nano=0) at /nfs/ims/proj/drl/mrt1/users/pnafremo/work/PBBC_64/drlvm/vm/thread/src/thread_native_thin_monitor.c:482 #6 0x002a96b97f15 in jthread_monitor_timed_wait (monitor=0x7fbfffcbc8, millis=0, nanos=0) at /nfs/ims/proj/drl/mrt1/users/pnafremo/work/PBBC_64/drlvm/vm/thread/src/thread_java_monitors.c:337 #7 0x002a96a29a08 in Java_java_lang_VMThreadManager_wait (env=0x594c58, clazz=0x7fbfffcbc0, monitor=0x7fbfffcbc8, millis=0, nanos=0) at /nfs/ims/proj/drl/mrt1/users/pnafremo/work/PBBC_64/drlvm/vm/vmcore/src/kernel_classes/native/java_lang_VMThreadManager.cpp:202 In HARMONY-2224 https://issues.apache.org/jira/browse/HARMONY-2224 I excluded failed tests from acceptance test set: StackTest exception.FinalizerStackTest on EM64T gc.LOS on Windows. I'll go check this out immediately geir BR. Pavel Afremov On 11/17/06, *Geir Magnusson Jr.* [EMAIL PROTECTED] mailto:[EMAIL PROTECTED] wrote: Rana Dasgupta wrote: Not surprising :-) The last big stack relatad checkin in 2018. Its comment notes say that Gregory actually saw the failure of StackTest and the new FinalizeStackTest... So... lets fix them... :) geir On 11/16/06, Geir Magnusson Jr. [EMAIL PROTECTED] mailto:[EMAIL PROTECTED] wrote: First test that fails is the most cherished and beloved StackTest, with a segmentation fault :) I'll try to find some more useful info... geir Geir Magnusson Jr. wrote: We now have DRLVM+Classlib cleanly building out of SVN and able to run basic programs on Ubuntu 6 on an em64T box. $ uname -a : Linux harmony-em64t 2.6.15-27-amd64-generic #1 SMP PREEMPT Sat Sep
Re: [drlvm][em64t] build fails
On 11/15/06, Geir Magnusson Jr. [EMAIL PROTECTED] wrote: How could there be no limit to stack size?? Limit is there but it's too large, like 2 in power 46. Is there a way the test framework could set this? Does DRLVM support -Xss yet? No, DRLVM doesn't support –Xss flag. …yet. I have a question: Is Xss flag significant feature or there are more important things? Pavel Afremov
Re: [drlvm] [testing] Excluding commit tests until the problem is fixed
As I understand Alexey means HARMONY-2073, but not HARMONY-2070. Alexei, is it correct? If not, could you clarify the point about exn_raise_by_name_internal in your initial letter, please? Pavel Afremov. On 11/8/06, Rana Dasgupta [EMAIL PROTECTED] wrote: OK thanks Pavel, I'll try the patch today. Rana On 11/8/06, Pavel Afremov [EMAIL PROTECTED] wrote: Hi Rana. I extend guard region as work around. It's only one way, which fix SOE on my SuSE Linux, without potential regression of your fix. On my Linux machine violation access signals happen one page before protected page on the stack. It's it. I ran all tests, and everything was OK. But strange misprint was fount in the new test. So I attach new fixed patch. Pavel Afremov. On 11/8/06, Rana Dasgupta [EMAIL PROTECTED] wrote: Though I tried several times, I could not repro 2070 or Alexey's specific problems. The test attached to 2018 repros, and that I think is enough. Pavel, 1. The patch looks good, but I could not apply and try it since my Linux box is down. 2. Did you run all tests ( smoke, cuint, kernel, and classlib )? Since this fully turns on lazy exceptions, we need to ensure that all tests pass, or at least have identical behaviour before and after the pacth. 3. Adding a finalizer based stack test to smoke is a good idea. 4. On Linux you extend the guard region up ( or down whatever ) by a page. Did you find a good reason for it, or is this just being careful? Rana On 11/7/06, Pavel Afremov [EMAIL PROTECTED] wrote: Rana, Everything is correct in you description, but it looks like that * HARMONY-2018* https://issues.apache.org/jira/browse/HARMONY-2018 should fix described bug. I think Alexei will have a chance to check it. Thank you. Pavel Afremov.
Re: [drlvm] New regression: java.lang.ClassGenericsTest4
Oh. It's cool fix for my stupid bug.
Thanks for Alexey very much.
Pavel Afremov.
On 11/15/06, Alexey Varlamov [EMAIL PROTECTED] wrote:
Pardon for my English - a bit sleepy already...
2006/11/15, Alexey Varlamov [EMAIL PROTECTED]:
Err, what I found is really trivial bug. But it took quite a few time
to discover - seems today was not my day :(
Index: vm/vmcore/src/exception/exceptions_impl.cpp
===
--- vm/vmcore/src/exception/exceptions_impl.cpp (revision 475132)
+++ vm/vmcore/src/exception/exceptions_impl.cpp (working copy)
@@ -281,7 +281,7 @@
if (NULL != exception-exc_cause) {
tmn_suspend_disable_recursive();
-jthrowable exc_cause = oh_allocate_local_handle();
+exc_cause = oh_allocate_local_handle();
exc_cause-object = exception-exc_cause;
tmn_suspend_enable_recursive();
}
OK, we definitely need a regression test for this.
2006/11/15, Gregory Shimansky [EMAIL PROTECTED]:
Alexey Varlamov wrote:
2006/11/15, Alexey Varlamov [EMAIL PROTECTED]:
2006/11/15, Gregory Shimansky [EMAIL PROTECTED]:
Alexey Varlamov wrote:
The guilty change is the following, which effectively turns on
VM_LAZY_EXCEPTION support in exceptions_impl.cpp:
Well this is a patch from HARMONY-2018 which doesn't hide the
fact that
it enables lazy exceptions. Why shouldn't we enable them?
Gregory,
I've just re-read my posts and couldn't find anything critique or
offending - please don't take regressions too personal. I'm sure we
will be able to fix this one quite soon.
I wasn't offended in any way. I was just thinking that you know some
secret knowledge that lazy exceptions do not work and thus enabling
them
is wrong.
--
Gregory
Re: [drlvm][em64t] build fails
On 11/13/06, Gregory Shimansky [EMAIL PROTECTED] wrote: So what is the point to have a test which would pass either way? Check that it doesn't crash the VM, is it the only purpose for it? I think yes. It should check that test doesn't crash VM if stack size isn't enough. Pasha Afremov.
Re: [drlvm][em64t] build fails
Hello I think that Stack test should print pass if no stack overflow error is happened. Test should check processing of this error but not existing of it. Optimizing compiler can do very deep recursion unrolling, and SOE can happen never in this case. Thanks Pavel Afremov. On 11/13/06, Gregory Shimansky [EMAIL PROTECTED] wrote: Gregory Shimansky wrote: On Saturday 11 November 2006 02:36 Pavel Pervov wrote: Gregory, Could you look at https://issues.apache.org/jira/browse/HARMONY-2152. I believe it'll fix the build. Thank you for a quick fix. The build works now. Don't try to run acceptance tests though. The StackTest is a machine killer. It eats all of the virtual memory in a moment because it cannot find any stack limit (ulimit -s 8192 may be used as a workaround) and maps all of 2^48 (or whatever number of bits are configured in kernel for virtual address space) bytes of virtual memory. After that only reset helps. Ok back to this bug. I decided that on x86_64 linuxes (and ia32 which don't have a stack limit for some reason) we need to at least fix the test so that it doesn't make OS unusable. I've modified StackTest to check firt 10,000,000 recursions and fail is SOE is not thrown. The patch is available in HARMONY-2175. -- Gregory
Re: [drlvm] [testing] Excluding commit tests until the problem is fixed
Hi Rana. I extend guard region as work around. It's only one way, which fix SOE on my SuSE Linux, without potential regression of your fix. On my Linux machine violation access signals happen one page before protected page on the stack. It's it. I ran all tests, and everything was OK. But strange misprint was fount in the new test. So I attach new fixed patch. Pavel Afremov. On 11/8/06, Rana Dasgupta [EMAIL PROTECTED] wrote: Though I tried several times, I could not repro 2070 or Alexey's specific problems. The test attached to 2018 repros, and that I think is enough. Pavel, 1. The patch looks good, but I could not apply and try it since my Linux box is down. 2. Did you run all tests ( smoke, cuint, kernel, and classlib )? Since this fully turns on lazy exceptions, we need to ensure that all tests pass, or at least have identical behaviour before and after the pacth. 3. Adding a finalizer based stack test to smoke is a good idea. 4. On Linux you extend the guard region up ( or down whatever ) by a page. Did you find a good reason for it, or is this just being careful? Rana On 11/7/06, Pavel Afremov [EMAIL PROTECTED] wrote: Rana, Everything is correct in you description, but it looks like that * HARMONY-2018* https://issues.apache.org/jira/browse/HARMONY-2018 should fix described bug. I think Alexei will have a chance to check it. Thank you. Pavel Afremov.
Re: [drlvm] [testing] Excluding commit tests until the problem is fixed
Hello. Could you be so kind to check *HARMONY-2018*https://issues.apache.org/jira/browse/HARMONY-2018 before start fixing and discussing this bug, please? I reported it and provided a fix a week ago. Thanks. Pavel Afremov. On 11/5/06, Fedotov, Alexei A [EMAIL PROTECTED] wrote: Rana, Pavel (Afremov), All, Geir's comment on r443504 (fix for http://issues.apache.org/jira/browse/HARMONY-1363 [drlvm] Java 1.5, 64 bit support, JVMTI improvements) reads: 1) Stack overflow exception stuff is broken. I had to remove the assert in signals_ia32.cpp line 336. Rana knows and will look. I also disabled the StackTest. I have noticed that the patch added a new function exn_raise_by_name_internal which fails on the first invocation checking an assertion about thread state, see http://issues.apache.org/jira/browse/HARMONY-2070 [drlvm][unit] org.apache.harmony.beans.tests.java.beans.PersistenceDelegateTest crashes DRLVM. I also have noticed that this function is called to create java.lang.StackOverflowError. Could you help me to understand the current status of the problem? With best regards, Alexei Fedotov, Intel Java XML Engineering -Original Message- From: Rana Dasgupta [mailto:[EMAIL PROTECTED] Sent: Wednesday, October 18, 2006 4:27 AM To: harmony-dev@incubator.apache.org Subject: Re: [drlvm] [testing] Excluding commit tests until the problem is fixed I fixed the StackOverflow functionality problem by going back and mapping all pages ( guard, alternate stack ) meticulously before trying to protect them. I think we should have done this in the first place. I also cleaned up the previous initialization workarounds and asserts Geir and I had discussed on the JIRA. The Stacktest and all other stack related tests now pass. I'll submit the patch against 1786 in the next few hours after running acceptance tests. Rana On 10/16/06, Rana Dasgupta [EMAIL PROTECTED] wrote: On 10/16/06, Gregory Shimansky [EMAIL PROTECTED] wrote: On Tuesday 17 October 2006 00:01 Geir Magnusson Jr. wrote: I tried to put some back. StackTest still doesn't work. It's hard to believe... so I gave up and just kept going :) I wonder if the test or the implementation are wrong. Maybe someone who added the test initially could know the answer. There is nothing wrong with the stacktest test itself. The implementation is not quite 100%complete( I think ), but has enough functionality and the test passes on Windows. On Linux, it fails. I am not sure if this is a regression, or if this ever worked. There is a JIRA issue 1786. In summary, memory protection setup for the guard page fails on the main thread(only). So the guard does not work and the overflow is not detected. mprotect fails with an ENOMEM which is either a mapping failure or a kernel failure. mprotect() has some known flakiness it seems, as per literature. The basic implementation on Linux is sound. There are secondary design issues,but we can only get to them later after we have figured out why the guard setup fails on the main thread.
Re: [drlvm] [testing] Excluding commit tests until the problem is fixed
Rana, Everything is correct in you description, but it looks like that * HARMONY-2018* https://issues.apache.org/jira/browse/HARMONY-2018 should fix described bug. I think Alexei will have a chance to check it. Thank you. Pavel Afremov. On 11/6/06, Rana Dasgupta [EMAIL PROTECTED] wrote: Alexei, The1363 submission added functionality for lazy exception support, for exceptions in the VM code. exn_raise_by_name_internal was such a function. This may not be bug free ( as is true of any code ) and we need to check out 2070. I will take a look, as should Pavel. I think that while committing 1363, StackTest failed and there were other asserts, which Geir disabled to not block the large submission. This is not really anything to do with lazy exceptions. This is an issue where parts of the main thread's stack virtual memory are unmapped on Linux. I later added a fix for this. The two issues are orthogonal to each other. Rana On 11/5/06, Fedotov, Alexei A [EMAIL PROTECTED] wrote: Rana, Pavel (Afremov), All, Geir's comment on r443504 (fix for http://issues.apache.org/jira/browse/HARMONY-1363 [drlvm] Java 1.5, 64 bit support, JVMTI improvements) reads: 1) Stack overflow exception stuff is broken. I had to remove the assert in signals_ia32.cpp line 336. Rana knows and will look. I also disabled the StackTest. I have noticed that the patch added a new function exn_raise_by_name_internal which fails on the first invocation checking an assertion about thread state, see http://issues.apache.org/jira/browse/HARMONY-2070 [drlvm][unit] org.apache.harmony.beans.tests.java.beans.PersistenceDelegateTest crashes DRLVM. I also have noticed that this function is called to create java.lang.StackOverflowError. Could you help me to understand the current status of the problem? With best regards, Alexei Fedotov, Intel Java XML Engineering -Original Message- From: Rana Dasgupta [mailto:[EMAIL PROTECTED] Sent: Wednesday, October 18, 2006 4:27 AM To: harmony-dev@incubator.apache.org Subject: Re: [drlvm] [testing] Excluding commit tests until the problem is fixed I fixed the StackOverflow functionality problem by going back and mapping all pages ( guard, alternate stack ) meticulously before trying to protect them. I think we should have done this in the first place. I also cleaned up the previous initialization workarounds and asserts Geir and I had discussed on the JIRA. The Stacktest and all other stack related tests now pass. I'll submit the patch against 1786 in the next few hours after running acceptance tests. Rana On 10/16/06, Rana Dasgupta [EMAIL PROTECTED] wrote: On 10/16/06, Gregory Shimansky [EMAIL PROTECTED] wrote: On Tuesday 17 October 2006 00:01 Geir Magnusson Jr. wrote: I tried to put some back. StackTest still doesn't work. It's hard to believe... so I gave up and just kept going :) I wonder if the test or the implementation are wrong. Maybe someone who added the test initially could know the answer. There is nothing wrong with the stacktest test itself. The implementation is not quite 100%complete( I think ), but has enough functionality and the test passes on Windows. On Linux, it fails. I am not sure if this is a regression, or if this ever worked. There is a JIRA issue 1786. In summary, memory protection setup for the guard page fails on the main thread(only). So the guard does not work and the overflow is not detected. mprotect fails with an ENOMEM which is either a mapping failure or a kernel failure. mprotect() has some known flakiness it seems, as per literature. The basic implementation on Linux is sound. There are secondary design issues,but we can only get to them later after we have figured out why the guard setup fails on the main thread.
Re: [dlrvm] ClassCircularityError in recursive compilation (Was: Re: [drlvm] smoke test : gc PhantomReferenceQueueTest is really unstable)
No. It couldn't. I don't now any solution which can do it. Pavel Afremov. On 10/25/06, Mikhail Fursov [EMAIL PROTECTED] wrote: On 10/25/06, Pavel Afremov [EMAIL PROTECTED] wrote: Your fix just switch off Finalization Work Balance Subsystem Pavel, could Work Balance Subsystem be implemented in finilizers threads directly. That is we will not have Java code executed from helpers? -- Mikhail Fursov
Re: [dlrvm] ClassCircularityError in recursive compilation (Was: Re: [drlvm] smoke test : gc PhantomReferenceQueueTest is really unstable)
Good idea Mikhail! I think I will base on it in my future solution. I see only one negative side for it. Quantity of the treads increased by one in normal situation. Any ideas? Thanks. Pavel Afremov. On 10/25/06, Mikhail Fursov [EMAIL PROTECTED] wrote: The Work Balance Subsystem task is to start new finalizing threads when all active threads are busy, isn't it? The solution could be: 1) Add +1 thread: finalizers manager 2) notify this thread (as Salikh proposed) to start finalization and do the Work Balance Subsystem job. Does it work or am I missing something? On 10/25/06, Pavel Afremov [EMAIL PROTECTED] wrote: No. It couldn't. I don't now any solution which can do it. Pavel Afremov. On 10/25/06, Mikhail Fursov [EMAIL PROTECTED] wrote: On 10/25/06, Pavel Afremov [EMAIL PROTECTED] wrote: Your fix just switch off Finalization Work Balance Subsystem Pavel, could Work Balance Subsystem be implemented in finilizers threads directly. That is we will not have Java code executed from helpers? -- Mikhail Fursov -- Mikhail Fursov
Re: [dlrvm] ClassCircularityError in recursive compilation (Was: Re: [drlvm] smoke test : gc PhantomReferenceQueueTest is really unstable)
Salikh, gc.Finalizer and gc PhantomReferenceQueueTest are synthetic test. And can't be called normal (do code review, if you doubt, please). But this test isn't source of the circularity error. The error can happen for usual application too. The CL an SM tests just show that source of fake circularity error exists. It's it. I think, this fake error can happen in usual applications. But it can be masked here or be covered by other issues... Pavel Afremov On 10/24/06, Salikh Zakirov [EMAIL PROTECTED] wrote: Gregory Shimansky wrote: Yes the test is synthetic. But the whole problem arose from the currently excluded smoke tests gc.PhantomReferenceQueueTest and gc.Finalizer. They too are synthetic but may be an example of another place where this problem appears. The idea to write user code tests was just to show that finalizers implementation is not the one to be blamed and the problem is reproducible on pure user code applications, not to show that this problem is so rare no one in right mind would ever write an application to encounter it :) Gregory, the problem as I see it is that gc.Finalizer and gc.PhantomReferenceQueueTest are *normal* test cases in the sense that they exercise finalizable objects and phantom references in *exactly* the same way as an application would. And Pavel's synthetic tests (HARMONY-1945) do something allowed by the spec, but entirely uncommon for an application -- recursive call from a classloader to the application.
Re: [dlrvm] ClassCircularityError in recursive compilation (Was: Re: [drlvm] smoke test : gc PhantomReferenceQueueTest is really unstable)
Hi I've developed two "impossible tests, which shows "fake" circularity errors. One test is more simple and use SecurityManager. The other is a bit more complex and usescustom ClassLoader. You can findthem in attachment. Thanks. Pavel Afremov On 10/17/06, Pavel Pervov [EMAIL PROTECTED] wrote: The scenario I described earlier is impossible. Resolution of any classreferenced in some other class is performed by class loader, which loaded that other class. So, no chance to load A and referencing class loader(UCL) with this UCL.Sorry for confusion.Regards, Pavel.P.S. Still there are concerns why lazy resolution should be supported by JITs. But it is absolutely another story.On 10/17/06, Mikhail Fursov [EMAIL PROTECTED] wrote: IMO we shall be between BEA and SUN: to work if both RI work, to fail if both RI fail and discuss each test in details if only one RI passes. On 10/17/06, Gregory Shimansky [EMAIL PROTECTED] wrote: On Friday 13 October 2006 08:04 Alexey Varlamov wrote: I'm curious, if we enable controlled recursion in classloading - will it resolve this kind of issues completely? I'm pretty sure it would resolve at least some scenarios - like the one Pavel described for gc.Finalizers or a case of classloading initiated within SecurityManager.checkPermission() which we also faced not once. Controlled recursion here means counting depth of recursion and allowing at least 1 recursive iteration. I've seen some tricks in URLClassLoader which assume such ability, but they do not work with DRLVM. I think it is different. URLClassLoader is system class which is loaded by bootstrap, so no recursion happens for classes which it itself requires to be loaded when it is being compiled.For the pure user code scenario Pavel suggested above, there may be some nuances leading to truly endless recursion, but still we need to look at particular test first. It is not endless but it is definitely more than 1 level deep. If user sets up his own class loaders, compiling them may trigger loading some of the user classes which are in turn loaded by class loaders set up by user. Shall we then throw NoClassDefFoundError: Class loading recursion limit reached. Please rewrite your code? :) -- Gregory Shimansky, Intel Middleware Products Division - Terms of use : http://incubator.apache.org/harmony/mailing.html To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] -- Mikhail Fursov
Re: [drlvm] Calling native methods from Java code: implementation details
Hi.
On First question I think that alternative 1 is better. It's allow to avoid
creating of special or magic registration subsystem which should
provide addresses for magic native call.
On the Second question I agree with Pavel P. Null should be OK.
On Third question, I think no additional actions is required. All dll's or
so's for DRL VM must be in jre/bin directory and classes jar's must be
jre/lib directory. Both should be loaded by as the others in these
directories now. It's the first part of third question. The second part
or fourth question is How to understand that this class contains magic
native functions?. I prefer answer by extends a special empty interface.
It's can provide opportunity writing libraries with fast native calls in
future. Also it's match with java security. DRL VM can control what
component can use Magic and what not via SecurityManager.
Mikhail could you summarize all issues which should be clarified and
possible solutions for these issues?
Thanks.
Pavel Afremov.
On 10/20/06, Pavel Pervov [EMAIL PROTECTED] wrote:
Prerequisite: each component carries its own helpers (which means no
allocation helper in VM Core, for example).
Yes, simple JIT will call helper as regular JNI method.
I missed several issues in my original message.
First: how to get address of helper?
Answer:
Two choices:
1) each component must provide static native long getHelperAddress(int)
in
its helper class. JIT will make a call to this function as regular JNICALL
(stdcall on Windows, cdecl on Linux). Function prototype can be typedef'ed
as typedef (JNICALL*)(JNIEnv*, jint). The issue I see here is naming:
each
component has its own helpers class name. So, the name for native method
to
call will be package name_class name_getHelperAddress.
2) add regular (OPEN) interface function to each component which will
provide such mapping (as it is now done by VM).
Second: what should be passed as JNIEnv to getHelperAddress?
Answer:
NULL. Component should not use JNIEnv pointer in this call.
Third: how to retrieve helpers class for a particular component?
Answer:
Two choices again:
1) component modifies boot class path on initialization and exports
component type.helpers_class system property. Then VM loads this class
and
JIT uses this class' name to retrieve fast path helpers.
2) component exports component type.helpers_jar system property on
initialization, then VM adds this JAR to boot class path, extracts the
manifest from it, retrieve Helpers-Class attribute from the manifest, and
exports it as component type.helpers_class. Then VM loads this class and
JIT uses this class' name to retrieve fast path helpers.
I, personally, prefer second choice here.
Anything else?..
Pavel.
On 10/20/06, Alex Astapchuk [EMAIL PROTECTED] wrote:
Pavel Pervov :
My alternative follows:
1) Each component which exports magics, provides JAR file with class
implementing these magics. Class name is described in (e.g.)
Main-Class
attribute of JAR's manifest.
2) The class contains java fast path for helper, and a pair of native
methods: one for fast-slow path, and one for slow-slow path.
3) Fast-slow path helper can have any (supported) calling convention.
4) Slow-slow path helper will be called through JNI - it is regular
native
method.
...
way and generate call to slow_alloc in annotated calling convention.
If JIT
does not support magics at all, it'll call jni_alloc in usual way - as
regular call to native Java method.
Helpers are neither called as JNI methods nor they are called through
the JNI.
Or do you propose to make a code for the helpers as the JNI methods?
How would the 'jni_alloc(Address class, int size);' look like in the C
code?
--
Thanks,
Alex
The example follows:
--
package org.apache.harmony.vm.helpers;
// Annotations follow here, which describe which method is which
helper,
// and which is fast-path, fast-slow path, slow-slow path
class GCv6Helpers {
// fast-path allocation here
public static Address alloc(Address class, int size) {
// ...
}
// annotation, describing calling convention for fast-slow path
public static native Address slow_alloc(Address class, int size);
public static native Address jni_alloc(Address class, int size);
}
Now, if JIT supports magics, it'll process GCv6Magics::alloc in a
special
way and generate call to slow_alloc in annotated calling convention.
If
JIT
does not support magics at all, it'll call jni_alloc in usual way - as
regular call to native Java method.
Yes, my idea suggests to change helper retrieval interface for JIT,
but
we
already started to make significant changes to architecture - let's
just
make them more generic.
Regards,
Pavel.
P.S. IMHO, this approach combines strong sides of Mikhail's and
Pavel's
ideas, as it use standard mechanisms of resolution and search of
methods
in
loaded classes
Re: [vote] Graduate Apache Harmony podling from the Incubator
+1 Pavel Afremov. On 10/21/06, Weldon Washburn [EMAIL PROTECTED] wrote: +1 On 10/20/06, Geir Magnusson Jr. [EMAIL PROTECTED] wrote: We're trying something a little different. I think Roy Fielding one said something along the lines of when a community gets organized enough to vote itself out of the Incubator, it's appropriate. So to bring the Harmony community and the Incubator community together for this important event in Harmony's life, I'm calling for a vote on graduating Harmony here on our own -dev list. The objective is for all in both the Harmony community and the Incubator community that have an opinion to vote together, in the same place, and have it hosted by the Harmony podling. This is an unconventional way to do this, but I think that it's valid and could provide one example to the Incubator on how it can work going forward. So, without any further ado : [ ] +1 Graduate Apache Harmony from incubation, and let it petition the board for Top Level Project status [ ] 0 No opinion [ ] -1 No, don't graduate Harmony. Here's why : This vote will end 72 hours from now + time of Apache mail outage. It will therefore end on Monday, October 23, at 3:30 PM eastern, + delta for mail outage. Thanks geir - Terms of use : http://incubator.apache.org/harmony/mailing.html To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] -- Weldon Washburn Intel Middleware Products Division
Re: [drlvm] Calling native methods from Java code: implementation details
It's better to use standard JNI call, to call libraries from the side. It's
provide creation of m2n and other required stuff. The other case if the
library is ours and we know that there are methods which can be called via
fast way. In this case we can do following
For example there is
class MyLibraryClass {
int methodA();
void methodB(int);
}
Wrapper should be added
.
Class MyFastLibraryClass extends MyLibraryClass {
CallAddress getCallAddress(String methodName); // which returns fast
call addresses.
}
BR
Pavel Afremov
On 10/18/06, Xiao-Feng Li [EMAIL PROTECTED] wrote:
Hi, I think Pavel and Mikhail's proposals are essentially the same.
Pavel's looks like more flexible, but I have a question with the
special interface approach: is it possible that sometimes we want to
call a library native method in fast way? If possible, shall we
require all library classes that have the native method implement the
special interface?
Or, is it possible for the VM developer to override a library native
method with a fast internal replacement that still keeps the
semantics? In this situation, this method is not really implemented in
the library class. Where will JIT find it?
And we probably don't want over-generality to the design of fast
native method invocation. We'd better keep them internally to the VM
at the moment until we see real need to open this contract interface
to external library/application developers. So if you agree with this,
the set of non-JNI native methods for JIT to handle are very limited
-- they are actually just some JVM internal functions (plus some
special cases for speedup). We probably want them more closed than
open just like the inlined helpers.
Thanks,
xiaofeng
On 10/18/06, Pavel Afremov [EMAIL PROTECTED] wrote:
Mikhail proposal is OK. But I think that following solution is better
1) The special interface MagicNativeCall should be created. The
interface has only one method CallAddress getCallAddress(String
methodName).
All native methods of classes which implement this interface should be
called by JIT via special magic way.
2) If JIT find call of native method, it tries to find call
address
using getCallAddress method. If null is returned by the method, usual
call
of JNI method should be processed. In other case magic native call can
be
included instead JNI call. Result returned by getCallAddress can
and should
be cashed by JIT for performance reasons.
3) I propose for the firs time use fastcall calling convention. In
future new additional method can be added MagicNativeCall interface. It
can
be named as getCallType for example. If it returns JNI – jit should use
stdcall, if FastWay – fast call. Also we can add special Magic type of
calling convention. In this type of call one can select Register and
stack
slot for every parameter and return value.
It can looks like:
ReturnValue methodName(InParameterList ) or
void methodName(InParameterList, OutParameterList).
ReturValue is an void or OutValue, InParameterList is an empty or
separated
by comma , list of the InValue, OutParameterList is an empty or
separated
by comma , list of the OutValue.
The OutValue can by one of OutRegisterParametr## or OutStackParametr##,
where ## number of register or call in the stack, which are possible on
the
current platform.
The InValue can be one of InRegisterParametr## or InStackParametr##.
Register## and Stack## is the short aliases for InRegisterParametr## and
InStackParametr## correspondently. Result is the short alias for
OutRegisterParametr00 (EAX on ia32).
Generics can be used for type safety.
Thanks.
Pavel Afremov.
On 10/17/06, Rana Dasgupta [EMAIL PROTECTED] wrote:
Mikhail,
All this looks reasonable to me. At least to go ahead. Regarding 2A,
could the jit cache this information for re-use?
Alternatively, the JIT can do all this at startup...by going thru the
contract class of fastpath java methods and querying the component
manager
for the native addresses of the slow counterparts.
Thanks,
Rana
On 10/17/06, Mikhail Fursov [EMAIL PROTECTED] wrote:
All,
Finally we have almost everything finished to post helper's
fast-path
inlining framework into JIRA.
The issue is left is how to call native slow-path versions of the
from
Java
code. We already discussed some of the aspects, but there was no
detailed
discussion with a final agreement what API we will use.
Let's make a final decision so I can add the code into JIRA.
I'm sending my vision of the solution. Correct me or advise another
approach.
Step 1: How JIT will know if a Java method must be replaced with a
native
helper call.
Solution 1.A: Every Java method that must be replaced with native
call
must
be a static method and must have special Native runtime method
annotation.
Step 2: How JIT will get the address of the native method
Re: [drlvm] Calling native methods from Java code: implementation details
Hello. My answers is here: a) About resolving during compilation time If Jit compiles call of the method from the current class, it means that resolving has been already done, and not additional resolving is not required to understand is class implements magic interface or not. If Jit compiles call of the method from other class it should resolve that class to understand is it usual call or JNI, as I understand. So no additional work again. b) About call of java method during compilation time. Jit now make class loading during compilation. It means that Jit make call of java method. So I don't think that it's the other case. To provide work of getCallAddress without class initializing, additional requirements can be put forward. If Jit will implements lazy resolution in future, compilation of the method call and getCallAddress invoking will be put into lazy resolution stub. c) Use of special interface is right for security issues. It's not clear for me how provide security in Mikhail scheme. Thanks Pavel Afremov. On 10/18/06, Mikhail Fursov [EMAIL PROTECTED] wrote: Pavel, I completely agree with Alex critics. But the thread is to discuss and to find the best solution that could be implemented fast. Sorry, some questions I have are the same as Alex's. On 10/18/06, Pavel Afremov [EMAIL PROTECTED] wrote: 1) The special interface MagicNativeCall should be created. The interface has only one method CallAddress getCallAddress(String methodName). All native methods of classes which implement this interface should be called by JIT via special magic way. a. Object instance? Where should I get it during the compilation? b. Why to use marker interface if we can use annotations to provide all knowledge we need in the same way (e.g. calling convention/if native/other knowledge that may be needed)? 2) If JIT find call of native method, it tries to find call address using getCallAddress method. If null is returned by the method, usual call of JNI method should be processed. In other case magic native call can be included instead JNI call. Result returned by getCallAddress can and should be cashed by JIT for performance reasons. a. Self-describing Java code (I mean getCallAddress()) - this is very good. b. Calling virtual method from the JIT during the compilation is not a good idea. 3) I propose for the firs time use fastcall calling convention. In future new additional method can be added MagicNativeCall interface. It can be named as getCallType for example. If it returns JNI – jit should use stdcall, if FastWay – fast call. Also we can add special Magic type of calling convention. In this type of call one can select Register and stack slot for every parameter and return value. a. The development is limited to support only basic CCs - I agree. All other calling convention could be discussed latter. b. Why to use abstract lists but not the named and typed parameters like in Java? -- Mikhail Fursov
Re: [drlvm] Calling native methods from Java code: implementation details
Mikhail proposal is OK. But I think that following solution is better 1) The special interface MagicNativeCall should be created. The interface has only one method CallAddress getCallAddress(String methodName). All native methods of classes which implement this interface should be called by JIT via special magic way. 2) If JIT find call of native method, it tries to find call address using getCallAddress method. If null is returned by the method, usual call of JNI method should be processed. In other case magic native call can be included instead JNI call. Result returned by getCallAddress can and should be cashed by JIT for performance reasons. 3) I propose for the firs time use fastcall calling convention. In future new additional method can be added MagicNativeCall interface. It can be named as getCallType for example. If it returns JNI – jit should use stdcall, if FastWay – fast call. Also we can add special Magic type of calling convention. In this type of call one can select Register and stack slot for every parameter and return value. It can looks like: ReturnValue methodName(InParameterList ) or void methodName(InParameterList, OutParameterList). ReturValue is an void or OutValue, InParameterList is an empty or separated by comma , list of the InValue, OutParameterList is an empty or separated by comma , list of the OutValue. The OutValue can by one of OutRegisterParametr## or OutStackParametr##, where ## number of register or call in the stack, which are possible on the current platform. The InValue can be one of InRegisterParametr## or InStackParametr##. Register## and Stack## is the short aliases for InRegisterParametr## and InStackParametr## correspondently. Result is the short alias for OutRegisterParametr00 (EAX on ia32). Generics can be used for type safety. Thanks. Pavel Afremov. On 10/17/06, Rana Dasgupta [EMAIL PROTECTED] wrote: Mikhail, All this looks reasonable to me. At least to go ahead. Regarding 2A, could the jit cache this information for re-use? Alternatively, the JIT can do all this at startup...by going thru the contract class of fastpath java methods and querying the component manager for the native addresses of the slow counterparts. Thanks, Rana On 10/17/06, Mikhail Fursov [EMAIL PROTECTED] wrote: All, Finally we have almost everything finished to post helper's fast-path inlining framework into JIRA. The issue is left is how to call native slow-path versions of the from Java code. We already discussed some of the aspects, but there was no detailed discussion with a final agreement what API we will use. Let's make a final decision so I can add the code into JIRA. I'm sending my vision of the solution. Correct me or advise another approach. Step 1: How JIT will know if a Java method must be replaced with a native helper call. Solution 1.A: Every Java method that must be replaced with native call must be a static method and must have special Native runtime method annotation. Step 2: How JIT will get the address of the native method to generate a direct call? Solution 2.A: Every Java method that must be replaced with native call must have special Component annotation. JIT will use this annotation to ask Component Manager to access to the specified component by it's name and call void* getAddress(const char* methodName) component's method to get the address of a helper. That is every component that have native calls from Java must provide this interface. Step 3: How JIT will know which calling convention to use? Solution 3.A: Use method's annotation again. These were all of the problems with native calls. Now we need to agree if the solution proposed is OK or find another one. Please note, that this is just the first implementation. We can extend it with more features in the future. -- Mikhail Fursov
Re: [drlvm] gc.Finalizer smoke test fails more often than not
I' m going look into this. Pavel Afremov. On 10/16/06, Geir Magnusson Jr. [EMAIL PROTECTED] wrote: Well, it just seems to have gotten worse lately Anyone working on a fix? geir Ivan Volosyuk wrote: I have created a JIRA issue for this problem some time ago. https://issues.apache.org/jira/browse/HARMONY-1556 It contains the description of what is going on. -- Ivan On 10/16/06, Geir Magnusson Jr [EMAIL PROTECTED] wrote: Usually with this : java: /home/geir/dev/apache/harmony/enhanced/trunk/working_vm/vm/vmcore/src/class_support/classloader.cpp:551: void ClassLoader::SuccessLoadingClass(const String*): Assertion `lc' failed. SIGABRT in VM code. Stack trace: addr2line: '[vdso]': No such file 1: ?? (�U��h���יڶ:-1) 2: abort (??:-1) 3: __assert_fail (??:-1) 4: ClassLoader::SuccessLoadingClass(String const*) (/home/geir/dev/apache/harmony/enhanced/trunk/working_vm/vm/vmcore/src/class_support/classloader.cpp:552) 5: ClassLoader::DefineClass(Global_Env*, char const*, unsigned char*, unsigned int, unsigned int, String const**) (/home/geir/dev/apache/harmony/enhanced/trunk/working_vm/vm/vmcore/src/class_support/classloader.cpp:346) 6: BootstrapClassLoader::LoadFromJarFile(JarFile*, char const*, String const*, bool*) (/home/geir/dev/apache/harmony/enhanced/trunk/working_vm/vm/vmcore/src/class_support/classloader.cpp:2008) 7: BootstrapClassLoader::LoadFromFile(String const*) (/home/geir/dev/apache/harmony/enhanced/trunk/working_vm/vm/vmcore/src/class_support/classloader.cpp:1911) 8: BootstrapClassLoader::LoadClass(Global_Env*, String const*) (/home/geir/dev/apache/harmony/enhanced/trunk/working_vm/vm/vmcore/src/class_support/classloader.cpp:1702) 9: ClassLoader::LoadVerifyAndPrepareClass(Global_Env*, String const*) (/home/geir/dev/apache/harmony/enhanced/trunk/working_vm/vm/vmcore/src/class_support/classloader.cpp:385) 10: _resolve_class (/home/geir/dev/apache/harmony/enhanced/trunk/working_vm/vm/vmcore/src/class_support/Resolve.cpp:157) 11: _resolve_class_new(Global_Env*, Class*, unsigned int) (/home/geir/dev/apache/harmony/enhanced/trunk/working_vm/vm/vmcore/src/class_support/Resolve.cpp:221) 12: resolve_class_new (/home/geir/dev/apache/harmony/enhanced/trunk/working_vm/vm/vmcore/src/class_support/Resolve.cpp:902) 13: vm_resolve_class_new (/home/geir/dev/apache/harmony/enhanced/trunk/working_vm/vm/vmcore/src/class_support/C_Interface.cpp:903) 14: Jitrino::Jet::Compiler::gen_magic() (??:-1) - Terms of use : http://incubator.apache.org/harmony/mailing.html To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Stack Overflow Error support issues
Hi The main issue is that new object can't be created in suspend disabled mode. DRL VM is in this state during GC unsafe operation, when GC have not been started. New SOE can't be created in this mode. But all information about exception can be stored in thread local storage. When Vm return control to managed code, function rethrow_current_thread_exception is called, and this function can decide to create exception (it's possible here) or throw it lazy. Now lazy exception supported for MANAGED code only, Alexey propose extend it for VM code. This technique should fix the most case when exception should be raised in suspend disabled mode. About suspended mode. It's misprint for suspend disabled mode. Sorry for confusion. The issue is in check available stack size before entering in suspend disabled mode, and raise new SOE if available stack size is not enough. The third point is not really fix. I think it's workaround for cases when VM can't create new exception object by different reasons. I suppose, VM can raise pre created SOE in the case when stack overflow happen in suspend disabled mode and stack can't be unwound destructively. Pavel Afremov.
Re: [vote] acceptance of HARMONY-856 : assorted bug fixed for DRLVM
+1 Pavel Afremov On 7/23/06, Geir Magnusson Jr [EMAIL PROTECTED] wrote: All is in order and in SVN for Harmony-856 wrt BCC and ACQ. Please vote to accept or reject this codebase into the Apache Harmony class library : [ ] + 1 Accept [ ] -1 Reject (provide reason below) Lets let this run a minimum of 3 days unless a) someone states they need more time or b) we get all committer votes before then. geir - Terms of use : http://incubator.apache.org/harmony/mailing.html To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Stack Overflow Error support issues
Hello Sorry for late replay. It was very nice weekend for me :) On 7/21/06, Rana Dasgupta [EMAIL PROTECTED] wrote: - I could not get the unwind failure that you have mentioned (with the overflow happening in the first two lines) though I played around with the test, but that may depend on what the specific setup of the stack is at that point? I rebase and recompile my version of DRLVM. And I can't reproduce it too. On my previous version some magic was required to reproduce it. I added several variables into main functions and into function func. This magic was different for different build and should be tuned after a small changes in VM code. Now I can't reproduce it using this magic. ... - But I also saw that you fail the JIT if you don't have 256 K of free stack space. The default Windows stack size is only 1 MB. Do we need to fail a compile of 10 lines of bytecode if we don't have 1/4 of the stack available? Maybe this can be less strict, or some heuristic based on method size? What do you think about this? Yes it's 1/4 of the stack. But in my test current implementation provide recursion with depth = 704220. Is more deep reqursion required anywhere? I think its very unlikely . In any case if more deep recursion will be required in the future, we can tune thread stack size for windows build. Thanks. Pavel Afremov
Re: Stack Overflow Error support issues
Hi On 7/22/06, Mikhail Fursov [EMAIL PROTECTED] wrote: I think this must be a JIT heuristics because even a small method can lead to inlining of whole classlib API :) Are You think this check should be removed from VM and puted into JIT only? BR Pavel Afremov.
Re: Stack Overflow Error support issues
Hi Weldon. There is a simple test in description of the *HARMONY-945*https://issues.apache.org/jira/browse/HARMONY-945 . I can add this test in the smoke tests of DRLVM. Is it OK? Pavel Afremov.
Re: Stack Overflow Error support issues
On 7/24/06, Mikhail Fursov [EMAIL PROTECTED] wrote: No, I think that VM can do this check but use lower border: e.g. 1/100 of initial. JIT must do this check more accurate: use knowledge of algorithms it uses. ... I think we can review this issue when the check appears in the JIT. Pavel Afremov.
Re: Stack Overflow Error support issues
Because more elegant decision wasn't proposed during current discussion, I'd like to put the patch with results of my experiments into JIRA, as Stack Overflow Implementation. You can find it in *HARMONY-945*https://issues.apache.org/jira/browse/HARMONY-945. Welcome to try it. Pavel Afremov.
Re: Stack Overflow Error support issues
Mikhail. During my experiments I already added two functions into DRLVM: size_t get_available_stack_size()and bool check_available_stack_size(size_t required_size). The first is used in the second to make required check and the second used to check, that there are 256 Kbytes of free stack, before starting compilation. So I can add both this functions into open interface and put a patch into JIRA. Will this solution be OK for all components? 2. If StackOverflowError is thrown during the first two commands of compiled method, function unwind of the JIT cannot always unwind frame correctly. What do you think about this issue? Thanks. Pavel Afremov.
Re: Stack Overflow Error support issues
You can see my comments inline. On 6/28/06, Weldon Washburn [EMAIL PROTECTED] wrote: Some comments and observations: 1) I like Pavel's approach of trying jitrino.JET. Is it possible to put constraints on the max amount of hardware stack that JET uses? Is it possible to calculate the max amount of stack that JET could use? If these numbers are knowable, it could help in the design. I think it's impossible to find max amount of stack that is required for complex components, like JIT. But it's possible to put constraints for some functions, which does simple compilation tasks. 2) Stack overflow in the VM is very much analogous to what I remember from old Linux/Windows kernels. A kernel stack overflow causes a crash. Thus the kernel is specifically designed to avoid kernel stack overflow. Other than the jit, are there other places in the VM that do deep recursion? If true, where are they and can they be redesigned to not abuse the stack? The other known for me component is verifier. I think it can be redesigned to add SOE support. But I'm not sure on 100% now. Can anybody investigate it? 3) One approach might be to put the JIT in a separate vm thread that never executes any code other than the JIT. Thus the only frames on the stack are due to the JIT itself. This might make recovery after overflow simpler. For example, let the JIT hit the guard page at the top of its stack and throw a hardware exception. The exception would be caught by the JVM itself -- no Java execption would ever be involved. Cleanup would involve free()'ing the stuff the jit malloc()'ed, unwinding JVM locks the JIT held, closing files, releasing class loader resources, etc. Once the cleanup is done, return an error to Java code and let the java code run its course. It's interesting idea. BR Pavel Afremov.
Stack Overflow Error support issues
Hello.
As far as I know, current implementation of DRL VM doesn't support Stack
Overflow Error (SOE).
I have found it out using following test:
public class Stack {
static int depth = 0;
public static void func() {
depth++;
func();
}
public static void main(String[] args) {
try {
func();
} catch (Throwable th) {
System.out.println(First SOE depth = + depth);
System.out.println(Caught = + th);
}
}
}
I'm experimenting to add Stack Overflow Error (SOE) support using protected
memory page on the stack. I see and use two main schemes of exception
processing.
1. If top frame is unwindable (regular java code), then signal handler or
vectored exception handler throws regular java exception, as it can happen
now for NullPointerException.
2. If top frame is non-unwindable (for example, JNI native code) VM sets
exceptions for the current thread and continues its execution from
interrupted place. A code which works in nonunwindable mode should
periodically check that no exception is raised.
During my experiments I found some problems in current VM implementation
(including JIT)
1. Some parts of VM which use long recursion calls in non-unwindable mode
(JIT compiler, verifier) don't check that StackOverflowError has occured. I
added check that there are 256 Kbytes of free stack, before starting
compilation. But I'm afraid it is not enough sometimes.
2. If StackOverflowError is thrown during the first two commands of compiled
method, function unwind of the JIT cannot always unwind frame correctly.
Are there any ideas how to fix them?
I have some code developed locally, and can submit it to JIRA if anyone is
interested in trying it.
Thanks.
Pavel Afremov.
