Re: Issue with BuiltinClassLoader.ucp field not being visible

[Luke Hutchison]

On Tue, Aug 10, 2021 at 1:34 AM Luke Hutchison  wrote:

> Maybe that should be part of the Instrumentation class, so that it is only
> accessible to Java agent code that is able to get a reference to an
> Instrumentation object.
>

Actually I take that last part back, that would not be very helpful to
libraries like ClassGraph. It would be much better if there were simply a
public method in `AppClassLoader` that could be used to get all classpath
elements that have been added for instrumentation.

Re: Issue with BuiltinClassLoader.ucp field not being visible

[Luke Hutchison]

On Tue, Aug 10, 2021 at 12:51 AM Luke Hutchison 
wrote:

> Could a getURLClassPath() method please be added to BuiltInClassLoader?
>

For security reasons, the getURLClassPath() method should return a copy of
ucp, not the reference to ucp, to prevent the caller from modifying the
system classpath.

Re: Calling a lambda expression from a new thread before the main method is run causes the thread to lock up

[Luke Hutchison]

Thank you Remi and David for your detailed explanations, this was very
helpful, and I understand the issues now. Please go ahead and close the
Jira bug I created for this (JDK-8173252), I see this is WAI.

On Tue, Jan 24, 2017 at 12:39 AM,  wrote:

> Here is your code slightly modified, with no lambda, that deadlock too:


Thanks, I discovered that I hit a deadlock with anonymous inner classes
when I access fields in the enclosing class too.

On Tue, Jan 24, 2017 at 1:39 AM, David Holmes 
 wrote:

> I have to second Remi's view here - hidden concurrency is an accident
> waiting to happen, far too many things can go wrong if the users of your
> API don't know that new threads can be involved.


FastClasspathScanner only uses concurrency for its own internal processing.
The standard API that most users call is synchronous, and blocks on
response. The user-provided lambdas in question are run single-threaded on
a separate thread, but it does not execute concurrently with the main
thread (which is blocked), precisely to prevent surprises. If the user
calls the async API, they should understand about concurrency. I agree that
in general the burden falls to the user to read the docs and understand the
threading considerations of a given library.

In the end I decided to throw an exception if there is a "" line in
the stacktrace. This is better than the possibility of hitting a deadlock.
The following makes me think though that I should refactor my code to
ensure that, for the synchronous API, user-supplied methods are always
executed on the calling thread:

If the lambda code is called on the main thread and accesses the LambdaBug
> class it will see it as a recursive initialization attempt and simply
> return.

Re: Calling a lambda expression from a new thread before the main method is run causes the thread to lock up

[Luke Hutchison]

On Tue, Jan 24, 2017 at 12:02 AM, Remi Forax  wrote:

> a worker thread of the executor will try to execute the code of the static
> method but because the static initializer is not finished, the worker
> thread has to wait


But what is the worker thread waiting for in the case of the lambda that it
is not waiting for in the case of the AIC? I assume this is due to the
lexical scoping of the lambda?

As a rule of thumb, never starts a thread in a static block, you will get
> this classical deadlock AND your code is hard to test because static blocks
> tend to  be executed in a random order (i.e. class loading is lazy so
> execution of static blocks are lazy too).
>

This is the problem: I am a library author, and a user of my library ran
into this problem. The user did not necessarily know that I was launching
new threads in my library, and as a library author, I did not know I needed
to advise users that they should not call my library from static blocks. It
seems to be quite a big problem that a static block is not a standard
execution context and can lead to deadlocks.

Is there any way I can detect this in my code? I guess I could just get a
stacktrace, and look for "" or similar somewhere in the stacktrace,
then throw an exception if called in this context? Is this a reliable
enough test for being run in a static initializer block?

Re: Calling a lambda expression from a new thread before the main method is run causes the thread to lock up

[Luke Hutchison]

On Mon, Jan 23, 2017 at 11:37 PM, Luke Hutchison <luke.hu...@gmail.com>
wrote:

> On Mon, Jan 23, 2017 at 11:21 PM, Luke Hutchison <luke.hu...@gmail.com>
> wrote:
>
>> That looks like a variation of the classic class initialization deadlock
>>> problem. Your main thread is blocked in es.submit(callable).get(); while
>>> still within the static initializer of the LambdaBug class. If the executor
>>> thread also needs to ensure LambdaBug is initialized (which it will in the
>>> lambda case) then it will wait for the main thread to complete the
>>> initialization. Hence deadlock.
>>
>>
PS I submitted a bug report before finding core-libs-dev, and that bug was
just posted to Jira:

http://bugs.java.com/bugdatabase/view_bug.do?bug_id=JDK-8173252

Re: Calling a lambda expression from a new thread before the main method is run causes the thread to lock up

[Luke Hutchison]

On Mon, Jan 23, 2017 at 11:21 PM, Luke Hutchison <luke.hu...@gmail.com>
wrote:

> That looks like a variation of the classic class initialization deadlock
>> problem. Your main thread is blocked in es.submit(callable).get(); while
>> still within the static initializer of the LambdaBug class. If the executor
>> thread also needs to ensure LambdaBug is initialized (which it will in the
>> lambda case) then it will wait for the main thread to complete the
>> initialization. Hence deadlock.
>>
>
> Thanks for the explanation -- although I don't understand why a lambda
> expression requires its defining class to be initialized, whereas an
> anonymous inner class does not.
>

Also, I'm trying to determine if it is possible to detect cases like this
at runtime without triggering the deadlock, since I have already had one
bug report from a user of FastClasspathScanner running into this problem:

https://github.com/lukehutch/fast-classpath-scanner/issues/103


This user could not understand why FastClasspathScanner appeared to
deadlock when he passed in a lambda, but not when he passed in an anonymous
inner class.

Is it possible to examine an object to determine (1) if it was defined as a
lambda, and (2) if the enclosing class (i.e. the class that defined the
lambda) is initialized? (I can't see foolproof ways to do either of these
things, based on a cursory search.)  If so, at least I will be able to
throw an exception in my code, which will be better than hitting a deadlock
situation.

Re: Calling a lambda expression from a new thread before the main method is run causes the thread to lock up

[Luke Hutchison]

On Mon, Jan 23, 2017 at 10:48 PM, David Holmes <david.hol...@oracle.com>
wrote:

> On 24/01/2017 2:41 PM, Luke Hutchison wrote:
>
>> If you run the code below, the active JVM thread (in the ExecutorService)
>> locks up when the lambda expression is called. The Eclipse debugger is not
>> able to stop the locked-up thread, or get a stacktrace beyond the call
>> into
>> the lambda.
>>
>
> That looks like a variation of the classic class initialization deadlock
> problem. Your main thread is blocked in es.submit(callable).get(); while
> still within the static initializer of the LambdaBug class. If the executor
> thread also needs to ensure LambdaBug is initialized (which it will in the
> lambda case) then it will wait for the main thread to complete the
> initialization. Hence deadlock.
>

Thanks for the explanation -- although I don't understand why a lambda
expression requires its defining class to be initialized, whereas an
anonymous inner class does not. The lambda body does not refer to the
LambdaBug class. This must be due to an implicit reference to the defining
class being added by the compiler for purposes of capturing the lexical
scope (with this issue not getting triggered in the case of anonymous inner
classes due to scoping differences)? However, why is the containing class
reference even used in this case, when variables in the containing scope
are not referenced?

If this is due to lexical scoping, shouldn't the compiler be able to add
some runtime code to detect cases of this kind of deadlock, and throw a
RuntimeException, or potentially even detect some of these cases statically?

Why is it possible to directly call lambdas inside a static initializer
block without causing a deadlock (in other words, why is there no deadlock
as long as the lambdas are not executed on a different thread while the
main thread blocks)? The calling class is still not completely initialized
when calling lambdas directly in a static initializer block on the main
thread.

Calling a lambda expression from a new thread before the main method is run causes the thread to lock up

[Luke Hutchison]

If you run the code below, the active JVM thread (in the ExecutorService)
locks up when the lambda expression is called. The Eclipse debugger is not
able to stop the locked-up thread, or get a stacktrace beyond the call into
the lambda.

The problem seems to be that some part of the Java 8 lambda system is not
properly initialized until just before the main method is called. This
problem is only exhibited if you call a lambda expression from a new
thread, spawned by a static initializer block, before the main method is
executed.

Reproducibility: 100%. (If you change the value of either or both of the
static boolean fields, the code completes as expected.)

This problem exists in both Oracle jdk1.8.0_121 and Fedora OpenJDK
1.8.0.111 (64 bit).



import java.util.concurrent.Callable;
import java.util.concurrent.ExecutionException;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;

public class LambdaBug {

// Code will hang after "Entering startUp()" if both of these are true
private static final boolean RUN_AS_STATIC_INITIALIZER = true;
private static final boolean RUN_USING_LAMBDA = true;

static {
if (RUN_AS_STATIC_INITIALIZER) {
startUp();
}
}

private static void startUp() {
System.out.println("Entering startUp()");
ExecutorService es = Executors.newSingleThreadExecutor();
try {
Callable callable;
if (RUN_USING_LAMBDA) {
callable = () -> {
System.out.println("Lambda executed");
return null;
};
} else {
callable = new Callable() {
@Override
public Void call() throws Exception {
System.out.println("Inner class method executed");
return null;
}
};
}
es.submit(callable).get();
} catch (InterruptedException | ExecutionException e) {
throw new RuntimeException(e);
} finally {
es.shutdown();
}
System.out.println("Exiting startUp()");
}

public static void main(String[] args) {
if (!RUN_AS_STATIC_INITIALIZER) {
startUp();
}
System.out.println("Exiting main");
}
}