Re: RFR: 8268129: LibraryLookup::ofDefault leaks symbols from loaded libraries [v7]
On Thu, 3 Jun 2021 20:49:44 GMT, Maurizio Cimadamore wrote: >> This patch overhauls the library loading mechanism used by the Foreign >> Linker API. We realized that, while handy, the *default* lookup abstraction >> (`LibraryLookup::ofDefault`) was behaving inconsistentlt across platforms. >> >> This patch replaces `LibraryLookup` with a simpler `SymbolLookup` >> abstraction, a functional interface. Crucially, `SymbolLookup` does not >> concern with library loading, only symbol lookup. For this reason, two >> factories are added: >> >> * `SymbolLookup::loaderLookup` - which obtains a lookup that can be used to >> lookup symbols in libraries loaded by current loader >> * `CLinker::systemLookup` - a more stable replacement for the *default* >> lookup, which looks for symbols in libc.so (or its equivalent in other >> platforms). The contents of this lookup are unspecified. >> >> Both factories are *restricted*, so they can only be called when >> `--enable-native-access` is set. > > Maurizio Cimadamore has updated the pull request incrementally with one > additional commit since the last revision: > > Address review comments on shim lib makefile If so, I am wondering whether there is anything else left (inherited from JDK16) in JDK17 we can leverage to support `libc.a` on AIX except the way similar to Windows. - PR: https://git.openjdk.java.net/jdk/pull/4316
Re: RFR: 8268129: LibraryLookup::ofDefault leaks symbols from loaded libraries [v7]
On Thu, 3 Jun 2021 20:49:44 GMT, Maurizio Cimadamore
wrote:
>> This patch overhauls the library loading mechanism used by the Foreign
>> Linker API. We realized that, while handy, the *default* lookup abstraction
>> (`LibraryLookup::ofDefault`) was behaving inconsistentlt across platforms.
>>
>> This patch replaces `LibraryLookup` with a simpler `SymbolLookup`
>> abstraction, a functional interface. Crucially, `SymbolLookup` does not
>> concern with library loading, only symbol lookup. For this reason, two
>> factories are added:
>>
>> * `SymbolLookup::loaderLookup` - which obtains a lookup that can be used to
>> lookup symbols in libraries loaded by current loader
>> * `CLinker::systemLookup` - a more stable replacement for the *default*
>> lookup, which looks for symbols in libc.so (or its equivalent in other
>> platforms). The contents of this lookup are unspecified.
>>
>> Both factories are *restricted*, so they can only be called when
>> `--enable-native-access` is set.
>
> Maurizio Cimadamore has updated the pull request incrementally with one
> additional commit since the last revision:
>
> Address review comments on shim lib makefile
Just tried with `System.load()` but still ended up with pretty much the same
failure given both of them eventually invokes `ClassLoader.loadLibrary` to load
the library in which case there is no big difference at this point.
static {
System.load("/usr/lib/libc.a");
}
Exception in thread "main" java.lang.UnsatisfiedLinkError: Can't load
/usr/lib/libc.a <
at java.base/java.lang.ClassLoader.loadLibrary(ClassLoader.java:1793)
at java.base/java.lang.System.load(System.java:672)
at StdLibTest.(StdLibTest.java:24)
-
PR: https://git.openjdk.java.net/jdk/pull/4316
Re: RFR: 8268129: LibraryLookup::ofDefault leaks symbols from loaded libraries
On Tue, 12 Oct 2021 15:04:02 GMT, Maurizio Cimadamore
wrote:
> Is libc.a loadable on AIX (e.g. with System.loadLibrary) ?
I tried to load `libc.a` and `libc` this way but neither of them works on AIX.
e.g.
public class StdLibTest {
private static CLinker clinker = CLinker.getInstance();
static {
System.loadLibrary("libc.a"); <-
}
private static final SymbolLookup defaultLibLookup =
SymbolLookup.loaderLookup();
public static void main(String args[]) throws Throwable {
Addressable strlenSymbol =
defaultLibLookup.lookup("strlen").get();
}
}
$ ./bin/java --enable-native-access=ALL-UNNAMED --add-modules
jdk.incubator.foreign
-Dforeign.restricted=permit --enable-native-access=ALL-UNNAMED StdLibTest
WARNING: Using incubator modules: jdk.incubator.foreign
Exception in thread "main" java.lang.UnsatisfiedLinkError: Can't load libc.a
<---
at java.base/java.lang.ClassLoader.loadLibrary(ClassLoader.java:1822)
at java.base/java.lang.System.loadLibrary(System.java:694)
at StdLibTest.(StdLibTest.java:23)
and
public class StdLibTest {
private static CLinker clinker = CLinker.getInstance();
static {
System.loadLibrary("libc"); <---
}
private static final SymbolLookup defaultLibLookup =
SymbolLookup.loaderLookup();
public static void main(String args[]) throws Throwable {
Addressable strlenSymbol =
defaultLibLookup.lookup("strlen").get();
}
}
$ ./bin/java --enable-native-access=ALL-UNNAMED --add-modules
jdk.incubator.foreign
-Dforeign.restricted=permit --enable-native-access=ALL-UNNAMED StdLibTest
WARNING: Using incubator modules: jdk.incubator.foreign
Exception in thread "main" java.lang.UnsatisfiedLinkError: Can't load libc
<--
at java.base/java.lang.ClassLoader.loadLibrary(ClassLoader.java:1822)
at java.base/java.lang.System.loadLibrary(System.java:694)
at StdLibTest.(StdLibTest.java:23)
-
PR: https://git.openjdk.java.net/jdk/pull/4316
Re: RFR: 8268129: LibraryLookup::ofDefault leaks symbols from loaded libraries
On Fri, 8 Oct 2021 21:29:19 GMT, Maurizio Cimadamore
wrote:
>> Hi @mcimadamore,
>>
>> As you mentioned at
>> https://github.com/openjdk/jdk/pull/4316#issuecomment-853238872, the system
>> lookup is supported by the underlying `NativeLibraries` which also works on
>> OpenJDK16 to support `LibraryLookup::ofDefault`.
>>
>> But my finding is that it `LibraryLookup::ofDefault` works good on AIX (with
>> `libc.a`) in OpenJDK16 but `CLinker.systemLookup()` ended up with
>> `NoSuchElementException` after changing the code in `SystemLookup.java` and
>> `CABI.java` as follows:
>>
>> private static final SymbolLookup syslookup = switch (CABI.current()) {
>> case SysV, LinuxAArch64, MacOsAArch64, AIX -> libLookup(libs ->
>> libs.loadLibrary("syslookup"));
>> case Win64 -> makeWindowsLookup(); // out of line to workaround
>> javac crash
>> };
>>
>> with a simple test & output:
>>
>> import jdk.incubator.foreign.CLinker;
>> import static jdk.incubator.foreign.CLinker.*;
>> import jdk.incubator.foreign.SymbolLookup;
>> import jdk.incubator.foreign.Addressable;
>>
>> public class Test {
>> private static CLinker clinker = CLinker.getInstance();
>> private static final SymbolLookup defaultLibLookup =
>> CLinker.systemLookup();
>>
>> public static void main(String args[]) throws Throwable {
>> Addressable strlenSymbol =
>> defaultLibLookup.lookup("strlen").get();
>> }
>> }
>>
>> bin/java --enable-native-access=ALL-UNNAMED --add-modules
>> jdk.incubator.foreign -Dforeign.restricted=permit Test
>> WARNING: Using incubator modules: jdk.incubator.foreign
>> Exception in thread "main" java.util.NoSuchElementException: No value
>> present <-
>> at java.base/java.util.Optional.get(Optional.java:143)
>> at Test.main(Test.java:11)
>>
>>
>> So I am wondering what happened to the system lookup in such case given
>> there should be no fundamental difference in leveraging `NativeLibraries` (I
>> assume the library loading in OpenJDK16 & 17 is the same at this point)
>> unless there is something else new in OpenJDK17 I am unaware of (e.g. the
>> changes in `Lib.gmk` or `lib-std.m4`, or a custom system lookup is required
>> on AIX, etc). Thanks.
>
>> So I am wondering what happened to the system lookup in such case given
>> there should be no fundamental difference in leveraging `NativeLibraries` (I
>> assume the library loading in OpenJDK16 & 17 is the same at this point)
>> unless there is something else new in OpenJDK17 I am unaware of (e.g. the
>> changes in `Lib.gmk` or `lib-std.m4`, or a custom system lookup is required
>> on AIX, etc). Thanks.
>
> In 17, SystemLookup loads a specific library that is generated at build time
> - which contains all the c stdlib symbols. That's what the Lib.gmk changes
> are for. What I suspect is that the library is not generated at all, or not
> correctly on AIX, which then causes the system lookup to misbehave.
>
> I would start by double checking that you have a file like this:
>
> /lib/libsyslookup.so
>
> And then, if the library exists, check that it has the right dependency; on
> my linux it's an empty library, but which depends on libc, libm and libdl:
>
>
> ldd lib/libsyslookup.so
> linux-vdso.so.1 (0x7fff2bdf7000)
> libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x7f35f1def000)
> libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x7f35f1ca)
> libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x7f35f1c9a000)
Hi @mcimadamore,
Here's output of `libsyslookup.so` on AIX:
$ ldd ./lib/libsyslookup.so
./lib/libsyslookup.so needs: <--- nothing here
$ whereis libc
libc: /usr/lib/libc.a /usr/lib/libc128.a /usr/ccs/lib/libc.a
So it is high likely that there were no dependencies in this generated library.
> perhaps on AIX something similar to what we did for Windows [1] would be
> better.
That's what I thought to be the only way around but might need to figure out
the specifics on AIX.
-
PR: https://git.openjdk.java.net/jdk/pull/4316
Re: RFR: 8268129: LibraryLookup::ofDefault leaks symbols from loaded libraries
On Wed, 2 Jun 2021 20:13:34 GMT, Maurizio Cimadamore
wrote:
>> This patch overhauls the library loading mechanism used by the Foreign
>> Linker API. We realized that, while handy, the *default* lookup abstraction
>> (`LibraryLookup::ofDefault`) was behaving inconsistentlt across platforms.
>>
>> This patch replaces `LibraryLookup` with a simpler `SymbolLookup`
>> abstraction, a functional interface. Crucially, `SymbolLookup` does not
>> concern with library loading, only symbol lookup. For this reason, two
>> factories are added:
>>
>> * `SymbolLookup::loaderLookup` - which obtains a lookup that can be used to
>> lookup symbols in libraries loaded by current loader
>> * `CLinker::systemLookup` - a more stable replacement for the *default*
>> lookup, which looks for symbols in libc.so (or its equivalent in other
>> platforms). The contents of this lookup are unspecified.
>>
>> Both factories are *restricted*, so they can only be called when
>> `--enable-native-access` is set.
>
>> _Mailing list message from [Chapman Flack](mailto:c...@anastigmatix.net) on
>> [security-dev](mailto:security-...@mail.openjdk.java.net):_
>>
>> On 06/02/21 13:30, Maurizio Cimadamore wrote:
>>
>> > This patch replaces `LibraryLookup` with a simpler `SymbolLookup`
>> > abstraction, a functional interface. Crucially, `SymbolLookup` does not
>> > concern with library loading, only symbol lookup. For this reason, two
>> > factories are added:
>>
>> Hi,
>>
>> While I am thinking about this, what will be the behavior when the JVM
>> itself has been dynamically loaded into an embedding application, and
>> launched with the JNI invocation API?
>>
>> Will there be a *Lookup flavor that is able to find any exported symbol
>> known in the embedding environment the JVM was loaded into? (This is the
>> sort of condition the Mac OS linker checks when given the -bundle_loader
>> option.)
>>
>> Regards,
>> Chapman Flack (maintainer of a project that happens to work that way)
>
> Hi,
> at the moment we don't have plans to add such a lookup, but I believe it
> should be possible to build such a lookup using `dlopen` and the linker API.
> I have provided an example elsewhere of how easy it easy to build a wrapper
> lookup around dlopen/dlsym:
>
> https://gist.github.com/mcimadamore/0883ea6f4836ae0c1d2a31c48197da1a
>
> Perhaps something like that could be useful in the use case you mention?
Hi @mcimadamore,
As you mentioned at
https://github.com/openjdk/jdk/pull/4316#issuecomment-853238872, the system
lookup is supported by the underlying `NativeLibraries` which also works on
OpenJDK16 to support `LibraryLookup::ofDefault`.
But my finding is that it `LibraryLookup::ofDefault` works good on AIX (with
`libc.a`) in OpenJDK16 but `CLinker.systemLookup()` ended up with
`NoSuchElementException` after changing the code in `SystemLookup.java` and
`CABI.java` as follows:
private static final SymbolLookup syslookup = switch (CABI.current()) {
case SysV, LinuxAArch64, MacOsAArch64, AIX -> libLookup(libs ->
libs.loadLibrary("syslookup"));
case Win64 -> makeWindowsLookup(); // out of line to workaround javac
crash
};
with a simple test & output:
import jdk.incubator.foreign.CLinker;
import static jdk.incubator.foreign.CLinker.*;
import jdk.incubator.foreign.SymbolLookup;
import jdk.incubator.foreign.Addressable;
public class Test {
private static CLinker clinker = CLinker.getInstance();
private static final SymbolLookup defaultLibLookup =
CLinker.systemLookup();
public static void main(String args[]) throws Throwable {
Addressable strlenSymbol =
defaultLibLookup.lookup("strlen").get();
}
}
bin/java --enable-native-access=ALL-UNNAMED --add-modules
jdk.incubator.foreign -Dforeign.restricted=permit Test
WARNING: Using incubator modules: jdk.incubator.foreign
Exception in thread "main" java.util.NoSuchElementException: No value present
<-
at java.base/java.util.Optional.get(Optional.java:143)
at Test.main(Test.java:11)
So I am wondering what happened to the system lookup in such case given there
should be no fundamental difference in leveraging `NativeLibraries` (I assume
the library loading in OpenJDK16 & 17 is the same at this point) unless there
is something else new in OpenJDK17 I am unaware of (e.g. the changes in
`Lib.gmk` or `lib-std.m4`, or a custom system lookup is required on AIX, etc).
Thanks.
-
PR: https://git.openjdk.java.net/jdk/pull/4316
