On Wed, 16 Mar 2022 14:38:55 GMT, Thomas Stuefe <stu...@openjdk.org> wrote:
>> As described in the linked issue, NullClassBytesTest fails due an >> OutOfMemoryError produced on AIX when the test calls defineClass with a byte >> array of size of 0. The native implementation of defineClass then calls >> malloc with a size of 0. On AIX malloc(0) returns NULL, while on other >> platforms it return a valid address. When NULL is produced by malloc for >> this reason, ClassLoader.c incorrectly interprets this as a failure due to a >> lack of memory. >> >> ~~This PR modifies ClassLoader.c to produce an OutOfMemoryError only when >> `errno == ENOMEM` and to produce a ClassFormatError with the message >> "ClassLoader internal allocation failure" in all other cases (in which >> malloc returns NULL).~~ [edit: The above no longer describes the PR's >> proposed fix. See discussion below] >> >> In addition, I performed some minor tidy-up work in ClassLoader.c by >> changing instances of `return 0` to `return NULL`, and `if (some_ptr == 0)` >> to `if (some_ptr == NULL)`. This was done to improve the clarity of the code >> in ClassLoader.c, but didn't feel worthy of opening a separate issue. >> >> ### Alternatives >> >> It would be possible to address this failure by modifying the test to accept >> the OutOfMemoryError on AIX. I thought it was a better solution to modify >> ClassLoader.c to produce an OutOfMemoryError only when the system is >> actually out of memory. >> >> ### Testing >> >> This change has been tested on AIX and Linux/x86. > > Btw, which malloc call was the problematic exactly? Cannot be the one in > getUTF, since that one already adds len + 1 and never gets called with a zero > length anyway. Thanks @tstuefe! Your suggestion lead to a better change, so I modified the PR. - ClassLoader.c no longer has any reason to throw a ClassFormatError, so that logic is removed. - The test no longer needs to recognize a new error message, so that is changed back as well. - I also alphabetized the header files, because that is the way I am :-) Note: I couldn't find an implementation of MAX2 in a C-friendly 'header.h' file, so I just used the ternary operator in the two places I needed it. ------------- PR: https://git.openjdk.java.net/jdk/pull/7829
