On Thu, 20 Aug 2026 08:40:46 GMT, Ashay Rane <[email protected]> wrote:
>> Prior to this patch, the `StackYellowPages` and `StackRedPages` >> influenced the total reserved stack memory, but they didn't change the >> minimum stack size specified as required by HotSpot to Windows. This >> patch adds the call to `SetThreadStackGuarantee()` to set the minimum >> stack size. >> >> The key change here is that the argument to `SetThreadStackGuarantee()` >> is set to one less than the total number of yellow pages. The one less >> page is because Windows uses a guard page (access to which tells Windows >> to commit more stack pages); in the terminal case, we want the guard >> page to coincide with the top-most yellow stack page, thus leaving all >> except one yellow page available as the stack size. >> >> Importantly, however, we do not use the red page count as the argument >> to `SetThreadStackGurantee()`, since the red stack pages are unavailable >> for handling recoverable overflows. The red page count impacts the >> _total reserved_ stack size, just not the _minimum_ stack size. Still, >> forcing the red stack page count to be included into the computation of >> the argument to `SetThreadStackGurantee()` causes HotSpot to fail with >> the assertion `assert(!in_vm) failed: Undersized StackShadowPages`, >> since Windows is unable to commit more stack pages due to the minimum >> stack size now being larger than just the yellow page count minus one. >> >> The accompanying test passes zero to `SetThreadStackGuarantee()` to >> probe the current minimum stack size, which we then compare against the >> expected size based on the yellow page count. The same test fails >> without this patch on both Windows/x64 and on Windows/ARM64. >> >> Validated this patch by running through all tier 1, 2, and 3 HotSpot >> jtreg tests on Windows/x64 and Windows/ARM64 in FastDebug config. >> >> --------- >> - [x] I confirm that I make this contribution in accordance with the >> [OpenJDK Interim AI Policy](https://openjdk.org/legal/ai). > > Ashay Rane has updated the pull request incrementally with one additional > commit since the last revision: > > Update comment block to be precise about `SetThreadStackGuarantee()` David, it's confusing. What you described is the Linux behavior and a reasonable behavior on Windows. However, after going back and forth with AI and looking at the Wine code, I think I finally figured out why things are broken in HotSpot on Windows. The critical point is when we bang the last guard page before the first yellow page. It seems that 3 things can happen here: 1. If we are in the reserved zone as defined by SetThreadStackGuarantee, we get a EXCEPTION_STACK_OVERFLOW 2. If we set the SetThreadStackGuarantee area too small, and the first yellow page is protected, then I believe we should get a EXCEPTION_ACCESS_VIOLATION. However I'm not sure if we will get it immediately or only when we bang the first yellow page. 3. What actually happens on HotSpot is that Windows unprotects the first yellow page. I was surprised that we ask for this, by protecting the page like so: 4055 return VirtualProtect(addr, bytes, PAGE_READWRITE | PAGE_GUARD, &old_status) != 0; instead of using PAGE_NOACCESS like I expected. Looking at the Wine source code, it looks like it will unprotect not just one page, but the reserved area as set by SetThreadStackGuarantee. So setting SetThreadStackGuarantee to the yellow zone size means the whole yellow zone gets unprotected. I am now thinking that guarding the yellow and red pages with PAGE_NOACCESS is the better choice. Then we will get EXCEPTION_ACCESS_VIOLATION when we hit the first yellow page. Setting SetThreadStackGuarantee correctly seems more problematic than I thought. It seems to add 1 page for a Windows red zone. If we wanted to have a bigger red zone in HotSpot, then we would need to include that in the SetThreadStackGuarantee size, assuming we want to get a EXCEPTION_STACK_OVERFLOW in the right place. But then Windows would unprotect both the yellow and the red zone, according to Wine. And we don't need Windows to unprotect pages for us. That's one reason why HotSpot has the shadow pages zone and it does a stack bang far above the current stack pointer. ------------- PR Comment: https://git.openjdk.org/jdk/pull/32365#issuecomment-5367436518
