On Windows, java allows the next file dialog to start while the first native dialog is still cleaning up, so the AWT toolkit message loop re-enters `AwtFileDialog::Show`.
This can create a recursive native stack: `GetOpenFileNameW -> AwtFileDialog::Show -> AwtToolkit::WndProc -> GetOpenFileNameW ...` Eventually, the process can hit a `STATUS_STACK_OVERFLOW` (`0xc00000fd`) and then terminates with `STATUS_FATAL_USER_CALLBACK_EXCEPTION` (`0xc000041d`, or `-1073740771` as reported). --- The fix is to allow only one native Windows file dialog at a time: - The show thread now waits until the native dialog reports selected/cancelled before another dialog can enter native show. - `_show` now reports whether posting the native show request succeeded. - `dispose`/`hide` clears a pending show request. - If a native dialog window appears after the java side has already cancelled the show request, it is hidden on the event handler thread. --- This PR includes an open sourced test, with a slight modification. The JDK-8381565 issue was discovered using this test, but it reproduces only on some machines. A reduced 10 ms delay case was added to the test to improve reproducibility, while the original 100 ms delay from the JDK-4411368 test was retained. `setDropTarget` is not relevant to this issue. Testing looks good. --------- - [x] I confirm that I make this contribution in accordance with the [OpenJDK Interim AI Policy](https://openjdk.org/legal/ai). ------------- Commit messages: - 8381565: FileDialogDropTargetTest.java fails: Unexpected exit from test [exit code: -1073740771] Changes: https://git.openjdk.org/jdk/pull/31757/files Webrev: https://webrevs.openjdk.org/?repo=jdk&pr=31757&range=00 Issue: https://bugs.openjdk.org/browse/JDK-8381565 Stats: 136 lines in 3 files changed: 130 ins; 0 del; 6 mod Patch: https://git.openjdk.org/jdk/pull/31757.diff Fetch: git fetch https://git.openjdk.org/jdk.git pull/31757/head:pull/31757 PR: https://git.openjdk.org/jdk/pull/31757
