damccorm opened a new pull request, #22196: URL: https://github.com/apache/beam/pull/22196
This collects a heap dump and uploads it to the temp_directory on Dataflow any time there is a worker panic before we exit. It departs from what #21797 recommends a little bit in that it doesn't use `GOTRACEBACK=crash` and instead relies on `debug.WriteHeapDump`. This is mostly because the GOTRACEBACK approach doesn't collect the dump until the top level Go routine exits with a panic - this currently isn't guaranteed to happen (and won't by default because we recover from panics and call os.exit), but if it did it would propogate up through boot.go. At that point, the VM is not guaranteed to be persisted (in fact, I think it will always be torn down), and this generally just breaks how we handle exiting. The approach of gathering the dump ourselves is also more in line with how Java handles this anyways - https://github.com/apache/beam/blob/4ffeae4d2b800f2df36d2ea2eab549f2204d5691/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/status/MemoryMonitor.java#L589 To test this, I inserted a panic into the harness code and verified that a heap dump was correctly uploaded to the temp directory: - pipeline - https://pantheon.corp.google.com/dataflow/jobs/us-central1/2022-07-08_07_34_58-5641449712707134536;bottomTab=WORKER_LOGS;logsSeverity=INFO?project=google.com:clouddfe&pageState=(%22dfTime%22:(%22l%22:%22dfJobMaxTime%22)) - dumps from multiple pipeline runs - https://pantheon.corp.google.com/storage/browser/clouddfe-test/dannymccormick-tmp/heapDumps?pageState=(%22StorageObjectListTable%22:(%22f%22:%22%255B%255D%22))&project=google.com:clouddfe&prefix=&forceOnObjectsSortingFiltering=false) Fixes #21797 ------------------------ Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [ ] Mention the appropriate issue in your description (for example: `addresses #123`), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment `fixes #<ISSUE NUMBER>` instead. - [ ] Update `CHANGES.md` with noteworthy changes. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier). To check the build health, please visit [https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md](https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md) GitHub Actions Tests Status (on master branch) ------------------------------------------------------------------------------------------------ [](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule) [](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule) [](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule) See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
