tarun-google commented on code in PR #40699: URL: https://github.com/apache/beam/pull/40699#discussion_r4384189561
########## CHANGES.md: ########## @@ -99,6 +99,8 @@ * (Python) `Regex.all_matches` now returns every group of the regex, instead of failing with a `TypeError` when the regex has no groups and dropping nested or unmatched trailing groups ([#40562](https://github.com/apache/beam/issues/40562)). * (Python) `--dataflow_endpoint` now defaults to `''` instead of `https://dataflow.googleapis.com`. When unset, the Google Cloud client library resolves the Dataflow endpoint and honors `GOOGLE_CLOUD_UNIVERSE_DOMAIN`, so Dataflow jobs can be submitted in non-default universes. Code that reads `GoogleCloudOptions.dataflow_endpoint` should handle an empty value ([#40569](https://github.com/apache/beam/issues/40569)). * (Python) The `xgboost` extras now install `xgboost>=2.0.0,<4` and `datatable>=1.1.0,<2`, moving both the lower and upper bounds for both. The previous minimum `xgboost` and `datatable` versions (`1.6.0` and `1.0.0` respectively) are no longer allowed ([#40694](https://github.com/apache/beam/issues/40694)). +* (Python) The Python SDK container now downloads each environment's staged files to `<semi_persist_dir>/<worker id>/staged` instead of `<semi_persist_dir>/staged`, so environments sharing a worker no longer overwrite each other's files. Code that hardcodes a staged file location, such as `/tmp/staged/...` or `/var/opt/google/staged/...`, must instead resolve it on the worker at runtime, for example `os.path.join(os.environ['SEMI_PERSISTENT_DIRECTORY'], 'staged', 'my_file')`. See [Accessing staged files on the workers](https://beam.apache.org/documentation/sdks/python-pipeline-dependencies/#accessing-staged-files-on-the-workers) ([#40699](https://github.com/apache/beam/issues/40699)). Review Comment: Yes, i do not have great answer to pipeline construction path, this might be heavy but customers can follow pattern like github.com/apache/beam/pull/40795 or use to gs:// If you really think there should be a migration path, we can copy the files to both /var/opt/google/staged/.. and /var/opt/google/staged/worker_id/..for a few versions and warn them to migrate. But it adds memory too. I do not know if this path is such a critical usecase. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
