tvalentyn commented on code in PR #40699: URL: https://github.com/apache/beam/pull/40699#discussion_r4383759832
########## CHANGES.md: ########## @@ -99,6 +99,8 @@ * (Python) `Regex.all_matches` now returns every group of the regex, instead of failing with a `TypeError` when the regex has no groups and dropping nested or unmatched trailing groups ([#40562](https://github.com/apache/beam/issues/40562)). * (Python) `--dataflow_endpoint` now defaults to `''` instead of `https://dataflow.googleapis.com`. When unset, the Google Cloud client library resolves the Dataflow endpoint and honors `GOOGLE_CLOUD_UNIVERSE_DOMAIN`, so Dataflow jobs can be submitted in non-default universes. Code that reads `GoogleCloudOptions.dataflow_endpoint` should handle an empty value ([#40569](https://github.com/apache/beam/issues/40569)). * (Python) The `xgboost` extras now install `xgboost>=2.0.0,<4` and `datatable>=1.1.0,<2`, moving both the lower and upper bounds for both. The previous minimum `xgboost` and `datatable` versions (`1.6.0` and `1.0.0` respectively) are no longer allowed ([#40694](https://github.com/apache/beam/issues/40694)). +* (Python) The Python SDK container now downloads each environment's staged files to `<semi_persist_dir>/<worker id>/staged` instead of `<semi_persist_dir>/staged`, so environments sharing a worker no longer overwrite each other's files. Code that hardcodes a staged file location, such as `/tmp/staged/...` or `/var/opt/google/staged/...`, must instead resolve it on the worker at runtime, for example `os.path.join(os.environ['SEMI_PERSISTENT_DIRECTORY'], 'staged', 'my_file')`. See [Accessing staged files on the workers](https://beam.apache.org/documentation/sdks/python-pipeline-dependencies/#accessing-staged-files-on-the-workers) ([#40699](https://github.com/apache/beam/issues/40699)). Review Comment: i'm thinking, resolving at runtime becomes non-trivial if the arg must be supplied at pipeline construction time. For example https://github.com/apache/beam/blob/b9a783ad24909fbb5e49e8d6e76740812b8aa12c/sdks/python/apache_beam/ml/inference/pytorch_inference.py#L201 accepts a path. if i use --files_to_stage to stage a model, how would is specify a path to that model on the worker correctly? the env var value is not known yet, it must be evaluated at runtime. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
