robreeves commented on issue #3297: URL: https://github.com/apache/iceberg-python/issues/3297#issuecomment-4346004984
> Hey, thanks for raising. Do you have a specific use case or workflow for using deepcopy on expressions? We use PyIceberg to load data for model training in [Ray](https://docs.ray.io/en/latest/index.html). As part of this we have PyIceberg filters that are pushed down to dataset reads as part of a config object. Ray is doing a deep copy and we hit this issue. Other PyIceberg filters don't have this issue because they don't override `__new__`. ``` File "/export/apps/python/3.12/lib/python3.12/site-packages/ray/train/v2/api/data_parallel_trainer.py", line 179, in fit callbacks=self._create_default_callbacks(), File "/export/apps/python/3.12/lib/python3.12/site-packages/ray/train/v2/api/data_parallel_trainer.py", line 206, in _create_default_callbacks datasets_callback = DatasetsSetupCallback( File "/export/apps/python/3.12/lib/python3.12/site-packages/ray/train/v2/_internal/callbacks/datasets.py", line 54, in __init__ self._data_config = copy.deepcopy(train_run_context.dataset_config) File "/export/apps/python/3.12/lib/python3.12/copy.py", line 162, in deepcopy y = _reconstruct(x, memo, *rv) File "/export/apps/python/3.12/lib/python3.12/copy.py", line 259, in _reconstruct state = deepcopy(state, memo) File "/export/apps/python/3.12/lib/python3.12/copy.py", line 136, in deepcopy y = copier(x, memo) File "/export/apps/python/3.12/lib/python3.12/copy.py", line 221, in _deepcopy_dict y[deepcopy(key, memo)] = deepcopy(value, memo) File "/export/apps/python/3.12/lib/python3.12/copy.py", line 136, in deepcopy y = copier(x, memo) File "/export/apps/python/3.12/lib/python3.12/copy.py", line 196, in _deepcopy_list append(deepcopy(a, memo)) File "/export/apps/python/3.12/lib/python3.12/copy.py", line 162, in deepcopy y = _reconstruct(x, memo, *rv) File "/export/apps/python/3.12/lib/python3.12/copy.py", line 259, in _reconstruct state = deepcopy(state, memo) File "/export/apps/python/3.12/lib/python3.12/copy.py", line 136, in deepcopy y = copier(x, memo) File "/export/apps/python/3.12/lib/python3.12/copy.py", line 221, in _deepcopy_dict y[deepcopy(key, memo)] = deepcopy(value, memo) File "/export/apps/python/3.12/lib/python3.12/copy.py", line 162, in deepcopy y = _reconstruct(x, memo, *rv) File "/export/apps/python/3.12/lib/python3.12/copy.py", line 253, in _reconstruct y = func(*args) File "/export/apps/python/3.12/lib/python3.12/copy.py", line 252, in <genexpr> args = (deepcopy(arg, memo) for arg in args) File "/export/apps/python/3.12/lib/python3.12/copy.py", line 143, in deepcopy y = copier(memo) File "/export/apps/python/3.12/lib/python3.12/site-packages/pydantic/main.py", line 977, in __deepcopy__ m = cls.__new__(cls) TypeError: And.__new__() missing 2 required positional arguments: 'left' and 'right' ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
