sjyangkevin commented on issue #52753:
URL: https://github.com/apache/airflow/issues/52753#issuecomment-3094803798
I attempted to fix the numpy.bool issue with the following approaches
without success.
### Approach 1: Simply add `numpy.bool` to the `serializers` and
`deserializers` list
```python
serializers = [
"numpy.int8",
"numpy.int16",
"numpy.int32",
"numpy.int64",
"numpy.uint8",
"numpy.uint16",
"numpy.uint32",
"numpy.uint64",
"numpy.bool",
"numpy.bool_",
"numpy.float64",
"numpy.float16",
"numpy.complex128",
"numpy.complex64",
]
```
This still result in the following test failures when Numpy version is
`1.26.4`. The issue is resolved when upgrading numpy to `2.3.1`.
```
FAILED test_serializers.py::TestSerializers::test_numpy - ImportError:
Module "numpy" does not define a "bool" attribute/class
FAILED
test_serializers.py::TestSerializers::test_numpy_deserialize_errors[float32-1-123-unsupported
numpy\\.float32] - ImportError: Module "numpy" does not define a "bool"
attribute/class
```
### Approach 2: Update serializers list based on numpy version
```python
#
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
from __future__ import annotations
from importlib import metadata
from typing import TYPE_CHECKING, Any
from packaging import version
from airflow.utils.module_loading import import_string, qualname
NUMPY_VERSION = metadata.version("numpy") # Identify numpy version
# lazy loading for performance reasons
serializers = [
"numpy.int8",
"numpy.int16",
"numpy.int32",
"numpy.int64",
"numpy.uint8",
"numpy.uint16",
"numpy.uint32",
"numpy.uint64",
"numpy.float64",
"numpy.float16",
"numpy.complex128",
"numpy.complex64",
]
# qualified name of numpy bool is resolved to <class 'numpy.bool_'> in numpy
1.x
# but it is resolved to <class 'numpy.bool'> in numpy 2.x
if version.parse(NUMPY_VERSION).major < 2:
serializers.extend(["numpy.bool_"])
else:
serializers.extend(["numpy.bool"])
if TYPE_CHECKING:
from airflow.serialization.serde import U
deserializers = serializers
__version__ = 1
def serialize(o: object) -> tuple[U, str, int, bool]:
if np is None:
return "", "", 0, False
name = qualname(o)
if isinstance(
o,
np.int_
| np.intc
| np.intp
| np.int8
| np.int16
| np.int32
| np.int64
| np.uint8
| np.uint16
| np.uint32
| np.uint64,
):
return int(o), name, __version__, True
# in numpy 2.x np.bool_ is an alias of np.bool.
if isinstance(o, np.bool_):
return bool(o), name, __version__, True
if isinstance(o, (np.float16, np.float32, np.float64, np.complex64,
np.complex128)):
return float(o), name, __version__, True
return "", "", 0, False
def deserialize(cls: type, version: int, data: str) -> Any:
if version > __version__:
raise TypeError("serialized version is newer than class version")
allowed_deserialize_classes = [import_string(classname) for classname in
deserializers]
if cls not in allowed_deserialize_classes:
raise TypeError(f"unsupported {qualname(cls)} found for numpy
deserialization")
return cls(data)
```
However, the following failure raised in pre-commit checks.
Generate Datamodels for TaskSDK client
```
importlib.metadata.PackageNotFoundError: No package metadata was found for
numpy
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
