potiuk commented on issue #60941:
URL: https://github.com/apache/airflow/issues/60941#issuecomment-3902213198
Actually, this is something I wanted to do a long time ago for serde - and
make serde extendable - in a very similar way @molcay explained. However I
think it's not really needed (and it would be dangerous or insecure if we do).
The issue is that eventually we want api-server to NOT have to have
providers installed - at all (except possibly api-server plugins) - we do not
want to run **any** non-core code in api-server. This is for example why we are
moving connection definitiions from providers to provider.yaml and eventually
we will store them in database, so that api-server will not have to discover
and run providers **at all**.
I think the right solution here @amoghrajesh @kaxil @ashb is to not FAIL
serialization if we can't serialize something - but to replace any
non-serializable object with a "non-serializable" sentinel - that might contain
a hash, or other "uniqueness" signature of the paramater passed. This
uniqueness could be usually be done by running`__str__` or `__repr__` (and
handling the case where those produce `<id>` of hte object - because the only
reason why we want to serialize those fields is to find out if they changed
since the last time we parsed the Dag - we absolutely don't care about
deserializaiton of those parameters if you ask me.
That would nicely work for those enums - without the need for per-provider
serialization
```python
from google.cloud.redis_v1 import FailoverInstanceRequest
n = FailoverInstanceRequest.DataProtectionMode
print(str(n))
```
produces:
```
<enum 'DataProtectionMode'>
```
And this is all we need for serialized Dag. We could store it as
`NonSerializableData("<enum 'DataProtectionMode'>") - object in serialized Dag.
We do not need anything more IMHO.
I think there is absolutly no need to make such non-serializable fields to
to serialized dag. Why would we need them at all? We only need serialized dags
for structure and meta-data, but serializing attributes is only needed to find
if they changed.
Or am I wrong?
WDYT @amoghrajesh @ashb @kaxil ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
