jroachgolf84 commented on code in PR #68133:
URL: https://github.com/apache/airflow/pull/68133#discussion_r3369676776
##########
airflow-core/src/airflow/api_fastapi/core_api/datamodels/task_store.py:
##########
@@ -66,8 +65,12 @@ def value_is_json_representable(cls, v: JsonValue) ->
JsonValue:
serialized = json.dumps(v, allow_nan=False)
except ValueError:
raise ValueError("value contains non-finite numbers; NaN and Inf
are not JSON representable")
- if len(serialized) > _MAX_SERIALIZED_BYTES:
- raise ValueError(f"value exceeds maximum serialized size of
{_MAX_SERIALIZED_BYTES} bytes")
+ limit = conf.getint("state_store", "max_value_storage_bytes")
Review Comment:
We don't want to throw this in a `validate_payload_size` function or
something to reuse like that, correct? I could be way off, but thought I'd
throw it out there.
##########
airflow-core/src/airflow/api_fastapi/core_api/datamodels/asset_store.py:
##########
@@ -67,6 +66,10 @@ def value_is_json_representable(cls, v: JsonValue) ->
JsonValue:
serialized = json.dumps(v, allow_nan=False)
except ValueError:
raise ValueError("value contains non-finite numbers; NaN and Inf
are not JSON representable")
- if len(serialized) > _MAX_SERIALIZED_BYTES:
- raise ValueError(f"value exceeds maximum serialized size of
{_MAX_SERIALIZED_BYTES} bytes")
+ limit = conf.getint("state_store", "max_value_storage_bytes")
+ if limit > 0 and len(serialized) > limit:
+ raise ValueError(
+ f"value exceeds max_value_storage_bytes ({limit}); "
+ "for large payloads configure a custom [state_store] backend"
+ )
Review Comment:
I'd lean towards putting a limit on this. I think we want to use it as a
"state store" rather than a "data store". I'd be worried about folks using as a
place to dump huge amounts of data, which I don't think is the intention.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
