kaxil commented on code in PR #68133:
URL: https://github.com/apache/airflow/pull/68133#discussion_r3392279292
##########
airflow-core/src/airflow/config_templates/config.yml:
##########
@@ -3239,6 +3239,21 @@ state_store:
type: integer
example: "10000"
default: "0"
+ max_value_storage_bytes:
+ description: |
+ Only applicable to MetastoreStoreBackend. Maximum size in bytes that a
single task or asset store
+ value written via the core or execution API can have. Values that
exceed this limit are rejected
+ at the API boundary.
+
+ Workers writing via the execution API log a warning to use custom
backends when this limit
+ is exceeded but the write is allowed to avoid interrupting a task mid
execution.
+
+ The default of 65535 bytes (64 KB) is appropriate for coordination
state values such as
+ job IDs, cursors, and small status maps. For larger payloads, use a
custom state backends.
+ version_added: 3.3.0
+ type: integer
+ example: "1048576"
+ default: "65535"
Review Comment:
Is there a reason for 65535 specifically? The column is
`Text().with_variant(MEDIUMTEXT, "mysql")`, so nothing DB-side enforces
anything at 64 KB (MEDIUMTEXT holds 16 MB, Postgres `Text` is unbounded). 65535
is the MySQL plain-`TEXT` max, which this column does not use. Fine as a
judgment call, but then the description should present it as a policy default
rather than implying a storage limit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
