amoghrajesh commented on code in PR #68133:
URL: https://github.com/apache/airflow/pull/68133#discussion_r3393457820
##########
airflow-core/src/airflow/config_templates/config.yml:
##########
@@ -3239,6 +3239,21 @@ state_store:
type: integer
example: "10000"
default: "0"
+ max_value_storage_bytes:
+ description: |
+ Only applicable to MetastoreStoreBackend. Maximum size in bytes that a
single task or asset store
+ value written via the core or execution API can have. Values that
exceed this limit are rejected
+ at the API boundary.
+
+ Workers writing via the execution API log a warning to use custom
backends when this limit
+ is exceeded but the write is allowed to avoid interrupting a task mid
execution.
+
+ The default of 65535 bytes (64 KB) is appropriate for coordination
state values such as
+ job IDs, cursors, and small status maps. For larger payloads, use a
custom state backends.
+ version_added: 3.3.0
+ type: integer
+ example: "1048576"
+ default: "65535"
Review Comment:
There is no specific such reason, it just felt like the rightly suited size
to use for such an intent to checkpoint with job id, cursor etc. Clarified to
this now:
```
The default of 65535 bytes (64 KB) is a policy default suited for
coordination state such as
job IDs, cursors, and small status maps — the underlying database
column (MEDIUMTEXT on MySQL,
unbounded Text on Postgres) does not enforce this limit. For larger
payloads, configure a custom
[workers] state_store_backend to offload values to external storage.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
