amoghrajesh commented on code in PR #68283:
URL: https://github.com/apache/airflow/pull/68283#discussion_r3491311392
##########
providers/common/io/provider.yaml:
##########
@@ -115,3 +115,31 @@ config:
type: string
example: "gz"
default: ""
+ state_store_objectstorage_path:
+ description: |
+ Base path on object storage for the task/asset state store backend,
in URL format.
+ When set, StateStoreObjectStorageBackend will persist task and asset
state under this
+ prefix, organised as <dag_id>/<run_id>/<task_id>/<map_index>/<key>
for tasks and
+ assets/<asset_identifier>/<key> for assets.
+ version_added: 1.8.0
+ type: string
+ example: "s3://conn_id@bucket/task-state/"
+ default: ""
+ state_store_objectstorage_threshold:
+ description: |
+ Threshold in bytes for offloading serialized state store values to
object storage. 0 means
+ always offload to object storage. Any positive number means values
will be offloaded
+ only when their serialized size is at least that many bytes. Must be
non-negative.
+ version_added: 1.8.0
+ type: integer
+ example: "1000000"
+ default: "0"
+ state_store_objectstorage_compression:
+ description: |
+ Compression algorithm to use when writing task/asset state store
values to object storage.
+ Supported algorithms are a.o.: snappy, zip, gzip, bz2, and lzma. If
not specified,
+ no compression will be used. The same algorithm must be available on
all workers.
+ version_added: 1.8.0
+ type: string
+ example: "gz"
Review Comment:
Fixed. Changed example from gz to gzip and updated the description to list
only the codec names that actually appear in fsspec.utils.compressions (gzip,
bz2, lzma, xz). Removed `snappy` and `zip` from the description since those
either require extra installs or aren't valid codec names for
`_compression_suffix()`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
