bugraoz93 commented on code in PR #47924: URL: https://github.com/apache/airflow/pull/47924#discussion_r2002639705
########## docs/apache-airflow/core-concepts/xcoms.rst: ########## @@ -83,29 +83,7 @@ An example of pushing multiple XComs and pulling them individually: Object Storage XCom Backend --------------------------- -The default XCom backend is the :class:`~airflow.models.xcom.BaseXCom` class, which stores XComs in the Airflow database. This is fine for small values, but can be problematic for large values, or for large numbers of XComs. - -To enable storing XComs in an object store, you can set the ``xcom_backend`` configuration option to ``airflow.providers.common.io.xcom.backend.XComObjectStorageBackend``. -You will also need to set ``xcom_objectstorage_path`` to the desired location. The connection id is obtained from the user part of the url that you will provide, e.g. ``xcom_objectstorage_path = s3://conn_id@mybucket/key``. Furthermore, ``xcom_objectstorage_threshold`` is required -to be something larger than -1. Any object smaller than the threshold in bytes will be stored in the database and anything larger will be be -put in object storage. This will allow a hybrid setup. If an xcom is stored on object storage a reference will be -saved in the database. Finally, you can set ``xcom_objectstorage_compression`` to fsspec supported compression methods like ``zip`` or ``snappy`` to -compress the data before storing it in object storage. - -So for example the following configuration will store anything above 1MB in S3 and will compress it using gzip:: - - [core] - xcom_backend = airflow.providers.common.io.xcom.backend.XComObjectStorageBackend - - [common.io] - xcom_objectstorage_path = s3://conn_id@mybucket/key - xcom_objectstorage_threshold = 1048576 - xcom_objectstorage_compression = gzip - - -.. note:: - - Compression requires the support for it is installed in your python environment. For example, to use ``snappy`` compression, you need to install ``python-snappy``. Zip, gzip and bz2 work out of the box. +The default XCom backend, BaseXCom, stores XComs in the Airflow database, which works well for small values but can cause issues with large values or a high volume of XComs. To overcome this limitation, object storage is recommended for efficiently handling larger data. For a detailed overview, refer to the :doc:`documentation <apache-airflow-providers-common-io:xcom_backend>`. Review Comment: Yes, that is already in the reference document, and there is no need to stay here while it was an example. I missed that. It is limited in the database for sure as a hard limit, but I think 1 MB was an example in this case. It depends on what you have set on the configuration, so it is on the application level. Let's not make this changes ########## docs/apache-airflow/core-concepts/xcoms.rst: ########## @@ -83,29 +83,7 @@ An example of pushing multiple XComs and pulling them individually: Object Storage XCom Backend --------------------------- -The default XCom backend is the :class:`~airflow.models.xcom.BaseXCom` class, which stores XComs in the Airflow database. This is fine for small values, but can be problematic for large values, or for large numbers of XComs. - -To enable storing XComs in an object store, you can set the ``xcom_backend`` configuration option to ``airflow.providers.common.io.xcom.backend.XComObjectStorageBackend``. -You will also need to set ``xcom_objectstorage_path`` to the desired location. The connection id is obtained from the user part of the url that you will provide, e.g. ``xcom_objectstorage_path = s3://conn_id@mybucket/key``. Furthermore, ``xcom_objectstorage_threshold`` is required -to be something larger than -1. Any object smaller than the threshold in bytes will be stored in the database and anything larger will be be -put in object storage. This will allow a hybrid setup. If an xcom is stored on object storage a reference will be -saved in the database. Finally, you can set ``xcom_objectstorage_compression`` to fsspec supported compression methods like ``zip`` or ``snappy`` to -compress the data before storing it in object storage. - -So for example the following configuration will store anything above 1MB in S3 and will compress it using gzip:: - - [core] - xcom_backend = airflow.providers.common.io.xcom.backend.XComObjectStorageBackend - - [common.io] - xcom_objectstorage_path = s3://conn_id@mybucket/key - xcom_objectstorage_threshold = 1048576 - xcom_objectstorage_compression = gzip - - -.. note:: - - Compression requires the support for it is installed in your python environment. For example, to use ``snappy`` compression, you need to install ``python-snappy``. Zip, gzip and bz2 work out of the box. +The default XCom backend, BaseXCom, stores XComs in the Airflow database, which works well for small values but can cause issues with large values or a high volume of XComs. To overcome this limitation, object storage is recommended for efficiently handling larger data. For a detailed overview, refer to the :doc:`documentation <apache-airflow-providers-common-io:xcom_backend>`. Review Comment: Yes, that is already in the reference document, and there is no need to stay here while it was an example. I missed that. It is limited in the database for sure as a hard limit, but I think 1 MB was an example in this case. It depends on what you have set on the configuration, so it is on the application level. Let's not make this change -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
