Hi all, While reviewing #76408 [1] I noticed it was the fourth fix of the same bug.
Object stores create a zero-byte "folder marker" for every folder. A transfer operator that lists keys and copies them one by one either fails on the marker or copies it as an empty object. The same rule (skip a marker that has keys under it) was applied to S3ToGCSOperator [2], GCSToS3Operator [3], AzureBlobStorageToS3Operator [4] and S3ToAzureBlobStorageOperator [1]. That is four PRs in three providers over six months, and main now carries three copies of the same helper, because provider packages cannot share code without a version floor. Nobody did anything wrong there. The structure leaves no other option. The structure is one operator per source/destination pair. On main today I count 86 transfer modules, about 17,500 lines, in 13 providers (google 37, amazon 21, microsoft.azure 6, apache.hive 6, and nine smaller ones). Sorted by what they do, roughly: * 28 copy files from one store to another (S3, GCS, Azure Blob, ADLS, SFTP, FTP, local, Drive, Samba) * 16 move rows from one database to another * 11 load files into a database * 26 read records from a database or an API and write them to files (SqlToS3, HttpToS3, HttpToGCS, MongoToS3, SalesforceToGCS, and BaseSQLToGCSOperator with six dialect subclasses) * 5 send records elsewhere (Slack, Sheets) Every fix and every feature (pagination, a file format, async, lineage) lands in one of them and is missing from the others. HttpOperator can paginate, for example; HttpToS3Operator and HttpToGCSOperator cannot. We have been here before, and we solved it. Until 2022 every database provider shipped its own operator. common.sql 1.3 replaced them with SQLExecuteQueryOperator on top of the DbApiHook contract, the dialect moved into the hooks, and the specific operators became deprecated shells. AIP-14 "Create composable operators using common interfaces" [5] asked for the same thing for transfers in 2019: hooks behind a common interface and one copy operator that takes a source hook and a destination hook. It was only partly implemented. Its file half landed as AIP-58 [6], which gave us ObjectStoragePath, and FileTransferOperator in common.io, since Airflow 2.8. The AIP-14 page is marked "Superseded & Completed" by AIP-58 since 2024, but the goal it was written for, no more X-to-Y operators, was never reached. So AIP-14 is incomplete in two ways. 1. The migration never happened for files. FileTransferOperator copies between any two registered filesystems, but the 28 pair operators were never rebased on it or deprecated. They are still what users find first, and what we keep fixing one by one. 2. There is no contract for sources that produce records instead of files: paginated HTTP/REST and SaaS APIs, and SQL when the destination is a file. AIP-14 mentions databases in one line and AIP-58 did not cover them. This is where new pair operators keep coming from. In my own team the latest request was an "MSGraphToS3Operator". The direction I would like to discuss, not a design yet: * Reading in batches, pagination included, is a capability of the hook, as get_df_by_chunks is for SQL. #75649 [7] proposes the same for MS Graph. * One generic operator connects a source hook to a destination hook. GenericTransfer looks like the natural starting point: it already takes source_hook_params and destination_hook_params for the two hooks and insert_args for the destination call. With the same on the source side, the arguments of each side are forwarded to that side's hook and checked when the task runs. The operator knows neither vocabulary. * File destinations go through ObjectStoragePath, so every registered filesystem works. * The existing pair operators become thin shells and are deprecated on the usual schedule, as the SQL operators were. * Out of scope here: Aldo is preparing a dedicated proposal on optimising GenericTransfer between ADBC-enabled hooks, moving Arrow record batches end to end instead of Python rows. That proposal stands on its own and I do not want to mix the two threads. It fits this direction well, since it also puts the capability in the hooks and keeps the operator generic. My questions to the list: 1. Do we agree that the per-pair operators are a maintenance problem worth solving, and should we stop accepting new ones in the meantime? 2. Can step one happen without any new design: rebase the file-to-file operators on ObjectStoragePath and deprecate them in favour of FileTransferOperator? 3. For record sources, where should the contract and the operator live: common.sql next to GenericTransfer, common.io next to FileTransferOperator, or somewhere new? 4. Does this need a new AIP? I think it does. AIP-14 is seven years old and closed, so a new one would reference it and cover the part that was never implemented. I also remember TP presenting something similar at Airflow Summit last year, I believe in "Assets: Past, Present, Future" [8], where Connection, Object Storage and Asset were tied closer together. TP, please correct me if I remember it wrong; I am especially interested in how this would fit with what you have in mind. I am willing to drive this: write the AIP if the list agrees one is needed, and implement it. It touches many providers though, so everyone who wants to contribute or help is very welcome. The author of the four folder-marker fixes has already offered to join [1]. Best regards, David [1] https://github.com/apache/airflow/pull/76408 [2] https://github.com/apache/airflow/pull/65724 [3] https://github.com/apache/airflow/pull/72497 [4] https://github.com/apache/airflow/pull/72892 [5] https://cwiki.apache.org/confluence/spaces/AIRFLOW/pages/103091837/AIP-14+Create+composable+operators+using+common+interfaces [6] https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=263430565 [7] https://github.com/apache/airflow/pull/75649 [8] https://airflowsummit.org/sessions/2025/assets-past-present-future/ General (Internal Property)
