dabla commented on code in PR #64941:
URL: https://github.com/apache/airflow/pull/64941#discussion_r3459964895
##########
providers/common/sql/src/airflow/providers/common/sql/datafusion/object_storage_provider.py:
##########
@@ -70,18 +44,41 @@ def get_scheme(self) -> str:
return "file://"
+_STORAGE_TYPE_PROVIDER_HINTS: dict[str, str] = {
+ "s3": "apache-airflow-providers-amazon[datafusion]",
+}
+
+
def get_object_storage_provider(storage_type: StorageType) ->
ObjectStorageProvider:
"""Get an object storage provider based on the storage type."""
- # TODO: Add support for GCS, Azure, HTTP:
https://datafusion.apache.org/python/autoapi/datafusion/object_store/index.html
Review Comment:
I think here you will still need both implementations, as older Airflow
versions which don't have the `object_storage_providers` property in the
`ProvidersManager` might use the newer `common.sql` provider, so it cannot
solely depend on the `object_storage_providers` property of the
`ProvidersManager`.
So here to be backward compatible, you'll need to check the
`AIRFLOW_VERSION` or check if the `ProvidersManager` has the
`object_storage_providers` property. If not, then rely on the old code with
dict containing S3 and LOCAL (you'll probably need to do some local imports as
well to both providers).
If `object_storage_providers` property, then the current code can be used.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
