Re: [PR] AIP-58: Add Airflow Store (AS) [airflow]

bolkedebruin commented on PR #34729: URL: https://github.com/apache/airflow/pull/34729#issuecomment-1770451078 @uranusjr @jens-scheffler-bosch This is ready for review I think. * Caching of filesystems registration and late initalization of providers is now implemented * Se

Re: [PR] AIP-58: Add Airflow Store (AS) [airflow]

bolkedebruin commented on code in PR #34729: URL: https://github.com/apache/airflow/pull/34729#discussion_r1363452843 ## docs/apache-airflow/extra-packages-ref.rst: ## @@ -68,6 +68,8 @@ python dependencies for the provided package. +-+--

Re: [PR] AIP-58: Add Airflow Store (AS) [airflow]

bolkedebruin commented on code in PR #34729: URL: https://github.com/apache/airflow/pull/34729#discussion_r1363410748 ## airflow/operators/file_transfer.py: ## @@ -0,0 +1,76 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreem

Re: [PR] AIP-58: Add Airflow Store (AS) [airflow]

bolkedebruin commented on code in PR #34729: URL: https://github.com/apache/airflow/pull/34729#discussion_r1363452843 ## docs/apache-airflow/extra-packages-ref.rst: ## @@ -68,6 +68,8 @@ python dependencies for the provided package. +-+--

Re: [PR] AIP-58: Add Airflow Store (AS) [airflow]

bolkedebruin commented on code in PR #34729: URL: https://github.com/apache/airflow/pull/34729#discussion_r1363423468 ## airflow/io/fsspec.py: ## @@ -0,0 +1,237 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the N

Re: [PR] AIP-58: Add Airflow Store (AS) [airflow]

bolkedebruin commented on code in PR #34729: URL: https://github.com/apache/airflow/pull/34729#discussion_r1363414834 ## airflow/io/__init__.py: ## @@ -0,0 +1,270 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the

Re: [PR] AIP-58: Add Airflow Store (AS) [airflow]

bolkedebruin commented on code in PR #34729: URL: https://github.com/apache/airflow/pull/34729#discussion_r1363410748 ## airflow/operators/file_transfer.py: ## @@ -0,0 +1,76 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreem

Re: [PR] AIP-58: Add Airflow Store (AS) [airflow]

jens-scheffler-bosch commented on code in PR #34729: URL: https://github.com/apache/airflow/pull/34729#discussion_r1363410409 ## airflow/io/fsspec.py: ## @@ -0,0 +1,237 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. S

Re: [PR] AIP-58: Add Airflow Store (AS) [airflow]

uranusjr commented on code in PR #34729: URL: https://github.com/apache/airflow/pull/34729#discussion_r1363407384 ## airflow/io/fsspec.py: ## @@ -0,0 +1,237 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTIC

Re: [PR] AIP-58: Add Airflow Store (AS) [airflow]

uranusjr commented on code in PR #34729: URL: https://github.com/apache/airflow/pull/34729#discussion_r1363387465 ## docs/apache-airflow/extra-packages-ref.rst: ## @@ -68,6 +68,8 @@ python dependencies for the provided package. +-+--

Re: [PR] AIP-58: Add Airflow Store (AS) [airflow]

jens-scheffler-bosch commented on code in PR #34729: URL: https://github.com/apache/airflow/pull/34729#discussion_r1363374666 ## airflow/operators/file_transfer.py: ## @@ -0,0 +1,76 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor licens

Re: [PR] AIP-58: Add Airflow Store (AS) [airflow]

uranusjr commented on code in PR #34729: URL: https://github.com/apache/airflow/pull/34729#discussion_r1363374194 ## airflow/io/__init__.py: ## @@ -0,0 +1,270 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOT

Re: [PR] AIP-58: Add Airflow Store (AS) [airflow]

uranusjr commented on code in PR #34729: URL: https://github.com/apache/airflow/pull/34729#discussion_r1363371466 ## airflow/operators/file_transfer.py: ## @@ -0,0 +1,76 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements

Re: [PR] AIP-58: Add Airflow Store (AS) [airflow]

bolkedebruin commented on code in PR #34729: URL: https://github.com/apache/airflow/pull/34729#discussion_r1363341578 ## docs/apache-airflow/extra-packages-ref.rst: ## @@ -68,6 +68,8 @@ python dependencies for the provided package. +-+--

Re: [PR] AIP-58: Add Airflow Store (AS) [airflow]

bolkedebruin commented on PR #34729: URL: https://github.com/apache/airflow/pull/34729#issuecomment-1767787735 > As a general comment, I feel this PR still contains too many different parts. It would be a good idea to split it into several ones that gradually introduce differnt concepts (e.

Re: [PR] AIP-58: Add Airflow Store (AS) [airflow]

bolkedebruin commented on code in PR #34729: URL: https://github.com/apache/airflow/pull/34729#discussion_r1363339168 ## airflow/io/__init__.py: ## @@ -0,0 +1,270 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the

Re: [PR] AIP-58: Add Airflow Store (AS) [airflow]

bolkedebruin commented on code in PR #34729: URL: https://github.com/apache/airflow/pull/34729#discussion_r1363337386 ## airflow/operators/file_transfer.py: ## @@ -0,0 +1,76 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreem

Re: [PR] AIP-58: Add Airflow Store (AS) [airflow]

bolkedebruin commented on code in PR #34729: URL: https://github.com/apache/airflow/pull/34729#discussion_r1363332718 ## airflow/io/store/path.py: ## @@ -0,0 +1,666 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See t

Re: [PR] AIP-58: Add Airflow Store (AS) [airflow]

bolkedebruin commented on code in PR #34729: URL: https://github.com/apache/airflow/pull/34729#discussion_r1363332718 ## airflow/io/store/path.py: ## @@ -0,0 +1,666 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See t

Re: [PR] AIP-58: Add Airflow Store (AS) [airflow]

bolkedebruin commented on code in PR #34729: URL: https://github.com/apache/airflow/pull/34729#discussion_r1363326815 ## airflow/io/store/path.py: ## @@ -0,0 +1,666 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See t

Re: [PR] AIP-58: Add Airflow Store (AS) [airflow]

bolkedebruin commented on code in PR #34729: URL: https://github.com/apache/airflow/pull/34729#discussion_r1363325324 ## airflow/io/store/path.py: ## @@ -0,0 +1,666 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See t

Re: [PR] AIP-58: Add Airflow Store (AS) [airflow]

bolkedebruin commented on code in PR #34729: URL: https://github.com/apache/airflow/pull/34729#discussion_r1363322869 ## airflow/io/__init__.py: ## @@ -0,0 +1,270 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the

Re: [PR] AIP-58: Add Airflow Store (AS) [airflow]

bolkedebruin commented on code in PR #34729: URL: https://github.com/apache/airflow/pull/34729#discussion_r1363321824 ## airflow/io/store/path.py: ## @@ -0,0 +1,666 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See t

Re: [PR] AIP-58: Add Airflow Store (AS) [airflow]

bolkedebruin commented on code in PR #34729: URL: https://github.com/apache/airflow/pull/34729#discussion_r1363318509 ## airflow/io/fsspec.py: ## @@ -0,0 +1,237 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the N

Re: [PR] AIP-58: Add Airflow Store (AS) [airflow]

uranusjr commented on PR #34729: URL: https://github.com/apache/airflow/pull/34729#issuecomment-1767611203 As a general comment, I feel this PR still contains too many different parts. It would be a good idea to split it into several ones that gradually introduce differnt concepts (e.g. may

Re: [PR] AIP-58: Add Airflow Store (AS) [airflow]

uranusjr commented on code in PR #34729: URL: https://github.com/apache/airflow/pull/34729#discussion_r1363146090 ## airflow/io/__init__.py: ## @@ -0,0 +1,270 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOT

Re: [PR] AIP-58: Add Airflow Store (AS) [airflow]

uranusjr commented on code in PR #34729: URL: https://github.com/apache/airflow/pull/34729#discussion_r1363146090 ## airflow/io/__init__.py: ## @@ -0,0 +1,270 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOT

Re: [PR] AIP-58: Add Airflow Store (AS) [airflow]

uranusjr commented on code in PR #34729: URL: https://github.com/apache/airflow/pull/34729#discussion_r1363146090 ## airflow/io/__init__.py: ## @@ -0,0 +1,270 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOT

Re: [PR] AIP-58: Add Airflow Store (AS) [airflow]

uranusjr commented on code in PR #34729: URL: https://github.com/apache/airflow/pull/34729#discussion_r1363142789 ## airflow/operators/file_transfer.py: ## @@ -0,0 +1,76 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements

Re: [PR] AIP-58: Add Airflow Store (AS) [airflow]

uranusjr commented on code in PR #34729: URL: https://github.com/apache/airflow/pull/34729#discussion_r1363135218 ## airflow/io/store/path.py: ## @@ -0,0 +1,666 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the N

Re: [PR] AIP-58: Add Airflow Store (AS) [airflow]

uranusjr commented on code in PR #34729: URL: https://github.com/apache/airflow/pull/34729#discussion_r1363133996 ## airflow/io/store/path.py: ## @@ -0,0 +1,666 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the N

Re: [PR] AIP-58: Add Airflow Store (AS) [airflow]

uranusjr commented on code in PR #34729: URL: https://github.com/apache/airflow/pull/34729#discussion_r1363130848 ## airflow/io/store/path.py: ## @@ -0,0 +1,666 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the N

Re: [PR] AIP-58: Add Airflow Store (AS) [airflow]

uranusjr commented on code in PR #34729: URL: https://github.com/apache/airflow/pull/34729#discussion_r1363124027 ## airflow/io/store/path.py: ## @@ -0,0 +1,666 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the N

Re: [PR] AIP-58: Add Airflow Store (AS) [airflow]

uranusjr commented on code in PR #34729: URL: https://github.com/apache/airflow/pull/34729#discussion_r1363114610 ## airflow/io/fsspec.py: ## @@ -0,0 +1,237 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTIC

Re: [PR] AIP-58: Add Airflow Store (AS) [airflow]

uranusjr commented on code in PR #34729: URL: https://github.com/apache/airflow/pull/34729#discussion_r1363114610 ## airflow/io/fsspec.py: ## @@ -0,0 +1,237 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTIC

Re: [PR] AIP-58: Add Airflow Store (AS) [airflow]

uranusjr commented on code in PR #34729: URL: https://github.com/apache/airflow/pull/34729#discussion_r1363061652 ## airflow/io/__init__.py: ## @@ -0,0 +1,270 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOT

Re: [PR] AIP-58: Add Airflow Store (AS) [airflow]

uranusjr commented on code in PR #34729: URL: https://github.com/apache/airflow/pull/34729#discussion_r1363060858 ## airflow/io/__init__.py: ## @@ -0,0 +1,270 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOT

Re: [PR] AIP-58: Add Airflow Store (AS) [airflow]

uranusjr commented on code in PR #34729: URL: https://github.com/apache/airflow/pull/34729#discussion_r1363058325 ## docs/apache-airflow/extra-packages-ref.rst: ## @@ -68,6 +68,8 @@ python dependencies for the provided package. +-+--

Re: [PR] AIP-58: Add Airflow Store (AS) [airflow]

uranusjr commented on code in PR #34729: URL: https://github.com/apache/airflow/pull/34729#discussion_r1363058325 ## docs/apache-airflow/extra-packages-ref.rst: ## @@ -68,6 +68,8 @@ python dependencies for the provided package. +-+--

Re: [PR] AIP-58: Add Airflow Store (AS) [airflow]

uranusjr commented on code in PR #34729: URL: https://github.com/apache/airflow/pull/34729#discussion_r1363056637 ## docs/apache-airflow/extra-packages-ref.rst: ## @@ -68,6 +68,8 @@ python dependencies for the provided package. +-+--

Re: [PR] AIP-58: Add Airflow Store (AS) [airflow]

uranusjr commented on code in PR #34729: URL: https://github.com/apache/airflow/pull/34729#discussion_r1363054879 ## airflow/io/__init__.py: ## @@ -0,0 +1,270 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOT

Re: [PR] AIP-58: Add Airflow Store (AS) [airflow]

uranusjr commented on code in PR #34729: URL: https://github.com/apache/airflow/pull/34729#discussion_r1363053865 ## airflow/io/__init__.py: ## @@ -0,0 +1,270 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOT

Re: [PR] AIP-58: Add Airflow Store (AS) [airflow]

uranusjr commented on code in PR #34729: URL: https://github.com/apache/airflow/pull/34729#discussion_r1363050368 ## airflow/io/__init__.py: ## @@ -0,0 +1,270 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOT

Re: [PR] AIP-58: Add Airflow Store (AS) [airflow]

bolkedebruin commented on code in PR #34729: URL: https://github.com/apache/airflow/pull/34729#discussion_r1362621961 ## airflow/io/store/util.py: ## @@ -0,0 +1,79 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See th

Re: [PR] AIP-58: Add Airflow Store (AS) [airflow]

bolkedebruin commented on code in PR #34729: URL: https://github.com/apache/airflow/pull/34729#discussion_r1362620600 ## airflow/io/store/__init__.py: ## @@ -0,0 +1,125 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. S

Re: [PR] AIP-58: Add Airflow Store (AS) [airflow]

bolkedebruin commented on code in PR #34729: URL: https://github.com/apache/airflow/pull/34729#discussion_r1362619850 ## airflow/io/__init__.py: ## @@ -0,0 +1,297 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the

Re: [PR] AIP-58: Add Airflow Store (AS) [airflow]

bolkedebruin commented on code in PR #34729: URL: https://github.com/apache/airflow/pull/34729#discussion_r1362619000 ## airflow/io/fsspec.py: ## @@ -0,0 +1,237 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the N

Re: [PR] AIP-58: Add Airflow Store (AS) [airflow]

bolkedebruin commented on code in PR #34729: URL: https://github.com/apache/airflow/pull/34729#discussion_r1362619000 ## airflow/io/fsspec.py: ## @@ -0,0 +1,237 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the N

Re: [PR] AIP-58: Add Airflow Store (AS) [airflow]

bolkedebruin commented on code in PR #34729: URL: https://github.com/apache/airflow/pull/34729#discussion_r1362617992 ## airflow/io/__init__.py: ## @@ -0,0 +1,297 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the

Re: [PR] AIP-58: Add Airflow Store (AS) [airflow]

bolkedebruin commented on code in PR #34729: URL: https://github.com/apache/airflow/pull/34729#discussion_r1362615725 ## airflow/io/__init__.py: ## @@ -0,0 +1,297 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the

Re: [PR] AIP-58: Add Airflow Store (AS) [airflow]

bolkedebruin commented on code in PR #34729: URL: https://github.com/apache/airflow/pull/34729#discussion_r1362607462 ## airflow/io/__init__.py: ## @@ -0,0 +1,297 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the

Re: [PR] AIP-58: Add Airflow Store (AS) [airflow]

jens-scheffler-bosch commented on code in PR #34729: URL: https://github.com/apache/airflow/pull/34729#discussion_r1362568182 ## airflow/io/__init__.py: ## @@ -0,0 +1,297 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements.

Re: [PR] AIP-58: Add Airflow Store (AS) [airflow]

bolkedebruin commented on PR #34729: URL: https://github.com/apache/airflow/pull/34729#issuecomment-1766388128 @uranusjr @jens-scheffler-bosch @ashb PTL. I have changed the code now so that ObjectStoragePath can be used as a PathLike object with some extensions. See also the updated initia

Re: [PR] AIP-58: Add Airflow Store (AS) [airflow]

bolkedebruin commented on PR #34729: URL: https://github.com/apache/airflow/pull/34729#issuecomment-1757001233 > It is necessary to keep track of mounts anyway, but my concern is this is much too low level to expose to DAG authors. The approach I would prefer is to give as little control as

Re: [PR] AIP-58: Add Airflow Store (AS) [airflow]

uranusjr commented on PR #34729: URL: https://github.com/apache/airflow/pull/34729#issuecomment-1756938101 It is necessary to keep track of mounts anyway, but my concern is this is much too low level to expose to DAG authors. The approach I would prefer is to give as little control as possi

Re: [PR] AIP-58: Add Airflow Store (AS) [airflow]

jens-scheffler-bosch commented on code in PR #34729: URL: https://github.com/apache/airflow/pull/34729#discussion_r1353202717 ## airflow/io/fs/__init__.py: ## @@ -0,0 +1,987 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreement