Re: [PR] AIP-58: Add Airflow ObjectStore (AS) [airflow]

2023-10-21 Thread via GitHub
hussein-awala commented on code in PR #34729: URL: https://github.com/apache/airflow/pull/34729#discussion_r1367713729 ## airflow/io/__init__.py: ## @@ -0,0 +1,90 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the

Re: [PR] AIP-58: Add Airflow ObjectStore (AS) [airflow]

2023-10-20 Thread via GitHub
potiuk commented on code in PR #34729: URL: https://github.com/apache/airflow/pull/34729#discussion_r1367433860 ## docs/apache-airflow/extra-packages-ref.rst: ## @@ -68,6 +68,8 @@ python dependencies for the provided package. +-+

Re: [PR] AIP-58: Add Airflow ObjectStore (AS) [airflow]

2023-10-20 Thread via GitHub
bolkedebruin commented on code in PR #34729: URL: https://github.com/apache/airflow/pull/34729#discussion_r1367427142 ## docs/apache-airflow/extra-packages-ref.rst: ## @@ -68,6 +68,8 @@ python dependencies for the provided package. +-+--

Re: [PR] AIP-58: Add Airflow ObjectStore (AS) [airflow]

2023-10-20 Thread via GitHub
potiuk commented on PR #34729: URL: https://github.com/apache/airflow/pull/34729#issuecomment-1772775554 You should be able to rebase + resolve conflicts now @bolkedebruin for the PROD image -- This is an automated message from the Apache Git Service. To respond to the message, please lo

Re: [PR] AIP-58: Add Airflow ObjectStore (AS) [airflow]

2023-10-19 Thread via GitHub
jens-scheffler-bosch commented on code in PR #34729: URL: https://github.com/apache/airflow/pull/34729#discussion_r1366184196 ## airflow/operators/file_transfer.py: ## Review Comment: If we create a new provider package (I am okay with this but have no strong opinion, also

Re: [PR] AIP-58: Add Airflow ObjectStore (AS) [airflow]

2023-10-19 Thread via GitHub
bolkedebruin commented on code in PR #34729: URL: https://github.com/apache/airflow/pull/34729#discussion_r1366077501 ## setup.py: ## @@ -496,6 +496,12 @@ def write_version(filename: str = str(AIRFLOW_SOURCES_ROOT / "airflow" / "git_ve "aiobotocore>=2.1.1", ] +s3fs = [

Re: [PR] AIP-58: Add Airflow ObjectStore (AS) [airflow]

2023-10-19 Thread via GitHub
bolkedebruin commented on PR #34729: URL: https://github.com/apache/airflow/pull/34729#issuecomment-1771543580 > In order to fix the CI failures - you need to add some exceptions in our compatibility test suite: > > * https://github.com/apache/airflow/blob/main/.github/workflows/

Re: [PR] AIP-58: Add Airflow ObjectStore (AS) [airflow]

2023-10-19 Thread via GitHub
potiuk commented on code in PR #34729: URL: https://github.com/apache/airflow/pull/34729#discussion_r1365937670 ## setup.py: ## @@ -496,6 +496,12 @@ def write_version(filename: str = str(AIRFLOW_SOURCES_ROOT / "airflow" / "git_ve "aiobotocore>=2.1.1", ] +s3fs = [ Revie

Re: [PR] AIP-58: Add Airflow ObjectStore (AS) [airflow]

2023-10-19 Thread via GitHub
potiuk commented on PR #34729: URL: https://github.com/apache/airflow/pull/34729#issuecomment-1771479607 In order to fix the CI failures - you need to add some exceptions in our compatibility test suite: * https://github.com/apache/airflow/blob/main/.github/workflows/ci.yml#L829 - fo

Re: [PR] AIP-58: Add Airflow ObjectStore (AS) [airflow]

2023-10-19 Thread via GitHub
bolkedebruin commented on code in PR #34729: URL: https://github.com/apache/airflow/pull/34729#discussion_r1365930524 ## airflow/providers/amazon/aws/fs/s3.py: ## @@ -0,0 +1,131 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agree

Re: [PR] AIP-58: Add Airflow ObjectStore (AS) [airflow]

2023-10-19 Thread via GitHub
bolkedebruin commented on code in PR #34729: URL: https://github.com/apache/airflow/pull/34729#discussion_r1365919381 ## setup.py: ## @@ -496,6 +496,12 @@ def write_version(filename: str = str(AIRFLOW_SOURCES_ROOT / "airflow" / "git_ve "aiobotocore>=2.1.1", ] +s3fs = [

Re: [PR] AIP-58: Add Airflow ObjectStore (AS) [airflow]

2023-10-19 Thread via GitHub
bolkedebruin commented on code in PR #34729: URL: https://github.com/apache/airflow/pull/34729#discussion_r1365919061 ## airflow/providers/amazon/aws/fs/s3.py: ## @@ -0,0 +1,131 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agree

Re: [PR] AIP-58: Add Airflow ObjectStore (AS) [airflow]

2023-10-19 Thread via GitHub
bolkedebruin commented on code in PR #34729: URL: https://github.com/apache/airflow/pull/34729#discussion_r1365911705 ## setup.py: ## @@ -496,6 +496,12 @@ def write_version(filename: str = str(AIRFLOW_SOURCES_ROOT / "airflow" / "git_ve "aiobotocore>=2.1.1", ] +s3fs = [

Re: [PR] AIP-58: Add Airflow ObjectStore (AS) [airflow]

2023-10-19 Thread via GitHub
bolkedebruin commented on code in PR #34729: URL: https://github.com/apache/airflow/pull/34729#discussion_r1365908132 ## airflow/providers/amazon/aws/fs/s3.py: ## @@ -0,0 +1,131 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agree

Re: [PR] AIP-58: Add Airflow ObjectStore (AS) [airflow]

2023-10-19 Thread via GitHub
potiuk commented on code in PR #34729: URL: https://github.com/apache/airflow/pull/34729#discussion_r1365876794 ## airflow/providers/common/io/provider.yaml: ## @@ -0,0 +1,34 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreemen

Re: [PR] AIP-58: Add Airflow ObjectStore (AS) [airflow]

2023-10-19 Thread via GitHub
potiuk commented on code in PR #34729: URL: https://github.com/apache/airflow/pull/34729#discussion_r1365864940 ## setup.py: ## @@ -496,6 +496,12 @@ def write_version(filename: str = str(AIRFLOW_SOURCES_ROOT / "airflow" / "git_ve "aiobotocore>=2.1.1", ] +s3fs = [ Revie

Re: [PR] AIP-58: Add Airflow ObjectStore (AS) [airflow]

2023-10-19 Thread via GitHub
potiuk commented on code in PR #34729: URL: https://github.com/apache/airflow/pull/34729#discussion_r1365861818 ## airflow/io/store/path.py: ## @@ -0,0 +1,666 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOT

Re: [PR] AIP-58: Add Airflow ObjectStore (AS) [airflow]

2023-10-19 Thread via GitHub
bolkedebruin commented on code in PR #34729: URL: https://github.com/apache/airflow/pull/34729#discussion_r1365834600 ## airflow/providers/amazon/aws/fs/s3.py: ## @@ -0,0 +1,131 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agree

Re: [PR] AIP-58: Add Airflow ObjectStore (AS) [airflow]

2023-10-19 Thread via GitHub
bolkedebruin commented on code in PR #34729: URL: https://github.com/apache/airflow/pull/34729#discussion_r1365791887 ## airflow/providers/amazon/aws/fs/s3.py: ## @@ -0,0 +1,131 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agree

Re: [PR] AIP-58: Add Airflow ObjectStore (AS) [airflow]

2023-10-19 Thread via GitHub
bolkedebruin commented on code in PR #34729: URL: https://github.com/apache/airflow/pull/34729#discussion_r1365774144 ## airflow/providers/amazon/aws/fs/s3.py: ## @@ -0,0 +1,131 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agree

Re: [PR] AIP-58: Add Airflow ObjectStore (AS) [airflow]

2023-10-19 Thread via GitHub
Taragolis commented on code in PR #34729: URL: https://github.com/apache/airflow/pull/34729#discussion_r1365821201 ## airflow/providers/amazon/aws/fs/s3.py: ## @@ -0,0 +1,131 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreemen

Re: [PR] AIP-58: Add Airflow ObjectStore (AS) [airflow]

2023-10-19 Thread via GitHub
bolkedebruin commented on code in PR #34729: URL: https://github.com/apache/airflow/pull/34729#discussion_r1365820648 ## airflow/io/store/path.py: ## @@ -0,0 +1,666 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See t

Re: [PR] AIP-58: Add Airflow ObjectStore (AS) [airflow]

2023-10-19 Thread via GitHub
Taragolis commented on code in PR #34729: URL: https://github.com/apache/airflow/pull/34729#discussion_r1365808618 ## airflow/providers/amazon/aws/fs/s3.py: ## @@ -0,0 +1,131 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreemen

Re: [PR] AIP-58: Add Airflow ObjectStore (AS) [airflow]

2023-10-19 Thread via GitHub
potiuk commented on code in PR #34729: URL: https://github.com/apache/airflow/pull/34729#discussion_r1365793654 ## airflow/io/store/path.py: ## @@ -0,0 +1,666 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOT

Re: [PR] AIP-58: Add Airflow ObjectStore (AS) [airflow]

2023-10-19 Thread via GitHub
potiuk commented on code in PR #34729: URL: https://github.com/apache/airflow/pull/34729#discussion_r1365792355 ## docs/apache-airflow/extra-packages-ref.rst: ## @@ -68,6 +68,8 @@ python dependencies for the provided package. +-+

Re: [PR] AIP-58: Add Airflow ObjectStore (AS) [airflow]

2023-10-19 Thread via GitHub
bolkedebruin commented on code in PR #34729: URL: https://github.com/apache/airflow/pull/34729#discussion_r1365791887 ## airflow/providers/amazon/aws/fs/s3.py: ## @@ -0,0 +1,131 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agree

Re: [PR] AIP-58: Add Airflow ObjectStore (AS) [airflow]

2023-10-19 Thread via GitHub
potiuk commented on code in PR #34729: URL: https://github.com/apache/airflow/pull/34729#discussion_r1365787115 ## airflow/providers/common/io/provider.yaml: ## @@ -0,0 +1,34 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreemen

Re: [PR] AIP-58: Add Airflow ObjectStore (AS) [airflow]

2023-10-19 Thread via GitHub
bolkedebruin commented on code in PR #34729: URL: https://github.com/apache/airflow/pull/34729#discussion_r1365788548 ## airflow/providers/common/io/provider.yaml: ## @@ -0,0 +1,34 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license ag

Re: [PR] AIP-58: Add Airflow ObjectStore (AS) [airflow]

2023-10-19 Thread via GitHub
potiuk commented on code in PR #34729: URL: https://github.com/apache/airflow/pull/34729#discussion_r1365787115 ## airflow/providers/common/io/provider.yaml: ## @@ -0,0 +1,34 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreemen

Re: [PR] AIP-58: Add Airflow ObjectStore (AS) [airflow]

2023-10-19 Thread via GitHub
bolkedebruin commented on code in PR #34729: URL: https://github.com/apache/airflow/pull/34729#discussion_r1365774144 ## airflow/providers/amazon/aws/fs/s3.py: ## @@ -0,0 +1,131 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agree

Re: [PR] AIP-58: Add Airflow ObjectStore (AS) [airflow]

2023-10-19 Thread via GitHub
bolkedebruin commented on code in PR #34729: URL: https://github.com/apache/airflow/pull/34729#discussion_r1365756321 ## airflow/providers/amazon/aws/fs/s3.py: ## @@ -0,0 +1,131 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agree

Re: [PR] AIP-58: Add Airflow ObjectStore (AS) [airflow]

2023-10-19 Thread via GitHub
Taragolis commented on code in PR #34729: URL: https://github.com/apache/airflow/pull/34729#discussion_r1365755441 ## docs/apache-airflow/extra-packages-ref.rst: ## @@ -68,6 +68,8 @@ python dependencies for the provided package. +-+-

Re: [PR] AIP-58: Add Airflow ObjectStore (AS) [airflow]

2023-10-19 Thread via GitHub
Taragolis commented on code in PR #34729: URL: https://github.com/apache/airflow/pull/34729#discussion_r1365752756 ## docs/apache-airflow/extra-packages-ref.rst: ## @@ -68,6 +68,8 @@ python dependencies for the provided package. +-+-

Re: [PR] AIP-58: Add Airflow ObjectStore (AS) [airflow]

2023-10-19 Thread via GitHub
Taragolis commented on code in PR #34729: URL: https://github.com/apache/airflow/pull/34729#discussion_r1365701214 ## airflow/providers/amazon/aws/fs/s3.py: ## @@ -0,0 +1,131 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreemen

Re: [PR] AIP-58: Add Airflow ObjectStore (AS) [airflow]

2023-10-19 Thread via GitHub
bolkedebruin commented on code in PR #34729: URL: https://github.com/apache/airflow/pull/34729#discussion_r1365467670 ## airflow/operators/file_transfer.py: ## Review Comment: My thoughts: * The FileTransferOperator really is a thin wrapper around airflow.io.store.p

Re: [PR] AIP-58: Add Airflow ObjectStore (AS) [airflow]

2023-10-19 Thread via GitHub
bolkedebruin commented on code in PR #34729: URL: https://github.com/apache/airflow/pull/34729#discussion_r1365467670 ## airflow/operators/file_transfer.py: ## Review Comment: My thoughts: * The FileTransferOperator really is a thin wrapper around airflow.io.store.p

Re: [PR] AIP-58: Add Airflow ObjectStore (AS) [airflow]

2023-10-19 Thread via GitHub
eladkal commented on code in PR #34729: URL: https://github.com/apache/airflow/pull/34729#discussion_r1365405287 ## airflow/operators/file_transfer.py: ## Review Comment: I mentioned it earlier but just to double check. Are we sure we want this operator in core? It me