Re: [PR] Adding airflow operator to submit and monitor Spark Apps/App Env on DPGDC clusters [airflow]

akashsriv07 commented on PR #37223: URL: https://github.com/apache/airflow/pull/37223#issuecomment-2084416950 > Setting request changes to avoid accedental merge Will explore the feasibility on creating the provider and the process to be followed. Thanks -- This is an automated

Re: [PR] Adding airflow operator to submit and monitor Spark Apps/App Env on DPGDC clusters [airflow]

github-actions[bot] commented on PR #37223: URL: https://github.com/apache/airflow/pull/37223#issuecomment-2083907076 This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for

Re: [PR] Adding airflow operator to submit and monitor Spark Apps/App Env on DPGDC clusters [airflow]

eladkal commented on PR #37223: URL: https://github.com/apache/airflow/pull/37223#issuecomment-1995201673 > Hey, Michal from Cloud Composer team here. +1 for this code living in a separate, GDC provider - GDC is based on slightly different infra than GCP, and the operators won't

Re: [PR] Adding airflow operator to submit and monitor Spark Apps/App Env on DPGDC clusters [airflow]

michalmodras commented on code in PR #37223: URL: https://github.com/apache/airflow/pull/37223#discussion_r1518845852 ## tests/system/providers/google/gdc/dataprocgdc/example_dataprocgdc.py: ## @@ -0,0 +1,77 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +#

Re: [PR] Adding airflow operator to submit and monitor Spark Apps/App Env on DPGDC clusters [airflow]

MaksYermak commented on code in PR #37223: URL: https://github.com/apache/airflow/pull/37223#discussion_r1504363088 ## airflow/providers/google/cloud/operators/dataprocgdc.py: ## @@ -0,0 +1,272 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more

Re: [PR] Adding airflow operator to submit and monitor Spark Apps/App Env on DPGDC clusters [airflow]

MaksYermak commented on code in PR #37223: URL: https://github.com/apache/airflow/pull/37223#discussion_r1504336354 ## tests/system/providers/google/gdc/dataprocgdc/example_dataprocgdc.py: ## @@ -0,0 +1,77 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +#

Re: [PR] Adding airflow operator to submit and monitor Spark Apps/App Env on DPGDC clusters [airflow]

michalmodras commented on PR #37223: URL: https://github.com/apache/airflow/pull/37223#issuecomment-1964943387 Hey, Michal from Cloud Composer team here. +1 for this code living in a separate, GDC provider - GDC is based on slightly different infra than GCP, and the operators won't

Re: [PR] Adding airflow operator to submit and monitor Spark Apps/App Env on DPGDC clusters [airflow]

molcay commented on PR #37223: URL: https://github.com/apache/airflow/pull/37223#issuecomment-1964082744 Hey @akashsriv07, When I read the following sentence on [the product page](https://cloud.google.com/distributed-cloud#section-1), I thought that this product **is not directly in

Re: [PR] Adding airflow operator to submit and monitor Spark Apps/App Env on DPGDC clusters [airflow]

akashsriv07 commented on PR #37223: URL: https://github.com/apache/airflow/pull/37223#issuecomment-1963317023 > Hi @akashsriv07, > > I will propose a different approach for the folder/module structure. Since GDC might have different APIs, or different inner working mechanisms, it can

Re: [PR] Adding airflow operator to submit and monitor Spark Apps/App Env on DPGDC clusters [airflow]

molcay commented on PR #37223: URL: https://github.com/apache/airflow/pull/37223#issuecomment-1961153709 Hi @akashsriv07, I will propose a different approach for the folder/module structure. Since GDC might have different APIs, or different inner working mechanisms, it can be good

Re: [PR] Adding airflow operator to submit and monitor Spark Apps/App Env on DPGDC clusters [airflow]

akashsriv07 commented on code in PR #37223: URL: https://github.com/apache/airflow/pull/37223#discussion_r1499362176 ## airflow/providers/google/cloud/sensors/dataprocgdc.py: ## @@ -0,0 +1,127 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more

Re: [PR] Adding airflow operator to submit and monitor Spark Apps/App Env on DPGDC clusters [airflow]

MaksYermak commented on code in PR #37223: URL: https://github.com/apache/airflow/pull/37223#discussion_r1499352165 ## tests/system/providers/google/cloud/dataprocgdc/example_dataprocgdc.py: ## @@ -0,0 +1,77 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +#

Re: [PR] Adding airflow operator to submit and monitor Spark Apps/App Env on DPGDC clusters [airflow]

hussein-awala commented on PR #37223: URL: https://github.com/apache/airflow/pull/37223#issuecomment-1939739020 > Hey Hussein, Dataproc on GDC is yet to go GA and this is one of the critical features to add as part of the same. Hence we don't have a public doc on the same. You can take a

Re: [PR] Adding airflow operator to submit and monitor Spark Apps/App Env on DPGDC clusters [airflow]

akashsriv07 commented on PR #37223: URL: https://github.com/apache/airflow/pull/37223#issuecomment-1935231856 Hey Hussein, Dataproc on GDC is yet to go GA and this is one of the critical features to add as part of the same. Hence we don't have a public doc on the same. You can take a

Re: [PR] Adding airflow operator to submit and monitor Spark Apps/App Env on DPGDC clusters [airflow]

akashsriv07 commented on code in PR #37223: URL: https://github.com/apache/airflow/pull/37223#discussion_r1482416871 ## airflow/providers/google/cloud/operators/dataprocgdc.py: ## @@ -0,0 +1,300 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more

Re: [PR] Adding airflow operator to submit and monitor Spark Apps/App Env on DPGDC clusters [airflow]

dirrao commented on code in PR #37223: URL: https://github.com/apache/airflow/pull/37223#discussion_r1482409590 ## airflow/providers/google/cloud/operators/dataprocgdc.py: ## @@ -0,0 +1,300 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more