Re: [PR] Add support for serverless job in Databricks operators [airflow]
pankajkoti commented on PR #45188: URL: https://github.com/apache/airflow/pull/45188#issuecomment-2635762360 Hi @HariGS-DB, Providers are typically released on a bi-weekly cycle, as outlined here: https://github.com/apache/airflow/blob/main/PROVIDERS.rst#community-providers-release-process. While it’s not possible to delay a release for a single PR, you can cast a negative vote if the provider release includes this PR and it introduces a breaking change. Release managers initiate a voting thread once the provider release candidates are created. If you’re not already subscribed to the dev list (https://airflow.apache.org/community/#mailing-list), I recommend doing so to stay informed about upcoming releases, allowing you to test and vote accordingly. In this case, it looks like the change is an addition rather than a regression, so proceeding with the release seems reasonable. Additional validation can be incorporated in the next release if needed, without delaying the current one. Does that sound good to you? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Add support for serverless job in Databricks operators [airflow]
HariGS-DB commented on PR #45188: URL: https://github.com/apache/airflow/pull/45188#issuecomment-2634927921 @pankajkoti Do you know when this PR will be released in the next airflow version? I will need to add another PR to introduce a validation for serverless (check for presence of environment_key in the json payload when submitting it for serverless ). I will submit this PR this week, is it possible to hold this PR in the next release until then? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Add support for serverless job in Databricks operators [airflow]
pankajkoti commented on PR #45188: URL: https://github.com/apache/airflow/pull/45188#issuecomment-2618823762 Thanks, @HariGS-DB, for the fantastic contribution! I’ve merged the PR. Since this was your first contribution, the CI pipeline required maintainer approval to run. Going forward, for your subsequent PRs, the CI pipelines will run automatically without needing approval. 🎉 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Add support for serverless job in Databricks operators [airflow]
boring-cyborg[bot] commented on PR #45188: URL: https://github.com/apache/airflow/pull/45188#issuecomment-2618820125 Awesome work, congrats on your first merged pull request! You are invited to check our [Issue Tracker](https://github.com/apache/airflow/issues) for additional contributions. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Add support for serverless job in Databricks operators [airflow]
pankajkoti merged PR #45188: URL: https://github.com/apache/airflow/pull/45188 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Add support for serverless job in Databricks operators [airflow]
HariGS-DB commented on PR #45188: URL: https://github.com/apache/airflow/pull/45188#issuecomment-2618661991 @pankajkoti I see you have pushed a new change which seem to have fixed the CI pipelines. Is there anything else need to be done for merging this. Please let me know -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Add support for serverless job in Databricks operators [airflow]
HariGS-DB commented on PR #45188: URL: https://github.com/apache/airflow/pull/45188#issuecomment-2616108920 @pankajkoti I have resolved the issues and the error check in the test and docs. But I am still getting a mypy error in the CI for code which I have not even touched. I might need some advice on how should I resolve them -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Add support for serverless job in Databricks operators [airflow]
HariGS-DB commented on PR #45188: URL: https://github.com/apache/airflow/pull/45188#issuecomment-2614513769 > @HariGS-DB some checks are failing in the CI. Could you please take care of that? And then we're good to merge I guess I have addressed some of the issues. Could you please re-check? I think it needs your approval before the complete tests are run. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Add support for serverless job in Databricks operators [airflow]
pankajkoti commented on PR #45188: URL: https://github.com/apache/airflow/pull/45188#issuecomment-2597562792 @HariGS-DB some checks are failing in the CI. Could you please take care of that? And then we're good to merge I guess -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Add support for serverless job in Databricks operators [airflow]
HariGS-DB commented on code in PR #45188: URL: https://github.com/apache/airflow/pull/45188#discussion_r1918554990 ## providers/src/airflow/providers/databricks/operators/databricks.py: ## @@ -292,6 +292,8 @@ class DatabricksCreateJobsOperator(BaseOperator): :param databricks_retry_delay: Number of seconds to wait between retries (it might be a floating point number). :param databricks_retry_args: An optional dictionary with arguments passed to ``tenacity.Retrying`` class. +:param databricks_environments: An optional list of task execution environment specifications Review Comment: have made the changes, please check -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Add support for serverless job in Databricks operators [airflow]
HariGS-DB commented on PR #45188: URL: https://github.com/apache/airflow/pull/45188#issuecomment-2595715484 > Thanks for the great addition @HariGS-DB . I have few minor suggestions. Happy to take another pass once you add your thoughts or have addressed the comments. > > Thanks @eladkal for requesting review here Hi @pankajkoti I have incorporated your review comments. I had to close and reopen the PR due to some merge conflicts which I have resolved. Please could you check and give me your new feedback -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Add support for serverless job in Databricks operators [airflow]
HariGS-DB commented on code in PR #45188:
URL: https://github.com/apache/airflow/pull/45188#discussion_r1918552802
##
providers/src/airflow/providers/databricks/operators/databricks.py:
##
@@ -1087,7 +1100,9 @@ def _get_run_json(self) -> dict[str, Any]:
elif self.existing_cluster_id:
run_json["existing_cluster_id"] = self.existing_cluster_id
else:
-raise ValueError("Must specify either existing_cluster_id or
new_cluster.")
Review Comment:
have made the changes, please check
##
providers/src/airflow/providers/databricks/operators/databricks.py:
##
@@ -1087,7 +1100,9 @@ def _get_run_json(self) -> dict[str, Any]:
elif self.existing_cluster_id:
run_json["existing_cluster_id"] = self.existing_cluster_id
else:
-raise ValueError("Must specify either existing_cluster_id or
new_cluster.")
+self.log.info("The task %s will be executed in serverless mode",
run_json["run_name"])
+if self.databricks_environments:
Review Comment:
have made the changes, please check
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] Add support for serverless job in Databricks operators [airflow]
HariGS-DB closed pull request #45188: Add support for serverless job in Databricks operators URL: https://github.com/apache/airflow/pull/45188 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Add support for serverless job in Databricks operators [airflow]
lennartkats-db commented on code in PR #45188: URL: https://github.com/apache/airflow/pull/45188#discussion_r1904099151 ## providers/src/airflow/providers/databricks/operators/databricks.py: ## @@ -292,6 +292,8 @@ class DatabricksCreateJobsOperator(BaseOperator): :param databricks_retry_delay: Number of seconds to wait between retries (it might be a floating point number). :param databricks_retry_args: An optional dictionary with arguments passed to ``tenacity.Retrying`` class. +:param databricks_environments: An optional list of task execution environment specifications Review Comment: +1, that name would also correspond with the name of this property in the Databricks API / SDK -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
