Re: [PR] Add MaxComputeSQLOperator, MaxComputeHook and AlibabaBaseHook to Alibaba Provider [airflow]
Lee-W merged PR #50178: URL: https://github.com/apache/airflow/pull/50178 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] Add MaxComputeSQLOperator, MaxComputeHook and AlibabaBaseHook to Alibaba Provider [airflow]
Lee-W commented on code in PR #50178:
URL: https://github.com/apache/airflow/pull/50178#discussion_r2110642244
##
providers/alibaba/src/airflow/providers/alibaba/cloud/hooks/maxcompute.py:
##
@@ -0,0 +1,248 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+
+import functools
+from typing import TYPE_CHECKING, Any, Callable, TypeVar
+
+from odps import ODPS
+
+from airflow.providers.alibaba.cloud.exceptions import
MaxComputeConfigurationException
+from airflow.providers.alibaba.cloud.hooks.base_alibaba import AlibabaBaseHook
+
+if TYPE_CHECKING:
+from odps.models import Instance
+
+RT = TypeVar("RT")
+
+
+def fallback_to_default_project_endpoint(func: Callable[..., RT]) ->
Callable[..., RT]:
+"""
+Provide fallback for MaxCompute project and endpoint to be used as a
decorator.
+
+If the project or endpoint is None it will be replaced with the project
from the
+connection extra definition.
+
+:param func: function to wrap
+:return: result of the function call
+"""
+
[email protected](func)
+def inner_wrapper(self, **kwargs) -> RT:
+kwargs["project"] = kwargs.get("project", self.project)
+kwargs["endpoint"] = kwargs.get("endpoint", self.endpoint)
+
+if not kwargs["project"]:
+raise MaxComputeConfigurationException(
+"The project must be passed either as "
+"keyword parameter or as extra "
+"in MaxCompute connection definition. Both are not set!"
+)
+
+if not kwargs["endpoint"]:
+raise MaxComputeConfigurationException(
+"The endpoint must be passed either as "
+"keyword parameter or as extra "
+"in MaxCompute connection definition. Both are not set!"
+)
+return func(self, **kwargs)
+
+return inner_wrapper
+
+
+class MaxComputeHook(AlibabaBaseHook):
+"""
+Interact with Alibaba MaxCompute (previously known as ODPS).
+
+:param maxcompute_conn_id: The connection ID to use when fetching
connection info.
+"""
+
+conn_name_attr = "maxcompute_conn_id"
+default_conn_name = "maxcompute_default"
+conn_type = "maxcompute"
+hook_name = "MaxCompute"
+
+def __init__(self, maxcompute_conn_id="maxcompute_default", **kwargs) ->
None:
Review Comment:
```suggestion
def __init__(self, maxcompute_conn_id: str="maxcompute_default",
**kwargs) -> None:
```
##
providers/alibaba/tests/unit/alibaba/cloud/hooks/test_maxcompute.py:
##
@@ -0,0 +1,117 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+
+from unittest import mock
+
+from airflow.providers.alibaba.cloud.hooks.maxcompute import MaxComputeHook
+
+MAXCOMPUTE_HOOK_MODULE = "airflow.providers.alibaba.cloud.hooks.maxcompute.{}"
+MOCK_MAXCOMPUTE_CONN_ID = "mock_id"
+MOCK_MAXCOMPUTE_PROJECT = "mock_project"
+MOCK_MAXCOMPUTE_ENDPOINT = "mock_endpoint"
+
+
+class TestMaxComputeHook:
+def setup_method(self):
+with mock.patch(
+MAXCOMPUTE_HOOK_MODULE.format("MaxComputeHook.get_connection"),
Review Comment:
We probably could move `MaxComputeHook` to `MAXCOMPUTE_HOOK_MODULE`
##
providers/alibaba/src/airflow/providers/alibaba/cloud/hooks/maxcompute.py:
##
@@ -0,0 +1,248 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.
Re: [PR] Add MaxComputeSQLOperator, MaxComputeHook and AlibabaBaseHook to Alibaba Provider [airflow]
jsjasonseba commented on PR #50178: URL: https://github.com/apache/airflow/pull/50178#issuecomment-2912739723 Made some adjustment to `conn_name_attr` for styling consistencies with other alibaba hooks -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] Add MaxComputeSQLOperator, MaxComputeHook and AlibabaBaseHook to Alibaba Provider [airflow]
Lee-W commented on code in PR #50178:
URL: https://github.com/apache/airflow/pull/50178#discussion_r2106640988
##
providers/alibaba/src/airflow/providers/alibaba/cloud/operators/maxcompute.py:
##
@@ -0,0 +1,146 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""This module contains Alibaba Cloud MaxCompute operators."""
+
+from __future__ import annotations
+
+from collections.abc import Sequence
+from typing import TYPE_CHECKING
+
+from airflow.models import BaseOperator
+from airflow.providers.alibaba.cloud.hooks.maxcompute import MaxComputeHook
+from airflow.providers.alibaba.cloud.links.maxcompute import
MaxComputeLogViewLink
+
+if TYPE_CHECKING:
+from odps.models import Instance
+
+from airflow.utils.context import Context
+
+
+class MaxComputeSQLOperator(BaseOperator):
+"""
+Executes an SQL statement in MaxCompute.
+
+Waits for the SQL task instance to complete and returns instance id.
+
+:param sql: The SQL statement to run.
+:param project: The project ID to use.
+:param endpoint: The endpoint to use.
+:param priority: The priority of the SQL statement ranges from 0 to 9,
+applicable to projects with the job priority feature enabled.
+Takes precedence over the `odps.instance.priority` setting from
`hints`.
+Defaults to 9.
+See
https://www.alibabacloud.com/help/en/maxcompute/user-guide/job-priority
+for details.
+:param running_cluster: The cluster to run the SQL statement on.
+:param hints: Hints for setting runtime parameters. See
+https://pyodps.readthedocs.io/en/latest/base-sql.html#id4 and
+
https://www.alibabacloud.com/help/en/maxcompute/user-guide/flag-parameters
+for details.
+:param aliases: Aliases for the SQL statement.
+:param default_schema: The default schema to use.
+:param quota_name: The quota name to use.
+Defaults to project default quota if not specified.
+:param alibabacloud_conn_id: The connection ID to use. Defaults to
+`alibabacloud_default` if not specified.
+:param cancel_on_kill: Flag which indicates whether to stop running
instance
+or not when task is killed. Default is True.
+"""
+
+template_fields: Sequence[str] = (
+"sql",
+"project",
+"endpoint",
+"priority",
+"running_cluster",
+"hints",
+"aliases",
+"default_schema",
+"quota_name",
+"alibabacloud_conn_id",
+)
+template_ext: Sequence[str] = (".sql",)
+template_fields_renderers = {"sql": "sql"}
+operator_extra_links = (MaxComputeLogViewLink(),)
+
+def __init__(
+self,
+*,
+sql: str,
+project: str | None = None,
Review Comment:
oh, if that's the case, let's use `project`. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] Add MaxComputeSQLOperator, MaxComputeHook and AlibabaBaseHook to Alibaba Provider [airflow]
jsjasonseba commented on code in PR #50178:
URL: https://github.com/apache/airflow/pull/50178#discussion_r2107159839
##
providers/alibaba/src/airflow/providers/alibaba/cloud/hooks/maxcompute.py:
##
@@ -0,0 +1,249 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+
+import functools
+from typing import TYPE_CHECKING, Any, Callable, TypeVar
+
+from odps import ODPS
+
+from airflow.providers.alibaba.cloud.exceptions import
MaxComputeConfigurationException
+from airflow.providers.alibaba.cloud.hooks.base_alibaba import AlibabaBaseHook
+
+if TYPE_CHECKING:
+from odps.models import Instance
+
+RT = TypeVar("RT")
+
+
+def fallback_to_default_project_endpoint(func: Callable[..., RT]) ->
Callable[..., RT]:
+"""
+Provide fallback for MaxCompute project and endpoint to be used as a
decorator.
+
+If the project or endpoint is None it will be replaced with the project
from the
+connection extra definition.
+
+:param func: function to wrap
+:return: result of the function call
+"""
+
[email protected](func)
+def inner_wrapper(self, *args, **kwargs) -> RT:
+if args:
Review Comment:
Oh right, it breaks anyway. Will refactor this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] Add MaxComputeSQLOperator, MaxComputeHook and AlibabaBaseHook to Alibaba Provider [airflow]
Lee-W commented on code in PR #50178:
URL: https://github.com/apache/airflow/pull/50178#discussion_r2106645550
##
providers/alibaba/src/airflow/providers/alibaba/cloud/hooks/maxcompute.py:
##
@@ -0,0 +1,249 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+
+import functools
+from typing import TYPE_CHECKING, Any, Callable, TypeVar
+
+from odps import ODPS
+
+from airflow.providers.alibaba.cloud.exceptions import
MaxComputeConfigurationException
+from airflow.providers.alibaba.cloud.hooks.base_alibaba import AlibabaBaseHook
+
+if TYPE_CHECKING:
+from odps.models import Instance
+
+RT = TypeVar("RT")
+
+
+def fallback_to_default_project_endpoint(func: Callable[..., RT]) ->
Callable[..., RT]:
+"""
+Provide fallback for MaxCompute project and endpoint to be used as a
decorator.
+
+If the project or endpoint is None it will be replaced with the project
from the
+connection extra definition.
+
+:param func: function to wrap
+:return: result of the function call
+"""
+
[email protected](func)
+def inner_wrapper(self, *args, **kwargs) -> RT:
+if args:
Review Comment:
Will it still work for functions with positional args if we change it into
`def inner_wrapper(self, **kwargs)`? I thought it's going to break
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] Add MaxComputeSQLOperator, MaxComputeHook and AlibabaBaseHook to Alibaba Provider [airflow]
jsjasonseba commented on code in PR #50178:
URL: https://github.com/apache/airflow/pull/50178#discussion_r2102786740
##
providers/alibaba/src/airflow/providers/alibaba/cloud/operators/maxcompute.py:
##
@@ -0,0 +1,146 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""This module contains Alibaba Cloud MaxCompute operators."""
+
+from __future__ import annotations
+
+from collections.abc import Sequence
+from typing import TYPE_CHECKING
+
+from airflow.models import BaseOperator
+from airflow.providers.alibaba.cloud.hooks.maxcompute import MaxComputeHook
+from airflow.providers.alibaba.cloud.links.maxcompute import
MaxComputeLogViewLink
+
+if TYPE_CHECKING:
+from odps.models import Instance
+
+from airflow.utils.context import Context
+
+
+class MaxComputeSQLOperator(BaseOperator):
+"""
+Executes an SQL statement in MaxCompute.
+
+Waits for the SQL task instance to complete and returns instance id.
+
+:param sql: The SQL statement to run.
+:param project: The project ID to use.
+:param endpoint: The endpoint to use.
+:param priority: The priority of the SQL statement ranges from 0 to 9,
+applicable to projects with the job priority feature enabled.
+Takes precedence over the `odps.instance.priority` setting from
`hints`.
+Defaults to 9.
+See
https://www.alibabacloud.com/help/en/maxcompute/user-guide/job-priority
+for details.
+:param running_cluster: The cluster to run the SQL statement on.
+:param hints: Hints for setting runtime parameters. See
+https://pyodps.readthedocs.io/en/latest/base-sql.html#id4 and
+
https://www.alibabacloud.com/help/en/maxcompute/user-guide/flag-parameters
+for details.
+:param aliases: Aliases for the SQL statement.
+:param default_schema: The default schema to use.
+:param quota_name: The quota name to use.
+Defaults to project default quota if not specified.
+:param alibabacloud_conn_id: The connection ID to use. Defaults to
+`alibabacloud_default` if not specified.
+:param cancel_on_kill: Flag which indicates whether to stop running
instance
+or not when task is killed. Default is True.
+"""
+
+template_fields: Sequence[str] = (
+"sql",
+"project",
+"endpoint",
+"priority",
+"running_cluster",
+"hints",
+"aliases",
+"default_schema",
+"quota_name",
+"alibabacloud_conn_id",
+)
+template_ext: Sequence[str] = (".sql",)
+template_fields_renderers = {"sql": "sql"}
+operator_extra_links = (MaxComputeLogViewLink(),)
+
+def __init__(
+self,
+*,
+sql: str,
+project: str | None = None,
Review Comment:
`project` is based on the parameter from the `pyodps`
[client](https://pyodps.readthedocs.io/en/latest/#quick-start). I do agree
though that `project_id` is a better name
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] Add MaxComputeSQLOperator, MaxComputeHook and AlibabaBaseHook to Alibaba Provider [airflow]
jsjasonseba commented on code in PR #50178:
URL: https://github.com/apache/airflow/pull/50178#discussion_r2102786740
##
providers/alibaba/src/airflow/providers/alibaba/cloud/operators/maxcompute.py:
##
@@ -0,0 +1,146 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""This module contains Alibaba Cloud MaxCompute operators."""
+
+from __future__ import annotations
+
+from collections.abc import Sequence
+from typing import TYPE_CHECKING
+
+from airflow.models import BaseOperator
+from airflow.providers.alibaba.cloud.hooks.maxcompute import MaxComputeHook
+from airflow.providers.alibaba.cloud.links.maxcompute import
MaxComputeLogViewLink
+
+if TYPE_CHECKING:
+from odps.models import Instance
+
+from airflow.utils.context import Context
+
+
+class MaxComputeSQLOperator(BaseOperator):
+"""
+Executes an SQL statement in MaxCompute.
+
+Waits for the SQL task instance to complete and returns instance id.
+
+:param sql: The SQL statement to run.
+:param project: The project ID to use.
+:param endpoint: The endpoint to use.
+:param priority: The priority of the SQL statement ranges from 0 to 9,
+applicable to projects with the job priority feature enabled.
+Takes precedence over the `odps.instance.priority` setting from
`hints`.
+Defaults to 9.
+See
https://www.alibabacloud.com/help/en/maxcompute/user-guide/job-priority
+for details.
+:param running_cluster: The cluster to run the SQL statement on.
+:param hints: Hints for setting runtime parameters. See
+https://pyodps.readthedocs.io/en/latest/base-sql.html#id4 and
+
https://www.alibabacloud.com/help/en/maxcompute/user-guide/flag-parameters
+for details.
+:param aliases: Aliases for the SQL statement.
+:param default_schema: The default schema to use.
+:param quota_name: The quota name to use.
+Defaults to project default quota if not specified.
+:param alibabacloud_conn_id: The connection ID to use. Defaults to
+`alibabacloud_default` if not specified.
+:param cancel_on_kill: Flag which indicates whether to stop running
instance
+or not when task is killed. Default is True.
+"""
+
+template_fields: Sequence[str] = (
+"sql",
+"project",
+"endpoint",
+"priority",
+"running_cluster",
+"hints",
+"aliases",
+"default_schema",
+"quota_name",
+"alibabacloud_conn_id",
+)
+template_ext: Sequence[str] = (".sql",)
+template_fields_renderers = {"sql": "sql"}
+operator_extra_links = (MaxComputeLogViewLink(),)
+
+def __init__(
+self,
+*,
+sql: str,
+project: str | None = None,
Review Comment:
`project` is based on the parameter from the `pyodps` client. I do agree
though that `project_id` is a better name
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] Add MaxComputeSQLOperator, MaxComputeHook and AlibabaBaseHook to Alibaba Provider [airflow]
jsjasonseba commented on code in PR #50178:
URL: https://github.com/apache/airflow/pull/50178#discussion_r2102793305
##
providers/alibaba/src/airflow/providers/alibaba/cloud/hooks/maxcompute.py:
##
@@ -0,0 +1,249 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+
+import functools
+from typing import TYPE_CHECKING, Any, Callable, TypeVar
+
+from odps import ODPS
+
+from airflow.providers.alibaba.cloud.exceptions import
MaxComputeConfigurationException
+from airflow.providers.alibaba.cloud.hooks.base_alibaba import AlibabaBaseHook
+
+if TYPE_CHECKING:
+from odps.models import Instance
+
+RT = TypeVar("RT")
+
+
+def fallback_to_default_project_endpoint(func: Callable[..., RT]) ->
Callable[..., RT]:
+"""
+Provide fallback for MaxCompute project and endpoint to be used as a
decorator.
+
+If the project or endpoint is None it will be replaced with the project
from the
+connection extra definition.
+
+:param func: function to wrap
+:return: result of the function call
+"""
+
[email protected](func)
+def inner_wrapper(self, *args, **kwargs) -> RT:
+if args:
Review Comment:
but, shouldn't we keep it in case someone use this decorator on non
keyword-only methods?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] Add MaxComputeSQLOperator, MaxComputeHook and AlibabaBaseHook to Alibaba Provider [airflow]
jsjasonseba commented on code in PR #50178:
URL: https://github.com/apache/airflow/pull/50178#discussion_r2102783069
##
providers/alibaba/src/airflow/providers/alibaba/cloud/hooks/maxcompute.py:
##
@@ -0,0 +1,249 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+
+import functools
+from typing import TYPE_CHECKING, Any, Callable, TypeVar
+
+from odps import ODPS
+
+from airflow.providers.alibaba.cloud.exceptions import
MaxComputeConfigurationException
+from airflow.providers.alibaba.cloud.hooks.base_alibaba import AlibabaBaseHook
+
+if TYPE_CHECKING:
+from odps.models import Instance
+
+RT = TypeVar("RT")
+
+
+def fallback_to_default_project_endpoint(func: Callable[..., RT]) ->
Callable[..., RT]:
+"""
+Provide fallback for MaxCompute project and endpoint to be used as a
decorator.
+
+If the project or endpoint is None it will be replaced with the project
from the
+connection extra definition.
+
+:param func: function to wrap
+:return: result of the function call
+"""
+
[email protected](func)
+def inner_wrapper(self, *args, **kwargs) -> RT:
+if args:
Review Comment:
right, it is not necessary since it is used on keyword-only methods anyway
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] Add MaxComputeSQLOperator, MaxComputeHook and AlibabaBaseHook to Alibaba Provider [airflow]
jsjasonseba commented on code in PR #50178:
URL: https://github.com/apache/airflow/pull/50178#discussion_r2102783069
##
providers/alibaba/src/airflow/providers/alibaba/cloud/hooks/maxcompute.py:
##
@@ -0,0 +1,249 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+
+import functools
+from typing import TYPE_CHECKING, Any, Callable, TypeVar
+
+from odps import ODPS
+
+from airflow.providers.alibaba.cloud.exceptions import
MaxComputeConfigurationException
+from airflow.providers.alibaba.cloud.hooks.base_alibaba import AlibabaBaseHook
+
+if TYPE_CHECKING:
+from odps.models import Instance
+
+RT = TypeVar("RT")
+
+
+def fallback_to_default_project_endpoint(func: Callable[..., RT]) ->
Callable[..., RT]:
+"""
+Provide fallback for MaxCompute project and endpoint to be used as a
decorator.
+
+If the project or endpoint is None it will be replaced with the project
from the
+connection extra definition.
+
+:param func: function to wrap
+:return: result of the function call
+"""
+
[email protected](func)
+def inner_wrapper(self, *args, **kwargs) -> RT:
+if args:
Review Comment:
right, it is not necessary since it is keyword-only anyway
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] Add MaxComputeSQLOperator, MaxComputeHook and AlibabaBaseHook to Alibaba Provider [airflow]
Lee-W commented on code in PR #50178:
URL: https://github.com/apache/airflow/pull/50178#discussion_r2101480966
##
providers/alibaba/src/airflow/providers/alibaba/cloud/operators/maxcompute.py:
##
@@ -0,0 +1,146 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""This module contains Alibaba Cloud MaxCompute operators."""
+
+from __future__ import annotations
+
+from collections.abc import Sequence
+from typing import TYPE_CHECKING
+
+from airflow.models import BaseOperator
+from airflow.providers.alibaba.cloud.hooks.maxcompute import MaxComputeHook
+from airflow.providers.alibaba.cloud.links.maxcompute import
MaxComputeLogViewLink
+
+if TYPE_CHECKING:
+from odps.models import Instance
+
+from airflow.utils.context import Context
+
+
+class MaxComputeSQLOperator(BaseOperator):
+"""
+Executes an SQL statement in MaxCompute.
+
+Waits for the SQL task instance to complete and returns instance id.
+
+:param sql: The SQL statement to run.
+:param project: The project ID to use.
+:param endpoint: The endpoint to use.
+:param priority: The priority of the SQL statement ranges from 0 to 9,
+applicable to projects with the job priority feature enabled.
+Takes precedence over the `odps.instance.priority` setting from
`hints`.
+Defaults to 9.
+See
https://www.alibabacloud.com/help/en/maxcompute/user-guide/job-priority
+for details.
+:param running_cluster: The cluster to run the SQL statement on.
+:param hints: Hints for setting runtime parameters. See
+https://pyodps.readthedocs.io/en/latest/base-sql.html#id4 and
+
https://www.alibabacloud.com/help/en/maxcompute/user-guide/flag-parameters
+for details.
+:param aliases: Aliases for the SQL statement.
+:param default_schema: The default schema to use.
+:param quota_name: The quota name to use.
+Defaults to project default quota if not specified.
+:param alibabacloud_conn_id: The connection ID to use. Defaults to
+`alibabacloud_default` if not specified.
+:param cancel_on_kill: Flag which indicates whether to stop running
instance
+or not when task is killed. Default is True.
+"""
+
+template_fields: Sequence[str] = (
+"sql",
+"project",
+"endpoint",
+"priority",
+"running_cluster",
+"hints",
+"aliases",
+"default_schema",
+"quota_name",
+"alibabacloud_conn_id",
+)
+template_ext: Sequence[str] = (".sql",)
+template_fields_renderers = {"sql": "sql"}
+operator_extra_links = (MaxComputeLogViewLink(),)
+
+def __init__(
+self,
+*,
+sql: str,
+project: str | None = None,
Review Comment:
If the arg name `project` is not a convention yet, `project_id` might be
more accurate
##
providers/alibaba/src/airflow/providers/alibaba/cloud/operators/maxcompute.py:
##
@@ -0,0 +1,146 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""This module contains Alibaba Cloud MaxCompute operators."""
+
+from __future__ import annotations
+
+from collections.abc import Sequence
+from typing import TYPE_CHECKING
+
+from airflow.models import BaseOperator
+from airflow.providers.alibaba.cloud.hooks.maxcompute import MaxComputeHook
+from airflow.providers.alibaba.cloud.links.maxcompute import
MaxComputeLogViewLink
+
+if TYPE_CHECKING:
+from odps.models import Instance
+
+
Re: [PR] Add MaxComputeSQLOperator, MaxComputeHook and AlibabaBaseHook to Alibaba Provider [airflow]
Lee-W commented on code in PR #50178:
URL: https://github.com/apache/airflow/pull/50178#discussion_r2101480028
##
providers/alibaba/src/airflow/providers/alibaba/cloud/hooks/maxcompute.py:
##
@@ -0,0 +1,249 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+
+import functools
+from typing import TYPE_CHECKING, Any, Callable, TypeVar
+
+from odps import ODPS
+
+from airflow.providers.alibaba.cloud.exceptions import
MaxComputeConfigurationException
+from airflow.providers.alibaba.cloud.hooks.base_alibaba import AlibabaBaseHook
+
+if TYPE_CHECKING:
+from odps.models import Instance
+
+RT = TypeVar("RT")
+
+
+def fallback_to_default_project_endpoint(func: Callable[..., RT]) ->
Callable[..., RT]:
+"""
+Provide fallback for MaxCompute project and endpoint to be used as a
decorator.
+
+If the project or endpoint is None it will be replaced with the project
from the
+connection extra definition.
+
+:param func: function to wrap
+:return: result of the function call
+"""
+
[email protected](func)
+def inner_wrapper(self, *args, **kwargs) -> RT:
+if args:
Review Comment:
Why not try something like `def inner_wrapper(self, **kwargs)`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] Add MaxComputeSQLOperator, MaxComputeHook and AlibabaBaseHook to Alibaba Provider [airflow]
jsjasonseba commented on code in PR #50178: URL: https://github.com/apache/airflow/pull/50178#discussion_r2100572459 ## providers/alibaba/src/airflow/providers/alibaba/cloud/hooks/base_alibaba.py: ## @@ -0,0 +1,100 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +from __future__ import annotations + +from typing import Any, NamedTuple + +from airflow.hooks.base import BaseHook + + +class AccessKeyCredentials(NamedTuple): +""" +A NamedTuple to store Alibaba Cloud Access Key credentials. + +:param access_key_id: The Access Key ID for Alibaba Cloud authentication. +:param access_key_secret: The Access Key Secret for Alibaba Cloud authentication. +""" + +access_key_id: str +access_key_secret: str + + +class AlibabaBaseHook(BaseHook): +""" +A base hook for Alibaba Cloud-related hooks. + +This hook provides a common interface for authenticating using Alibaba Cloud credentials. + +Supports Access Key-based authentication. + +:param alibaba_cloud_conn_id: The connection ID to use when fetching connection info. +""" + +conn_name_attr = "alibabacloud_conn_id" Review Comment: It stands for alibaba cloud conn id. I didn’t choose the nam. I followed the naming convention used in other existing Alibaba hooks for consistency. https://github.com/jsjasonseba/airflow/blob/8b5f0cf343f4c61bf67dc4d5d8a17f865ca8/providers/alibaba/src/airflow/providers/alibaba/cloud/hooks/oss.py#L79-L82 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] Add MaxComputeSQLOperator, MaxComputeHook and AlibabaBaseHook to Alibaba Provider [airflow]
jsjasonseba commented on code in PR #50178:
URL: https://github.com/apache/airflow/pull/50178#discussion_r2100703254
##
providers/alibaba/src/airflow/providers/alibaba/cloud/hooks/maxcompute.py:
##
@@ -0,0 +1,250 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+
+import functools
+from typing import TYPE_CHECKING, Any, Callable, TypeVar
+
+from odps import ODPS
+
+from airflow.exceptions import AirflowException
+from airflow.providers.alibaba.cloud.hooks.base_alibaba import AlibabaBaseHook
+
+if TYPE_CHECKING:
+from odps.models import Instance
+
+RT = TypeVar("RT")
+
+
+def fallback_to_default_project_endpoint(func: Callable[..., RT]) ->
Callable[..., RT]:
+"""
+Provide fallback for MaxCompute project and endpoint to be used as a
decorator.
+
+If the project or endpoint is None it will be replaced with the project
from the
+connection extra definition.
+
+:param func: function to wrap
+:return: result of the function call
+"""
+
[email protected](func)
+def inner_wrapper(self, *args, **kwargs) -> RT:
+if args:
+raise AirflowException("You must use keyword arguments in this
methods rather than positional")
+if "project" in kwargs:
+kwargs["project"] = kwargs["project"] or self.project
+else:
+kwargs["project"] = self.project
+
+if "endpoint" in kwargs:
+kwargs["endpoint"] = kwargs["endpoint"] or self.endpoint
+else:
+kwargs["endpoint"] = self.endpoint
+
+if not kwargs["project"]:
+raise AirflowException(
+"The project must be passed either as "
+"keyword parameter or as extra "
+"in MaxCompute connection definition. Both are not set!"
+)
+
+if not kwargs["endpoint"]:
+raise AirflowException(
+"The endpoint must be passed either as "
+"keyword parameter or as extra "
+"in MaxCompute connection definition. Both are not set!"
+)
+return func(self, *args, **kwargs)
+
+return inner_wrapper
+
+
+class MaxComputeHook(AlibabaBaseHook):
+"""
+Interact with Alibaba MaxCompute (previously known as ODPS).
+
+:param maxcompute_conn_id: The connection ID to use when fetching
connection info.
+"""
+
+conn_name_attr = "alibabacloud_conn_id"
+default_conn_name = "alibabacloud_default"
+conn_type = "maxcompute"
+hook_name = "MaxCompute"
+
+@classmethod
+def get_connection_form_widgets(cls) -> dict[str, Any]:
+"""Return connection widgets to add to connection form."""
+from flask_appbuilder.fieldwidgets import BS3TextFieldWidget
+from flask_babel import lazy_gettext
+from wtforms import StringField
+
+connection_form_widgets = super().get_connection_form_widgets()
+
+connection_form_widgets["project"] = StringField(
+lazy_gettext("Project"),
+widget=BS3TextFieldWidget(),
+)
+connection_form_widgets["endpoint"] = StringField(
+lazy_gettext("Endpoint"),
+widget=BS3TextFieldWidget(),
+)
+
+return connection_form_widgets
+
+@classmethod
+def get_ui_field_behaviour(cls) -> dict[str, Any]:
+"""Return custom field behaviour."""
+return {
+"hidden_fields": ["host", "schema", "login", "password", "port",
"extra"],
+"relabeling": {},
+}
+
+@property
+def project(self) -> str:
+"""
+Returns project ID.
+
+:return: ID of the project
+"""
+return self._get_field("project")
+
+@property
+def endpoint(self) -> str:
+"""
+Returns MaxCompute Endpoint.
+
+:return: Endpoint of the MaxCompute project
+"""
+return self._get_field("endpoint")
+
+@fallback_to_default_project_endpoint
+def get_client(self, project: str, endpoint: str) -> ODPS:
+"""
+Get an authenticated MaxCompute ODPS Client.
+
+:param project_id: Project ID for the project which the client acts on
behalf of.
+:param lo
Re: [PR] Add MaxComputeSQLOperator, MaxComputeHook and AlibabaBaseHook to Alibaba Provider [airflow]
jsjasonseba commented on code in PR #50178: URL: https://github.com/apache/airflow/pull/50178#discussion_r2100572459 ## providers/alibaba/src/airflow/providers/alibaba/cloud/hooks/base_alibaba.py: ## @@ -0,0 +1,100 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +from __future__ import annotations + +from typing import Any, NamedTuple + +from airflow.hooks.base import BaseHook + + +class AccessKeyCredentials(NamedTuple): +""" +A NamedTuple to store Alibaba Cloud Access Key credentials. + +:param access_key_id: The Access Key ID for Alibaba Cloud authentication. +:param access_key_secret: The Access Key Secret for Alibaba Cloud authentication. +""" + +access_key_id: str +access_key_secret: str + + +class AlibabaBaseHook(BaseHook): +""" +A base hook for Alibaba Cloud-related hooks. + +This hook provides a common interface for authenticating using Alibaba Cloud credentials. + +Supports Access Key-based authentication. + +:param alibaba_cloud_conn_id: The connection ID to use when fetching connection info. +""" + +conn_name_attr = "alibabacloud_conn_id" Review Comment: `conn_name_attr` is derived from `BaseHook` for mapping connection type to hook. I believe it should refer to the parameter name for assigning the connection id in the `__init__`. Most hooks I see assigns that var. https://airflow.apache.org/docs/apache-airflow/stable/_api/airflow/hooks/base/index.html#airflow.hooks.base.DiscoverableHook The value stands for alibaba cloud conn id. I didn’t choose the name. I followed the naming convention used in other existing Alibaba hooks for consistency. https://github.com/jsjasonseba/airflow/blob/8b5f0cf343f4c61bf67dc4d5d8a17f865ca8/providers/alibaba/src/airflow/providers/alibaba/cloud/hooks/oss.py#L79-L82 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] Add MaxComputeSQLOperator, MaxComputeHook and AlibabaBaseHook to Alibaba Provider [airflow]
jsjasonseba commented on code in PR #50178:
URL: https://github.com/apache/airflow/pull/50178#discussion_r2100603355
##
providers/alibaba/src/airflow/providers/alibaba/cloud/hooks/base_alibaba.py:
##
@@ -0,0 +1,100 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+
+from typing import Any, NamedTuple
+
+from airflow.hooks.base import BaseHook
+
+
+class AccessKeyCredentials(NamedTuple):
+"""
+A NamedTuple to store Alibaba Cloud Access Key credentials.
+
+:param access_key_id: The Access Key ID for Alibaba Cloud authentication.
+:param access_key_secret: The Access Key Secret for Alibaba Cloud
authentication.
+"""
+
+access_key_id: str
+access_key_secret: str
+
+
+class AlibabaBaseHook(BaseHook):
+"""
+A base hook for Alibaba Cloud-related hooks.
+
+This hook provides a common interface for authenticating using Alibaba
Cloud credentials.
+
+Supports Access Key-based authentication.
+
+:param alibaba_cloud_conn_id: The connection ID to use when fetching
connection info.
+"""
+
+conn_name_attr = "alibabacloud_conn_id"
+default_conn_name = "alibabacloud_default"
+conn_type = "alibaba_cloud"
+hook_name = "Alibaba Cloud"
+
+def __init__(
+self,
+alibabacloud_conn_id: str = "alibabacloud_default",
+**kwargs,
+) -> None:
+super().__init__(**kwargs)
+self.alibaba_cloud_conn_id = alibabacloud_conn_id
+self.extras: dict =
self.get_connection(self.alibaba_cloud_conn_id).extra_dejson
+
+@classmethod
+def get_connection_form_widgets(cls) -> dict[str, Any]:
+"""Return connection widgets to add to connection form."""
+from flask_appbuilder.fieldwidgets import BS3PasswordFieldWidget
+from flask_babel import lazy_gettext
+from wtforms import PasswordField
+
+return {
+"access_key_id": PasswordField(lazy_gettext("Access Key ID"),
widget=BS3PasswordFieldWidget()),
+"access_key_secret": PasswordField(
+lazy_gettext("Access Key Secret"),
widget=BS3PasswordFieldWidget()
+),
+}
+
+@classmethod
+def get_ui_field_behaviour(cls) -> dict[str, Any]:
+"""Return custom field behaviour."""
+return super().get_ui_field_behaviour()
+
+def _get_field(self, field_name: str, default: Any = None) -> Any:
+"""Fetch a field from extras, and returns it."""
+extras = getattr(self, "extras", {})
Review Comment:
yeah, I guess this is unnecessary since it is always assigned from `init`. I
was taking this implementation from `GoogleBaseHook` but I don't get the
rationale either. Let me remove it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] Add MaxComputeSQLOperator, MaxComputeHook and AlibabaBaseHook to Alibaba Provider [airflow]
jsjasonseba commented on code in PR #50178: URL: https://github.com/apache/airflow/pull/50178#discussion_r2100572459 ## providers/alibaba/src/airflow/providers/alibaba/cloud/hooks/base_alibaba.py: ## @@ -0,0 +1,100 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +from __future__ import annotations + +from typing import Any, NamedTuple + +from airflow.hooks.base import BaseHook + + +class AccessKeyCredentials(NamedTuple): +""" +A NamedTuple to store Alibaba Cloud Access Key credentials. + +:param access_key_id: The Access Key ID for Alibaba Cloud authentication. +:param access_key_secret: The Access Key Secret for Alibaba Cloud authentication. +""" + +access_key_id: str +access_key_secret: str + + +class AlibabaBaseHook(BaseHook): +""" +A base hook for Alibaba Cloud-related hooks. + +This hook provides a common interface for authenticating using Alibaba Cloud credentials. + +Supports Access Key-based authentication. + +:param alibaba_cloud_conn_id: The connection ID to use when fetching connection info. +""" + +conn_name_attr = "alibabacloud_conn_id" Review Comment: It stands for alibaba cloud conn id. I didn’t choose the name. I followed the naming convention used in other existing Alibaba hooks for consistency. https://github.com/jsjasonseba/airflow/blob/8b5f0cf343f4c61bf67dc4d5d8a17f865ca8/providers/alibaba/src/airflow/providers/alibaba/cloud/hooks/oss.py#L79-L82 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] Add MaxComputeSQLOperator, MaxComputeHook and AlibabaBaseHook to Alibaba Provider [airflow]
Lee-W commented on code in PR #50178:
URL: https://github.com/apache/airflow/pull/50178#discussion_r2095305175
##
providers/alibaba/src/airflow/providers/alibaba/cloud/hooks/base_alibaba.py:
##
@@ -0,0 +1,100 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+
+from typing import Any, NamedTuple
+
+from airflow.hooks.base import BaseHook
+
+
+class AccessKeyCredentials(NamedTuple):
+"""
+A NamedTuple to store Alibaba Cloud Access Key credentials.
+
+:param access_key_id: The Access Key ID for Alibaba Cloud authentication.
+:param access_key_secret: The Access Key Secret for Alibaba Cloud
authentication.
+"""
+
+access_key_id: str
+access_key_secret: str
+
+
+class AlibabaBaseHook(BaseHook):
+"""
+A base hook for Alibaba Cloud-related hooks.
+
+This hook provides a common interface for authenticating using Alibaba
Cloud credentials.
+
+Supports Access Key-based authentication.
+
+:param alibaba_cloud_conn_id: The connection ID to use when fetching
connection info.
+"""
+
+conn_name_attr = "alibabacloud_conn_id"
Review Comment:
I'm not sure I understand what `conn_name_attr` is 🤔 Is it correctly named
##
providers/alibaba/src/airflow/providers/alibaba/cloud/hooks/maxcompute.py:
##
@@ -0,0 +1,250 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+
+import functools
+from typing import TYPE_CHECKING, Any, Callable, TypeVar
+
+from odps import ODPS
+
+from airflow.exceptions import AirflowException
+from airflow.providers.alibaba.cloud.hooks.base_alibaba import AlibabaBaseHook
+
+if TYPE_CHECKING:
+from odps.models import Instance
+
+RT = TypeVar("RT")
+
+
+def fallback_to_default_project_endpoint(func: Callable[..., RT]) ->
Callable[..., RT]:
+"""
+Provide fallback for MaxCompute project and endpoint to be used as a
decorator.
+
+If the project or endpoint is None it will be replaced with the project
from the
+connection extra definition.
+
+:param func: function to wrap
+:return: result of the function call
+"""
+
[email protected](func)
+def inner_wrapper(self, *args, **kwargs) -> RT:
+if args:
+raise AirflowException("You must use keyword arguments in this
methods rather than positional")
+if "project" in kwargs:
+kwargs["project"] = kwargs["project"] or self.project
+else:
+kwargs["project"] = self.project
+
+if "endpoint" in kwargs:
+kwargs["endpoint"] = kwargs["endpoint"] or self.endpoint
+else:
+kwargs["endpoint"] = self.endpoint
+
+if not kwargs["project"]:
+raise AirflowException(
Review Comment:
same here
##
providers/alibaba/src/airflow/providers/alibaba/cloud/hooks/maxcompute.py:
##
@@ -0,0 +1,250 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the Lice
Re: [PR] Add MaxComputeSQLOperator, MaxComputeHook and AlibabaBaseHook to Alibaba Provider [airflow]
jsjasonseba commented on PR #50178: URL: https://github.com/apache/airflow/pull/50178#issuecomment-2872055138 cc @potiuk @kaxil -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] Add MaxComputeSQLOperator, MaxComputeHook and AlibabaBaseHook to Alibaba Provider [airflow]
jsjasonseba commented on PR #50178: URL: https://github.com/apache/airflow/pull/50178#issuecomment-2869524629 Hi @eladkal, would you mind reviewing this PR when you have a chance? I saw that you previously reviewed changes related to the Alibaba provider, so your input here would be valuable. Thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
