Taragolis commented on code in PR #37458:
URL: https://github.com/apache/airflow/pull/37458#discussion_r1491369765
##########
airflow/providers/yandex/provider.yaml:
##########
@@ -62,25 +61,41 @@ integrations:
logo: /integration-logos/yandex/Yandex-Cloud.png
tags: [service]
+ - integration-name: Yandex.Cloud YQ
+ external-doc-url: https://cloud.yandex.com/en/services/query
+ how-to-guide:
+ - /docs/apache-airflow-providers-yandex/operators.rst
+ logo: /integration-logos/yandex/Yandex-Cloud.png
+ tags: [service]
+
operators:
- integration-name: Yandex.Cloud Dataproc
python-modules:
- airflow.providers.yandex.operators.yandexcloud_dataproc
+ - integration-name: Yandex.Cloud YQ
+ python-modules:
+ - airflow.providers.yandex.operators.yandexcloud_yq
+
hooks:
- integration-name: Yandex.Cloud
python-modules:
- airflow.providers.yandex.hooks.yandex
- integration-name: Yandex.Cloud Dataproc
python-modules:
- airflow.providers.yandex.hooks.yandexcloud_dataproc
+ - integration-name: Yandex.Cloud YQ
+ python-modules:
+ - airflow.providers.yandex.hooks.yandexcloud_yq
+ - airflow.providers.yandex.hooks.http_client
+ - airflow.providers.yandex.hooks.query_results
connection-types:
- hook-class-name: airflow.providers.yandex.hooks.yandex.YandexCloudBaseHook
connection-type: yandexcloud
-secrets-backends:
- - airflow.providers.yandex.secrets.lockbox.LockboxSecretBackend
Review Comment:
I guess you accidentally delete support of Secrets Backend
##########
airflow/providers/yandex/hooks/http_client.py:
##########
Review Comment:
This doesn't look like Airflow Hook to me
##########
airflow/providers/yandex/hooks/query_results.py:
##########
Review Comment:
This doesn't look as Airflow Hook to me
##########
airflow/providers/yandex/hooks/http_client.py:
##########
@@ -0,0 +1,321 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# This file is a copy of
https://github.com/ydb-platform/ydb/tree/284b7efb67edcdade0b12c849b7fad40739ad62b/ydb/core/fq/libs/http_api_client
+# It is highly recommended to modify original file first in YDB project and
merge it here afterwards
Review Comment:
And are you check that original licence allow to copy the code to another
project?
##########
airflow/providers/yandex/hooks/http_client.py:
##########
@@ -0,0 +1,321 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# This file is a copy of
https://github.com/ydb-platform/ydb/tree/284b7efb67edcdade0b12c849b7fad40739ad62b/ydb/core/fq/libs/http_api_client
+# It is highly recommended to modify original file first in YDB project and
merge it here afterwards
Review Comment:
Vendored parts should be stored into the `_vendor`, may I know the reason
why we need vendor this code?
##########
airflow/providers/yandex/operators/yandexcloud_yq.py:
##########
@@ -0,0 +1,100 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+
+from typing import TYPE_CHECKING, Sequence, Any
+from airflow.configuration import conf
+
+from airflow.providers.common.sql.operators.sql import SQLExecuteQueryOperator
+from airflow.models import BaseOperator, BaseOperatorLink, XCom
+from airflow.models.taskinstancekey import TaskInstanceKey
+from airflow.providers.yandex.hooks.yandexcloud_yq import YQHook
+
+if TYPE_CHECKING:
+ from airflow.utils.context import Context
+
+XCOM_WEBLINK_KEY = "web_link"
+
+
+class YQLink(BaseOperatorLink):
+ name = "Yandex Query"
+
+ def get_link(self, operator: BaseOperator, *, ti_key: TaskInstanceKey):
+ return XCom.get_value(key=XCOM_WEBLINK_KEY, ti_key=ti_key) or
"https://yq.cloud.yandex.ru";
+
+
+class YQExecuteQueryOperator(SQLExecuteQueryOperator):
+ """
+ Executes sql code using Yandex Query service.
+
+ :param sql: the SQL code to be executed as a single string
+ :param name: name of the query in YandexQuery
+ :param folder_id: cloud folder id where to create query
+ :param connection_id: Airflow connection ID to get parameters from
+ :param folder_id: cloud folder id where to create query
+ """
+
+ operator_extra_links = (YQLink(),)
+ template_fields: Sequence[str] = ("sql",)
+ template_fields_renderers = {"sql": "sql"}
+ template_ext: Sequence[str] = (".sql",)
+ ui_color = "#ededed"
+
+ def __init__(
+ self,
+ *,
+ name: str | None = None,
+ folder_id: str | None = None,
+ connection_id: str | None = None,
+ public_ssh_key: str | None = None,
+ service_account_id: str | None = None,
+ deferrable: bool = conf.getboolean("operators", "default_deferrable",
fallback=False),
Review Comment:
I can't find any deferrable implementations
##########
airflow/providers/yandex/hooks/http_client.py:
##########
@@ -0,0 +1,321 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# This file is a copy of
https://github.com/ydb-platform/ydb/tree/284b7efb67edcdade0b12c849b7fad40739ad62b/ydb/core/fq/libs/http_api_client
+# It is highly recommended to modify original file first in YDB project and
merge it here afterwards
+
+from __future__ import annotations
+
+import logging
+import time
+from datetime import datetime
+import requests
+
+from requests.adapters import HTTPAdapter
+from typing import Any
+from urllib3.util.retry import Retry
+
+from .query_results import YQResults
Review Comment:
We do not use relative imports into the Airflow repo
##########
airflow/providers/yandex/hooks/yandexcloud_yq.py:
##########
@@ -0,0 +1,152 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+from requests.packages.urllib3.util.retry import Retry
+
+from datetime import timedelta
+import logging
+import requests
+import time
+from typing import Any
+import jwt
+
+# These two lines enable debugging at httplib level
(requests->urllib3->http.client)
+# You will see the REQUEST, including HEADERS and DATA, and RESPONSE with
HEADERS but without DATA.
+# The only thing missing will be the response.body which is not logged.
+
+import http.client
+
+http.client.HTTPConnection.debuglevel = 1
+
+# You must initialize logging, otherwise you'll not see debug output.
+logging.basicConfig()
+logging.getLogger().setLevel(logging.DEBUG)
+requests_log = logging.getLogger("requests.packages.urllib3")
+requests_log.setLevel(logging.DEBUG)
+requests_log.propagate = True
Review Comment:
```suggestion
```
Looks like some debug code
##########
airflow/providers/yandex/hooks/http_client.py:
##########
@@ -0,0 +1,321 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# This file is a copy of
https://github.com/ydb-platform/ydb/tree/284b7efb67edcdade0b12c849b7fad40739ad62b/ydb/core/fq/libs/http_api_client
+# It is highly recommended to modify original file first in YDB project and
merge it here afterwards
+
+from __future__ import annotations
+
+import logging
+import time
+from datetime import datetime
+import requests
+
+from requests.adapters import HTTPAdapter
+from typing import Any
+from urllib3.util.retry import Retry
+
+from .query_results import YQResults
+
+MAX_RETRY_FOR_SESSION = 4
+BACK_OFF_FACTOR = 0.3
+TIME_BETWEEN_RETRIES = 1000
+ERROR_CODES = (500, 502, 504)
+
+
+def requests_retry_session(
+ session, retries=MAX_RETRY_FOR_SESSION, back_off_factor=BACK_OFF_FACTOR,
status_force_list=ERROR_CODES
+):
+ retry = Retry(
+ total=retries,
+ read=retries,
+ connect=retries,
+ backoff_factor=back_off_factor,
+ status_forcelist=status_force_list,
+ allowed_methods=frozenset(["GET", "POST"]),
+ )
+ adapter = HTTPAdapter(max_retries=retry)
+ session.mount("http://";, adapter)
+ session.mount("https://";, adapter)
+ return session
+
+
+class YQHttpClientConfig(object):
+ def __init__(
+ self,
+ token: str | None = None,
+ project: str | None = None,
+ user_agent: str | None = "Python YQ HTTP SDK",
+ ) -> None:
+ assert len(token) > 0, "empty token"
+ self.token = token
+ self.project = project
+ self.user_agent = user_agent
+
+ # urls should not contain trailing /
+ self.endpoint: str = "https://api.yandex-query.cloud.yandex.net";
+ self.web_base_url: str = "https://yq.cloud.yandex.ru";
+ self.token_prefix = "Bearer "
+
+
+class YQHttpClientException(Exception):
+ def __init__(self, message: str, status: str = None, msg: str = None,
details: Any = None) -> None:
+ super().__init__(message)
+ self.status = status
+ self.msg = msg
+ self.details = details
+
+
+class YQHttpClient(object):
Review Comment:
```suggestion
class YQHttpClient:
```
##########
airflow/providers/yandex/operators/yandexcloud_yq.py:
##########
@@ -0,0 +1,100 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+
+from typing import TYPE_CHECKING, Sequence, Any
+from airflow.configuration import conf
+
+from airflow.providers.common.sql.operators.sql import SQLExecuteQueryOperator
+from airflow.models import BaseOperator, BaseOperatorLink, XCom
+from airflow.models.taskinstancekey import TaskInstanceKey
+from airflow.providers.yandex.hooks.yandexcloud_yq import YQHook
+
+if TYPE_CHECKING:
+ from airflow.utils.context import Context
+
+XCOM_WEBLINK_KEY = "web_link"
+
+
+class YQLink(BaseOperatorLink):
+ name = "Yandex Query"
+
+ def get_link(self, operator: BaseOperator, *, ti_key: TaskInstanceKey):
+ return XCom.get_value(key=XCOM_WEBLINK_KEY, ti_key=ti_key) or
"https://yq.cloud.yandex.ru";
Review Comment:
Better to move links into the separate sub-package
-
https://github.com/apache/airflow/tree/main/airflow/providers/amazon/aws/links
-
https://github.com/apache/airflow/tree/main/airflow/providers/google/cloud/links
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
