[GitHub] [airflow] ryanyuan commented on pull request #8575: [AIRFLOW-6981] Move Google Cloud Build from Discovery API to Python Library
ryanyuan commented on pull request #8575: URL: https://github.com/apache/airflow/pull/8575#issuecomment-806276124 Hi @turbaszek @mik-laj, any updates on this? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] ryanyuan commented on pull request #8575: [AIRFLOW-6981] Move Google Cloud Build from Discovery API to Python Library
ryanyuan commented on pull request #8575: URL: https://github.com/apache/airflow/pull/8575#issuecomment-731509034 @mik-laj Please let me know when you are going to merge this PR. I will do the rebase again. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] ryanyuan commented on pull request #8575: [AIRFLOW-6981] Move Google Cloud Build from Discovery API to Python Library
ryanyuan commented on pull request #8575: URL: https://github.com/apache/airflow/pull/8575#issuecomment-721019201 @mik-laj no worries. PTAL. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] ryanyuan commented on pull request #8575: [AIRFLOW-6981] Move Google Cloud Build from Discovery API to Python Library
ryanyuan commented on pull request #8575:
URL: https://github.com/apache/airflow/pull/8575#issuecomment-687160392
> I tried running a system test
`test_run_example_gcp_cloud_build_trigger_dag` and I got:
>
> ```
> [2020-09-02 14:36:56,913] {taskinstance.py:1169} INFO - Exporting the
following env vars:
> AIRFLOW_CTX_DAG_OWNER=airflow
> AIRFLOW_CTX_DAG_ID=example_gcp_cloud_build_trigger
> AIRFLOW_CTX_TASK_ID=run_build_trigger
> AIRFLOW_CTX_EXECUTION_DATE=2020-09-01T00:00:00+00:00
> AIRFLOW_CTX_DAG_RUN_ID=backfill__2020-09-01T00:00:00+00:00
> [2020-09-02 14:36:56,926] {cloud_build.py:556} INFO - Start running build
trigger: d4b43269-6ab6-46e3-950e-7541fc213a6d.
> [2020-09-02 14:36:56,927] {taskinstance.py:1310} ERROR - Protocol message
RepoSource has no "repo_source" field.
> Traceback (most recent call last):
> File "/opt/airflow/airflow/models/taskinstance.py", line 1076, in
_run_raw_task
> self._prepare_and_execute_task_with_callbacks(context, task)
> File "/opt/airflow/airflow/models/taskinstance.py", line 1181, in
_prepare_and_execute_task_with_callbacks
> result = self._execute_task(context, task_copy)
> File "/opt/airflow/airflow/models/taskinstance.py", line 1216, in
_execute_task
> result = task_copy.execute(context=context)
> File
"/opt/airflow/airflow/providers/google/cloud/operators/cloud_build.py", line
778, in execute
> metadata=self.metadata,
> File
"/opt/airflow/airflow/providers/google/common/hooks/base_google.py", line 373,
in inner_wrapper
> return func(self, *args, **kwargs)
> File "/opt/airflow/airflow/providers/google/cloud/hooks/cloud_build.py",
line 564, in run_build_trigger
> metadata=metadata,
> File
"/usr/local/lib/python3.7/site-packages/google/cloud/devtools/cloudbuild_v1/gapic/cloud_build_client.py",
line 946, in run_build_trigger
> project_id=project_id, trigger_id=trigger_id, source=source
> ValueError: Protocol message RepoSource has no "repo_source" field.
> ```
Should be okay now.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [airflow] ryanyuan commented on pull request #8575: [AIRFLOW-6981] Move Google Cloud Build from Discovery API to Python Library
ryanyuan commented on pull request #8575: URL: https://github.com/apache/airflow/pull/8575#issuecomment-685725941 > > For the exception you had, it seems you ran the example dag but it failed in the midway and you re-ran it so that it said the trigger already existed. I tried to add a try catch to return the trigger if it already exists but I am unable to find a good way to get the trigger id, which will be used in CloudBuildHook.get_build_trigger(). > > Should we then add note that the operator is not idempotent? Good idea. I've put a message in cloud_build.rst to remind the users the build ID and the trigger ID are not idempotent. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] ryanyuan commented on pull request #8575: [AIRFLOW-6981] Move Google Cloud Build from Discovery API to Python Library
ryanyuan commented on pull request #8575: URL: https://github.com/apache/airflow/pull/8575#issuecomment-679062298 @turbaszek, could you PTAL at this? Cheers. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] ryanyuan commented on pull request #8575: [AIRFLOW-6981] Move Google Cloud Build from Discovery API to Python Library
ryanyuan commented on pull request #8575: URL: https://github.com/apache/airflow/pull/8575#issuecomment-667849407 @mik-laj Any further updates on this? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] ryanyuan commented on pull request #8575: [AIRFLOW-6981] Move Google Cloud Build from Discovery API to Python Library
ryanyuan commented on pull request #8575:
URL: https://github.com/apache/airflow/pull/8575#issuecomment-646460871
> I saw that a new DAG was not added to the system tests. I prepared change
- https://termbin.com/7ezm
> To add to your branch, run the following command:
>
> ```shell
> curl https://termbin.com/7ezm | git am
> ```
>
> When I did bug, I saw that there was a bug. One of the tasks ends with the
following error.
>
> ```
>
logs/previous_runs/2020-06-09_19_00_50/example_gcp_cloud_build_trigger/create_build_trigger/2020-06-08T00:00:00+00:00/1.log
> ```
>
> ```
> [2020-06-09 19:00:33,927] {taskinstance.py:719} INFO - Dependencies all
met for
> [2020-06-09 19:00:33,944] {taskinstance.py:719} INFO - Dependencies all
met for
> [2020-06-09 19:00:33,945] {taskinstance.py:908} INFO -
>
> [2020-06-09 19:00:33,945] {taskinstance.py:909} INFO - Starting attempt 1
of 1
> [2020-06-09 19:00:33,946] {taskinstance.py:910} INFO -
>
> [2020-06-09 19:00:33,953] {taskinstance.py:929} INFO - Executing
on
2020-06-08T00:00:00+00:00
> [2020-06-09 19:00:33,957] {standard_task_runner.py:53} INFO - Started
process 9042 to run task
> [2020-06-09 19:00:34,039] {logging_mixin.py:91} INFO - Running
on host 70cf9669db62
> [2020-06-09 19:00:34,067] {taskinstance.py:1001} INFO - Exporting the
following env vars:
> AIRFLOW_CTX_DAG_OWNER=airflow
> AIRFLOW_CTX_DAG_ID=example_gcp_cloud_build_trigger
> AIRFLOW_CTX_TASK_ID=create_build_trigger
> AIRFLOW_CTX_EXECUTION_DATE=2020-06-08T00:00:00+00:00
> AIRFLOW_CTX_DAG_RUN_ID=backfill__2020-06-08T00:00:00+00:00
> [2020-06-09 19:00:35,144] {taskinstance.py:1194} ERROR - 409 trigger
(198907790164, test-cloud-build-trigger) already exists
> Traceback (most recent call last):
> File
"/usr/local/lib/python3.8/site-packages/google/api_core/grpc_helpers.py", line
57, in error_remapped_callable
> return callable_(*args, **kwargs)
> File "/usr/local/lib/python3.8/site-packages/grpc/_channel.py", line
826, in __call__
> return _end_unary_response_blocking(state, call, False, None)
> File "/usr/local/lib/python3.8/site-packages/grpc/_channel.py", line
729, in _end_unary_response_blocking
> raise _InactiveRpcError(state)
> grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated
with:
>status = StatusCode.ALREADY_EXISTS
>details = "trigger (198907790164, test-cloud-build-trigger) already
exists"
>debug_error_string =
"{"created":"@1591729235.142798800","description":"Error received from peer
ipv4:216.58.215.74:443","file":"src/core/lib/surface/call.cc","file_line":1056,"grpc_message":"trigger
(198907790164, test-cloud-build-trigger) already exists","grpc_status":6}"
> >
>
> The above exception was the direct cause of the following exception:
>
> Traceback (most recent call last):
> File "/opt/airflow/airflow/models/taskinstance.py", line 1026, in
_run_raw_task
> result = task_copy.execute(context=context)
> File
"/opt/airflow/airflow/providers/google/cloud/operators/cloud_build.py", line
239, in execute
> result = hook.create_build_trigger(
> File
"/opt/airflow/airflow/providers/google/common/hooks/base_google.py", line 356,
in inner_wrapper
> return func(self, *args, **kwargs)
> File "/opt/airflow/airflow/providers/google/cloud/hooks/cloud_build.py",
line 188, in create_build_trigger
> return client.create_build_trigger(
> File
"/usr/local/lib/python3.8/site-packages/google/cloud/devtools/cloudbuild_v1/gapic/cloud_build_client.py",
line 595, in create_build_trigger
> return self._inner_api_calls["create_build_trigger"](
> File
"/usr/local/lib/python3.8/site-packages/google/api_core/gapic_v1/method.py",
line 143, in __call__
> return wrapped_func(*args, **kwargs)
> File
"/usr/local/lib/python3.8/site-packages/google/api_core/grpc_helpers.py", line
59, in error_remapped_callable
> six.raise_from(exceptions.from_grpc_error(exc), exc)
> File "", line 3, in raise_from
> google.api_core.exceptions.AlreadyExists: 409 trigger (198907790164,
test-cloud-build-trigger) already exists
> [2020-06-09 19:00:35,158] {taskinstance.py:1231} INFO - Marking task as
FAILED. dag_id=example_gcp_cloud_build_trigger, task_id=create_build_trigger,
execution_date=20200608T00, start_date=20200609T190033,
end_date=20200609T190035
> [2020-06-09 19:00:35,917] {logging_mixin.py:91} INFO - [2020-06-09
19:00:35,917] {local_task_job.py:103} INFO - Task exited with return code 1
> ```
>
> Do you have an idea how to solve it?
@mik-laj Do you still have this issue when you run the test?
[GitHub] [airflow] ryanyuan commented on pull request #8575: [AIRFLOW-6981] Move Google Cloud Build from Discovery API to Python Library
ryanyuan commented on pull request #8575: URL: https://github.com/apache/airflow/pull/8575#issuecomment-641730079 Thanks @mik-laj, I've applied your patch. For the exception you had, it seems you ran the example dag but it failed in the midway and you re-ran it so that it said the trigger already existed. I tried to add a try catch to return the trigger if it already exists but I am unable to find a good way to get the trigger id, which will be used in CloudBuildHook.get_build_trigger(). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] ryanyuan commented on pull request #8575: [AIRFLOW-6981] Move Google Cloud Build from Discovery API to Python Library
ryanyuan commented on pull request #8575: URL: https://github.com/apache/airflow/pull/8575#issuecomment-639213173 Thanks @mik-laj All good now. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] ryanyuan commented on pull request #8575: [AIRFLOW-6981] Move Google Cloud Build from Discovery API to Python Library
ryanyuan commented on pull request #8575: URL: https://github.com/apache/airflow/pull/8575#issuecomment-638772774 @mik-laj I'll rebase and run the tests again. It seems a lot of hive-related test failed last time. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] ryanyuan commented on pull request #8575: [AIRFLOW-6981] Move Google Cloud Build from Discovery API to Python Library
ryanyuan commented on pull request #8575: URL: https://github.com/apache/airflow/pull/8575#issuecomment-620944984 thanks @potiuk This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] ryanyuan commented on pull request #8575: [AIRFLOW-6981] Move Google Cloud Build from Discovery API to Python Library
ryanyuan commented on pull request #8575: URL: https://github.com/apache/airflow/pull/8575#issuecomment-620354199 Cheers @potiuk. This time it failed at Quarantined:Pg9.6,Py3.6. But the error log didn't say much. Any suggestions? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] ryanyuan commented on pull request #8575: [AIRFLOW-6981] Move Google Cloud Build from Discovery API to Python Library
ryanyuan commented on pull request #8575: URL: https://github.com/apache/airflow/pull/8575#issuecomment-619945144 @mik-laj Any ideas why the builds failed on Requirements 3.7 and 3.6? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
