[
https://issues.apache.org/jira/browse/BEAM-8215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16930766#comment-16930766
]
Valentyn Tymofieiev edited comment on BEAM-8215 at 9/16/19 6:41 PM:
--------------------------------------------------------------------
I am running these benchmarks locally, without using PKB, and I am seeing
"invalid_grant" "reauth related error (rapt_required)",
errors from GCS IO. Test execution fails after several instances of
exponential back-off:
{noformat}
root: WARNING: Retry with exponential backoff: waiting for 229.59369729011206
seconds before retrying list_prefix because we caught exception:
oauth2client.client.HttpAccessTokenRefreshError: invalid_grant: reauth related
error (rapt_required)
Traceback for above exception (most recent call last):
File
"/usr/local/google/home/valentyn/projects/beam/beam2/beam/sdks/python/apache_beam/utils/retry.py",
line 206, in wrapper
return fun(*args, **kwargs)
File
"/usr/local/google/home/valentyn/projects/beam/beam2/beam/sdks/python/apache_beam/io/gcp/gcsio.py",
line 463, in list_prefix
response = self.client.objects.List(request)
File
"/usr/local/google/home/valentyn/projects/beam/beam2/beam/sdks/python/apache_beam/io/gcp/internal/clients/storage/storage_v1_client.py",
line 1182, in List
config, request, global_params=global_params)
File
"/usr/local/google/home/valentyn/projects/beam/beam2/beam/build/gradleenv/-1734967053/lib/python3.6/site-packages/apitools/base/py/base_api.py",
line 729, in _RunMethod
http, http_request, **opts)
File
"/usr/local/google/home/valentyn/projects/beam/beam2/beam/build/gradleenv/-1734967053/lib/python3.6/site-packages/apitools/base/py/http_wrapper.py",
line 356, in MakeRequest
max_retry_wait, total_wait_sec))
File
"/usr/local/google/home/valentyn/projects/beam/beam2/beam/sdks/python/apache_beam/io/gcp/gcsio_overrides.py",
line 43, in retry_func
return http_wrapper.HandleExceptionsAndRebuildHttpConnections(retry_args)
File
"/usr/local/google/home/valentyn/projects/beam/beam2/beam/build/gradleenv/-1734967053/lib/python3.6/site-packages/apitools/base/py/http_wrapper.py",
line 304, in HandleExceptionsAndRebuildHttpConnections
raise retry_args.exc
File
"/usr/local/google/home/valentyn/projects/beam/beam2/beam/build/gradleenv/-1734967053/lib/python3.6/site-packages/apitools/base/py/http_wrapper.py",
line 346, in MakeRequest
check_response_func=check_response_func)
File
"/usr/local/google/home/valentyn/projects/beam/beam2/beam/build/gradleenv/-1734967053/lib/python3.6/site-packages/apitools/base/py/http_wrapper.py",
line 396, in _MakeRequestNoRetry
redirections=redirections, connection_type=connection_type)
File
"/usr/local/google/home/valentyn/projects/beam/beam2/beam/build/gradleenv/-1734967053/lib/python3.6/site-packages/oauth2client/transport.py",
line 153, in new_request
credentials._refresh(orig_request_method)
File
"/usr/local/google/home/valentyn/projects/beam/beam2/beam/build/gradleenv/-1734967053/lib/python3.6/site-packages/oauth2client/client.py",
line 765, in _refresh
self._do_refresh_request(http_request)
File
"/usr/local/google/home/valentyn/projects/beam/beam2/beam/build/gradleenv/-1734967053/lib/python3.6/site-packages/oauth2client/client.py",
line 834, in _do_refresh_request
raise HttpAccessTokenRefreshError(error_msg, status=resp.status)
root: INFO: Starting the size estimation of the input
oauth2client.transport: INFO: Attempting refresh to obtain initial access_token
oauth2client.client: INFO: Refreshing access_token
oauth2client.client: INFO: Failed to retrieve access token:
{ "error": "invalid_grant", "error_description": "reauth related error
(rapt_required)", "error_subtype": "rapt_required" }
{noformat}
was (Author: tvalentyn):
I am running these benchmarks locally, without using PKB, and I am seeing
"invalid_grant" "reauth related error (rapt_required)",
errors from GCS IO, which the test does not recover from even with
exponential back-off:
{noformat}
root: WARNING: Retry with exponential backoff: waiting for 229.59369729011206
seconds before retrying list_prefix because we caught exception:
oauth2client.client.HttpAccessTokenRefreshError: invalid_grant: reauth related
error (rapt_required)
Traceback for above exception (most recent call last):
File
"/usr/local/google/home/valentyn/projects/beam/beam2/beam/sdks/python/apache_beam/utils/retry.py",
line 206, in wrapper
return fun(*args, **kwargs)
File
"/usr/local/google/home/valentyn/projects/beam/beam2/beam/sdks/python/apache_beam/io/gcp/gcsio.py",
line 463, in list_prefix
response = self.client.objects.List(request)
File
"/usr/local/google/home/valentyn/projects/beam/beam2/beam/sdks/python/apache_beam/io/gcp/internal/clients/storage/storage_v1_client.py",
line 1182, in List
config, request, global_params=global_params)
File
"/usr/local/google/home/valentyn/projects/beam/beam2/beam/build/gradleenv/-1734967053/lib/python3.6/site-packages/apitools/base/py/base_api.py",
line 729, in _RunMethod
http, http_request, **opts)
File
"/usr/local/google/home/valentyn/projects/beam/beam2/beam/build/gradleenv/-1734967053/lib/python3.6/site-packages/apitools/base/py/http_wrapper.py",
line 356, in MakeRequest
max_retry_wait, total_wait_sec))
File
"/usr/local/google/home/valentyn/projects/beam/beam2/beam/sdks/python/apache_beam/io/gcp/gcsio_overrides.py",
line 43, in retry_func
return http_wrapper.HandleExceptionsAndRebuildHttpConnections(retry_args)
File
"/usr/local/google/home/valentyn/projects/beam/beam2/beam/build/gradleenv/-1734967053/lib/python3.6/site-packages/apitools/base/py/http_wrapper.py",
line 304, in HandleExceptionsAndRebuildHttpConnections
raise retry_args.exc
File
"/usr/local/google/home/valentyn/projects/beam/beam2/beam/build/gradleenv/-1734967053/lib/python3.6/site-packages/apitools/base/py/http_wrapper.py",
line 346, in MakeRequest
check_response_func=check_response_func)
File
"/usr/local/google/home/valentyn/projects/beam/beam2/beam/build/gradleenv/-1734967053/lib/python3.6/site-packages/apitools/base/py/http_wrapper.py",
line 396, in _MakeRequestNoRetry
redirections=redirections, connection_type=connection_type)
File
"/usr/local/google/home/valentyn/projects/beam/beam2/beam/build/gradleenv/-1734967053/lib/python3.6/site-packages/oauth2client/transport.py",
line 153, in new_request
credentials._refresh(orig_request_method)
File
"/usr/local/google/home/valentyn/projects/beam/beam2/beam/build/gradleenv/-1734967053/lib/python3.6/site-packages/oauth2client/client.py",
line 765, in _refresh
self._do_refresh_request(http_request)
File
"/usr/local/google/home/valentyn/projects/beam/beam2/beam/build/gradleenv/-1734967053/lib/python3.6/site-packages/oauth2client/client.py",
line 834, in _do_refresh_request
raise HttpAccessTokenRefreshError(error_msg, status=resp.status)
root: INFO: Starting the size estimation of the input
oauth2client.transport: INFO: Attempting refresh to obtain initial access_token
oauth2client.client: INFO: Refreshing access_token
oauth2client.client: INFO: Failed to retrieve access token:
{ "error": "invalid_grant", "error_description": "reauth related error
(rapt_required)", "error_subtype": "rapt_required" }
{noformat}
> Wordcount 1GB Python PKB benchmarks sometimes fail with uninformative error
> ---------------------------------------------------------------------------
>
> Key: BEAM-8215
> URL: https://issues.apache.org/jira/browse/BEAM-8215
> Project: Beam
> Issue Type: Bug
> Components: testing
> Reporter: Valentyn Tymofieiev
> Assignee: Mark Liu
> Priority: Major
>
> Example:
> https://builds.apache.org/job/beam_PerformanceTests_WordCountIT_Py36/452/console
> {noformat}
> 12:09:27 2019-09-11 19:09:27,655 a47400ce MainThread
> beam_integration_benchmark(1/1) ERROR Error during benchmark
> beam_integration_benchmark
> 12:09:27 Traceback (most recent call last):
> 12:09:27 File
> "/home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_WordCountIT_Py36/PerfKitBenchmarker/perfkitbenchmarker/pkb.py",
> line 841, in RunBenchmark
> 12:09:27 DoRunPhase(spec, collector, detailed_timer)
> 12:09:27 File
> "/home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_WordCountIT_Py36/PerfKitBenchmarker/perfkitbenchmarker/pkb.py",
> line 687, in DoRunPhase
> 12:09:27 samples = spec.BenchmarkRun(spec)
> 12:09:27 File
> "/home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_WordCountIT_Py36/PerfKitBenchmarker/perfkitbenchmarker/linux_benchmarks/beam_integration_benchmark.py",
> line 160, in Run
> 12:09:27 job_type=job_type)
> 12:09:27 File
> "/home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_WordCountIT_Py36/PerfKitBenchmarker/perfkitbenchmarker/providers/gcp/gcp_dpb_dataflow.py",
> line 91, in SubmitJob
> 12:09:27 assert retcode == 0, "Integration Test Failed."
> 12:09:27 AssertionError: Integration Test Failed.
> {noformat}
> It seems like Job submission failed, but there are no details. I talked with
> [~markflyhigh], and sounds like we plan to stop using PKB in favor of another
> framework.
> Assigning to Mark for now to triage follow up or reassign as appropriate.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
