[
https://issues.apache.org/jira/browse/BEAM-5514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16644101#comment-16644101
]
Chamikara Jayalath edited comment on BEAM-5514 at 10/9/18 9:35 PM:
-------------------------------------------------------------------
Thanks.
I believe HTTP 403 issues in general are considered non-retriable. So it makes
sense for Dataflow to not to retry requests at the client. In-fact BigQuery
support page provides following instructions regarding HTTP 403 quotaExceeded
errors.
https://cloud.google.com/bigquery/troubleshooting-errors
"View the {{message}} property of the error object for more information about
which quota was exceeded. To reset or raise a BigQuery quota, [contact
support|https://cloud.google.com/support]. To modify a custom quota, submit a
request from the [Google Cloud Platform
Console|https://console.cloud.google.com/iam-admin/quotas] page.";
So basically this is asking to fix the issue (request a quota increase) before
retrying.
The issues is, due to the architecture if Dataflow streaming jobs, even though
we do not retry at the client, we do in fact retry all work items indefinitely.
So we end up sending a large number of requests to BigQuery whenever a user hit
quota errors.
was (Author: chamikara):
Kevin,
Sounds like
> BigQueryIO doesn't handle quotaExceeded errors properly
> -------------------------------------------------------
>
> Key: BEAM-5514
> URL: https://issues.apache.org/jira/browse/BEAM-5514
> Project: Beam
> Issue Type: Bug
> Components: io-java-gcp
> Reporter: Kevin Peterson
> Assignee: Reuven Lax
> Priority: Major
>
> When exceeding a streaming quota for BigQuery insertAll requests, BigQuery
> returns a 403 with reason "quotaExceeded".
> The current implementation of BigQueryIO does not consider this to be a rate
> limited exception, and therefore does not perform exponential backoff
> properly, leading to repeated calls to BQ.
> The actual error is in the
> [ApiErrorExtractor|https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryServicesImpl.java#L739]
> class, which is called from
> [BigQueryServicesImpl|https://github.com/GoogleCloudPlatform/bigdata-interop/blob/master/util/src/main/java/com/google/cloud/hadoop/util/ApiErrorExtractor.java#L263]
> to determine how to retry the failure.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
