[ https://issues.apache.org/jira/browse/BEAM-8195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17000297#comment-17000297 ]
Alan Myrvold edited comment on BEAM-8195 at 12/19/19 7:12 PM: -------------------------------------------------------------- I also see inconsistent retry behavior between the Python SDK and Java SDK. For 429 errors, Python retries 4 times and Java retries 5 times. For 503 errors, Python retries 19 times and Java retries 5 times. Python's submit_job_description uses the default retry_on_server_errors_filter which does not retry on 4xx errors. The extra Python retries are in apitools/base/py/base_api.py and can be disabled in apache_beam/runners/dataflow/internal/clients/dataflow/dataflow_v1b3_client.py to allow beam to handle the retries. To verify, ran: {{a=0; while true ; do a=$((a+1)); echo -e "HTTP/1.1 503 Message\n" | nc -q 1 -N -l -p 1500 > /dev/null ; echo "req $a $(date)"; done}} or {{a=0; while true ; do a=$((a+1)); echo -e "HTTP/1.1 429 Message\n" | nc -q 1 -N -l -p 1500 > /dev/null ; echo "req $a $(date)"; done}} Locally, then ran dataflow pipeline with --dataflow_endpoint=[http://localhost:1500|http://localhost:1500/] (Python) or --dataflowEndpoint=[http://localhost:1500|http://localhost:1500/] (java) was (Author: alanmyrvold): I also see inconsistent retry behavior between the Python SDK and Java SDK. For 429 errors, Python retries 4 times and Java retries 5 times. For 503 errors, Python retries 19 times and Java retries 5 times. Python's submit_job_description uses the default retry_on_server_errors_filter which does not retry on 4xx errors. Not sure where the extra python retry's are happening, but likely on the http transport level. To verify, ran: {{a=0; while true ; do a=$((a+1)); echo -e "HTTP/1.1 503 Message\n" | nc -q 1 -N -l -p 1500 > /dev/null ; echo "req $a $(date)"; done}} or {{a=0; while true ; do a=$((a+1)); echo -e "HTTP/1.1 429 Message\n" | nc -q 1 -N -l -p 1500 > /dev/null ; echo "req $a $(date)"; done}} Locally, then ran dataflow pipeline with \-\-dataflow_endpoint=http://localhost:1500 (Python) or --dataflowEndpoint=http://localhost:1500 (java) > Quota exceeded for create requests > ---------------------------------- > > Key: BEAM-8195 > URL: https://issues.apache.org/jira/browse/BEAM-8195 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core, testing > Reporter: Ahmet Altay > Assignee: Yifan Zou > Priority: Critical > > Post commits failied with the following error: > HttpError accessing > <https://dataflow.googleapis.com/v1b3/projects/apache-beam-testing/locations/us-central1/jobs?alt=json>: > response: <{'server': 'ESF', '-content-encoding': 'gzip', 'content-type': > 'application/json; charset=UTF-8', 'content-length': '598', > 'transfer-encoding': 'chunked', 'cache-control': 'private', > 'x-xss-protection': '0', 'date': 'Tue, 10 Sep 2019 12:02:24 GMT', 'vary': > 'Origin, X-Origin, Referer', 'x-frame-options': 'SAMEORIGIN', 'status': > '429', 'x-content-type-options': 'nosniff'}>, content <{ > "error": { > "code": 429, > "message": "Quota exceeded for quota metric > 'dataflow.googleapis.com/create_requests' and limit > 'CreateRequestsPerMinutePerUser' of service 'dataflow.googleapis.com' for > consumer 'project_number:844138762903'.", > "status": "RESOURCE_EXHAUSTED", > "details": [ > { > Could we increase the quota? > /cc [~alanmyrvold] [~kenn] -- This message was sent by Atlassian Jira (v8.3.4#803005)