[jira] [Work logged] (BEAM-7933) Adding timeout to JobServer grpc calls
[ https://issues.apache.org/jira/browse/BEAM-7933?focusedWorklogId=322702=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-322702 ] ASF GitHub Bot logged work on BEAM-7933: Author: ASF GitHub Bot Created on: 03/Oct/19 15:58 Start Date: 03/Oct/19 15:58 Worklog Time Spent: 10m Work Description: ibzib commented on pull request #9673: [BEAM-7933] Add job server request timeout (default to 60 seconds) URL: https://github.com/apache/beam/pull/9673 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 322702) Time Spent: 3h 50m (was: 3h 40m) > Adding timeout to JobServer grpc calls > -- > > Key: BEAM-7933 > URL: https://issues.apache.org/jira/browse/BEAM-7933 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Affects Versions: 2.14.0 >Reporter: Enrico Canzonieri >Assignee: Enrico Canzonieri >Priority: Minor > Labels: portability > Time Spent: 3h 50m > Remaining Estimate: 0h > > grpc calls to the JobServer from the Python SDK do not have timeouts. That > means that the call to pipeline.run()could hang forever if the JobServer is > not running (or failing to start). > E.g. > [https://github.com/apache/beam/blob/master/sdks/python/apache_beam/runners/portability/portable_runner.py#L307] > the call to Prepare() doesn't provide any timeout value and the same applies > to other JobServer requests. > As part of this ticket we could add a default timeout of 60 seconds as the > default timeout for http client. > Additionally, we could consider adding a --job-server-request-timeout to the > [PortableOptions|https://github.com/apache/beam/blob/master/sdks/python/apache_beam/options/pipeline_options.py#L805] > class to be used in the JobServer interactions inside probable_runner.py. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-7933) Adding timeout to JobServer grpc calls
[ https://issues.apache.org/jira/browse/BEAM-7933?focusedWorklogId=322640=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-322640 ] ASF GitHub Bot logged work on BEAM-7933: Author: ASF GitHub Bot Created on: 03/Oct/19 14:27 Start Date: 03/Oct/19 14:27 Worklog Time Spent: 10m Work Description: ecanzonieri commented on issue #9673: [BEAM-7933] Add job server request timeout (default to 60 seconds) URL: https://github.com/apache/beam/pull/9673#issuecomment-537969691 Squashed everything to a single commit now. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 322640) Time Spent: 3h 40m (was: 3.5h) > Adding timeout to JobServer grpc calls > -- > > Key: BEAM-7933 > URL: https://issues.apache.org/jira/browse/BEAM-7933 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Affects Versions: 2.14.0 >Reporter: Enrico Canzonieri >Assignee: Enrico Canzonieri >Priority: Minor > Labels: portability > Time Spent: 3h 40m > Remaining Estimate: 0h > > grpc calls to the JobServer from the Python SDK do not have timeouts. That > means that the call to pipeline.run()could hang forever if the JobServer is > not running (or failing to start). > E.g. > [https://github.com/apache/beam/blob/master/sdks/python/apache_beam/runners/portability/portable_runner.py#L307] > the call to Prepare() doesn't provide any timeout value and the same applies > to other JobServer requests. > As part of this ticket we could add a default timeout of 60 seconds as the > default timeout for http client. > Additionally, we could consider adding a --job-server-request-timeout to the > [PortableOptions|https://github.com/apache/beam/blob/master/sdks/python/apache_beam/options/pipeline_options.py#L805] > class to be used in the JobServer interactions inside probable_runner.py. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-7933) Adding timeout to JobServer grpc calls
[ https://issues.apache.org/jira/browse/BEAM-7933?focusedWorklogId=322637=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-322637 ] ASF GitHub Bot logged work on BEAM-7933: Author: ASF GitHub Bot Created on: 03/Oct/19 14:24 Start Date: 03/Oct/19 14:24 Worklog Time Spent: 10m Work Description: ecanzonieri commented on issue #9673: [BEAM-7933] Add job server request timeout (default to 60 seconds) URL: https://github.com/apache/beam/pull/9673#issuecomment-537968350 I messed up the rebase. I'll fix it and repush. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 322637) Time Spent: 3.5h (was: 3h 20m) > Adding timeout to JobServer grpc calls > -- > > Key: BEAM-7933 > URL: https://issues.apache.org/jira/browse/BEAM-7933 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Affects Versions: 2.14.0 >Reporter: Enrico Canzonieri >Assignee: Enrico Canzonieri >Priority: Minor > Labels: portability > Time Spent: 3.5h > Remaining Estimate: 0h > > grpc calls to the JobServer from the Python SDK do not have timeouts. That > means that the call to pipeline.run()could hang forever if the JobServer is > not running (or failing to start). > E.g. > [https://github.com/apache/beam/blob/master/sdks/python/apache_beam/runners/portability/portable_runner.py#L307] > the call to Prepare() doesn't provide any timeout value and the same applies > to other JobServer requests. > As part of this ticket we could add a default timeout of 60 seconds as the > default timeout for http client. > Additionally, we could consider adding a --job-server-request-timeout to the > [PortableOptions|https://github.com/apache/beam/blob/master/sdks/python/apache_beam/options/pipeline_options.py#L805] > class to be used in the JobServer interactions inside probable_runner.py. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-7933) Adding timeout to JobServer grpc calls
[
https://issues.apache.org/jira/browse/BEAM-7933?focusedWorklogId=322201=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-322201
]
ASF GitHub Bot logged work on BEAM-7933:
Author: ASF GitHub Bot
Created on: 02/Oct/19 20:04
Start Date: 02/Oct/19 20:04
Worklog Time Spent: 10m
Work Description: ibzib commented on pull request #9673: [BEAM-7933] Add
job server request timeout (default to 60 seconds)
URL: https://github.com/apache/beam/pull/9673#discussion_r330744414
##
File path: sdks/python/apache_beam/options/pipeline_options.py
##
@@ -818,11 +818,15 @@ class PortableOptions(PipelineOptions):
"""
@classmethod
def _add_argparse_args(cls, parser):
-parser.add_argument('--job_endpoint',
-default=None,
-help=
-('Job service endpoint to use. Should be in the form '
- 'of address and port, e.g. localhost:3000'))
+parser.add_argument(
+'--job_endpoint', default=None,
+help=('Job service endpoint to use. Should be in the form of address '
+ 'and port, e.g. localhost:3000'))
+parser.add_argument(
+'--job-server-timeout', default=60, type=int,
+help=('Job service request timeout in seconds. The timeout '
+ 'determines the max time the driver program will wait to '
+ 'get a response from the job server.'))
Review comment:
Might want to make a note here that timeouts do not apply to the actual
pipeline itself.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 322201)
Time Spent: 3h 20m (was: 3h 10m)
> Adding timeout to JobServer grpc calls
> --
>
> Key: BEAM-7933
> URL: https://issues.apache.org/jira/browse/BEAM-7933
> Project: Beam
> Issue Type: Improvement
> Components: sdk-py-core
>Affects Versions: 2.14.0
>Reporter: Enrico Canzonieri
>Assignee: Enrico Canzonieri
>Priority: Minor
> Labels: portability
> Time Spent: 3h 20m
> Remaining Estimate: 0h
>
> grpc calls to the JobServer from the Python SDK do not have timeouts. That
> means that the call to pipeline.run()could hang forever if the JobServer is
> not running (or failing to start).
> E.g.
> [https://github.com/apache/beam/blob/master/sdks/python/apache_beam/runners/portability/portable_runner.py#L307]
> the call to Prepare() doesn't provide any timeout value and the same applies
> to other JobServer requests.
> As part of this ticket we could add a default timeout of 60 seconds as the
> default timeout for http client.
> Additionally, we could consider adding a --job-server-request-timeout to the
> [PortableOptions|https://github.com/apache/beam/blob/master/sdks/python/apache_beam/options/pipeline_options.py#L805]
> class to be used in the JobServer interactions inside probable_runner.py.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-7933) Adding timeout to JobServer grpc calls
[ https://issues.apache.org/jira/browse/BEAM-7933?focusedWorklogId=322105=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-322105 ] ASF GitHub Bot logged work on BEAM-7933: Author: ASF GitHub Bot Created on: 02/Oct/19 18:04 Start Date: 02/Oct/19 18:04 Worklog Time Spent: 10m Work Description: ibzib commented on issue #9673: [BEAM-7933] Add job server request timeout (default to 60 seconds) URL: https://github.com/apache/beam/pull/9673#issuecomment-537612066 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 322105) Time Spent: 3h 10m (was: 3h) > Adding timeout to JobServer grpc calls > -- > > Key: BEAM-7933 > URL: https://issues.apache.org/jira/browse/BEAM-7933 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Affects Versions: 2.14.0 >Reporter: Enrico Canzonieri >Assignee: Enrico Canzonieri >Priority: Minor > Labels: portability > Time Spent: 3h 10m > Remaining Estimate: 0h > > grpc calls to the JobServer from the Python SDK do not have timeouts. That > means that the call to pipeline.run()could hang forever if the JobServer is > not running (or failing to start). > E.g. > [https://github.com/apache/beam/blob/master/sdks/python/apache_beam/runners/portability/portable_runner.py#L307] > the call to Prepare() doesn't provide any timeout value and the same applies > to other JobServer requests. > As part of this ticket we could add a default timeout of 60 seconds as the > default timeout for http client. > Additionally, we could consider adding a --job-server-request-timeout to the > [PortableOptions|https://github.com/apache/beam/blob/master/sdks/python/apache_beam/options/pipeline_options.py#L805] > class to be used in the JobServer interactions inside probable_runner.py. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-7933) Adding timeout to JobServer grpc calls
[ https://issues.apache.org/jira/browse/BEAM-7933?focusedWorklogId=322078=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-322078 ] ASF GitHub Bot logged work on BEAM-7933: Author: ASF GitHub Bot Created on: 02/Oct/19 17:29 Start Date: 02/Oct/19 17:29 Worklog Time Spent: 10m Work Description: ibzib commented on issue #9673: [BEAM-7933] Add job server request timeout (default to 60 seconds) URL: https://github.com/apache/beam/pull/9673#issuecomment-537597980 Docs failure is probably https://issues.apache.org/jira/browse/BEAM-8286 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 322078) Time Spent: 3h (was: 2h 50m) > Adding timeout to JobServer grpc calls > -- > > Key: BEAM-7933 > URL: https://issues.apache.org/jira/browse/BEAM-7933 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Affects Versions: 2.14.0 >Reporter: Enrico Canzonieri >Assignee: Enrico Canzonieri >Priority: Minor > Labels: portability > Time Spent: 3h > Remaining Estimate: 0h > > grpc calls to the JobServer from the Python SDK do not have timeouts. That > means that the call to pipeline.run()could hang forever if the JobServer is > not running (or failing to start). > E.g. > [https://github.com/apache/beam/blob/master/sdks/python/apache_beam/runners/portability/portable_runner.py#L307] > the call to Prepare() doesn't provide any timeout value and the same applies > to other JobServer requests. > As part of this ticket we could add a default timeout of 60 seconds as the > default timeout for http client. > Additionally, we could consider adding a --job-server-request-timeout to the > [PortableOptions|https://github.com/apache/beam/blob/master/sdks/python/apache_beam/options/pipeline_options.py#L805] > class to be used in the JobServer interactions inside probable_runner.py. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-7933) Adding timeout to JobServer grpc calls
[ https://issues.apache.org/jira/browse/BEAM-7933?focusedWorklogId=321940=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-321940 ] ASF GitHub Bot logged work on BEAM-7933: Author: ASF GitHub Bot Created on: 02/Oct/19 14:36 Start Date: 02/Oct/19 14:36 Worklog Time Spent: 10m Work Description: ecanzonieri commented on issue #9673: [BEAM-7933] Add job server request timeout (default to 60 seconds) URL: https://github.com/apache/beam/pull/9673#issuecomment-537502726 Apparently `:sdks:python:test-suites:tox:py2:docs` failed. I'll run locally to check whether related to my pr. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 321940) Time Spent: 2h 50m (was: 2h 40m) > Adding timeout to JobServer grpc calls > -- > > Key: BEAM-7933 > URL: https://issues.apache.org/jira/browse/BEAM-7933 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Affects Versions: 2.14.0 >Reporter: Enrico Canzonieri >Assignee: Enrico Canzonieri >Priority: Minor > Labels: portability > Time Spent: 2h 50m > Remaining Estimate: 0h > > grpc calls to the JobServer from the Python SDK do not have timeouts. That > means that the call to pipeline.run()could hang forever if the JobServer is > not running (or failing to start). > E.g. > [https://github.com/apache/beam/blob/master/sdks/python/apache_beam/runners/portability/portable_runner.py#L307] > the call to Prepare() doesn't provide any timeout value and the same applies > to other JobServer requests. > As part of this ticket we could add a default timeout of 60 seconds as the > default timeout for http client. > Additionally, we could consider adding a --job-server-request-timeout to the > [PortableOptions|https://github.com/apache/beam/blob/master/sdks/python/apache_beam/options/pipeline_options.py#L805] > class to be used in the JobServer interactions inside probable_runner.py. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-7933) Adding timeout to JobServer grpc calls
[ https://issues.apache.org/jira/browse/BEAM-7933?focusedWorklogId=321884=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-321884 ] ASF GitHub Bot logged work on BEAM-7933: Author: ASF GitHub Bot Created on: 02/Oct/19 13:53 Start Date: 02/Oct/19 13:53 Worklog Time Spent: 10m Work Description: ecanzonieri commented on issue #9673: [BEAM-7933] Add job server request timeout (default to 60 seconds) URL: https://github.com/apache/beam/pull/9673#issuecomment-537502726 Apparently `:sdks:python:test-suites:tox:py2:docs` failed. I'm locally to check whether related to my pr. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 321884) Time Spent: 2h 40m (was: 2.5h) > Adding timeout to JobServer grpc calls > -- > > Key: BEAM-7933 > URL: https://issues.apache.org/jira/browse/BEAM-7933 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Affects Versions: 2.14.0 >Reporter: Enrico Canzonieri >Assignee: Enrico Canzonieri >Priority: Minor > Labels: portability > Time Spent: 2h 40m > Remaining Estimate: 0h > > grpc calls to the JobServer from the Python SDK do not have timeouts. That > means that the call to pipeline.run()could hang forever if the JobServer is > not running (or failing to start). > E.g. > [https://github.com/apache/beam/blob/master/sdks/python/apache_beam/runners/portability/portable_runner.py#L307] > the call to Prepare() doesn't provide any timeout value and the same applies > to other JobServer requests. > As part of this ticket we could add a default timeout of 60 seconds as the > default timeout for http client. > Additionally, we could consider adding a --job-server-request-timeout to the > [PortableOptions|https://github.com/apache/beam/blob/master/sdks/python/apache_beam/options/pipeline_options.py#L805] > class to be used in the JobServer interactions inside probable_runner.py. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-7933) Adding timeout to JobServer grpc calls
[ https://issues.apache.org/jira/browse/BEAM-7933?focusedWorklogId=321789=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-321789 ] ASF GitHub Bot logged work on BEAM-7933: Author: ASF GitHub Bot Created on: 02/Oct/19 09:55 Start Date: 02/Oct/19 09:55 Worklog Time Spent: 10m Work Description: ecanzonieri commented on issue #9673: [BEAM-7933] Add job server request timeout (default to 60 seconds) URL: https://github.com/apache/beam/pull/9673#issuecomment-537422905 Yeah, sorry about the lint errors. I still need to find a good configuration for PyCharm that actually works for Beam. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 321789) Time Spent: 2.5h (was: 2h 20m) > Adding timeout to JobServer grpc calls > -- > > Key: BEAM-7933 > URL: https://issues.apache.org/jira/browse/BEAM-7933 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Affects Versions: 2.14.0 >Reporter: Enrico Canzonieri >Assignee: Enrico Canzonieri >Priority: Minor > Labels: portability > Time Spent: 2.5h > Remaining Estimate: 0h > > grpc calls to the JobServer from the Python SDK do not have timeouts. That > means that the call to pipeline.run()could hang forever if the JobServer is > not running (or failing to start). > E.g. > [https://github.com/apache/beam/blob/master/sdks/python/apache_beam/runners/portability/portable_runner.py#L307] > the call to Prepare() doesn't provide any timeout value and the same applies > to other JobServer requests. > As part of this ticket we could add a default timeout of 60 seconds as the > default timeout for http client. > Additionally, we could consider adding a --job-server-request-timeout to the > [PortableOptions|https://github.com/apache/beam/blob/master/sdks/python/apache_beam/options/pipeline_options.py#L805] > class to be used in the JobServer interactions inside probable_runner.py. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-7933) Adding timeout to JobServer grpc calls
[ https://issues.apache.org/jira/browse/BEAM-7933?focusedWorklogId=321603=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-321603 ] ASF GitHub Bot logged work on BEAM-7933: Author: ASF GitHub Bot Created on: 01/Oct/19 22:33 Start Date: 01/Oct/19 22:33 Worklog Time Spent: 10m Work Description: ibzib commented on issue #9673: [BEAM-7933] Add job server request timeout (default to 60 seconds) URL: https://github.com/apache/beam/pull/9673#issuecomment-537260245 Please fix lint errors. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 321603) Time Spent: 2h 20m (was: 2h 10m) > Adding timeout to JobServer grpc calls > -- > > Key: BEAM-7933 > URL: https://issues.apache.org/jira/browse/BEAM-7933 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Affects Versions: 2.14.0 >Reporter: Enrico Canzonieri >Assignee: Enrico Canzonieri >Priority: Minor > Labels: portability > Time Spent: 2h 20m > Remaining Estimate: 0h > > grpc calls to the JobServer from the Python SDK do not have timeouts. That > means that the call to pipeline.run()could hang forever if the JobServer is > not running (or failing to start). > E.g. > [https://github.com/apache/beam/blob/master/sdks/python/apache_beam/runners/portability/portable_runner.py#L307] > the call to Prepare() doesn't provide any timeout value and the same applies > to other JobServer requests. > As part of this ticket we could add a default timeout of 60 seconds as the > default timeout for http client. > Additionally, we could consider adding a --job-server-request-timeout to the > [PortableOptions|https://github.com/apache/beam/blob/master/sdks/python/apache_beam/options/pipeline_options.py#L805] > class to be used in the JobServer interactions inside probable_runner.py. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-7933) Adding timeout to JobServer grpc calls
[ https://issues.apache.org/jira/browse/BEAM-7933?focusedWorklogId=321602=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-321602 ] ASF GitHub Bot logged work on BEAM-7933: Author: ASF GitHub Bot Created on: 01/Oct/19 22:32 Start Date: 01/Oct/19 22:32 Worklog Time Spent: 10m Work Description: ibzib commented on pull request #9673: [BEAM-7933] Add job server request timeout (default to 60 seconds) URL: https://github.com/apache/beam/pull/9673#discussion_r330305499 ## File path: sdks/python/apache_beam/runners/portability/portable_runner.py ## @@ -251,7 +253,11 @@ def send_options_request(max_retries=5): # This reports channel is READY but connections may fail # Seems to be only an issue on Mac with port forwardings return job_service.DescribePipelineOptions( - beam_job_api_pb2.DescribePipelineOptionsRequest()) + beam_job_api_pb2.DescribePipelineOptionsRequest(), + timeout=portable_options.job_server_timeout) Review comment: Oh, I read everything wrong. :sweat_smile: It'd be great to have real tests to verify all these timeouts, but I guess they would not make suitable unit tests, as they would all require at least 1 second to complete. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 321602) Time Spent: 2h 10m (was: 2h) > Adding timeout to JobServer grpc calls > -- > > Key: BEAM-7933 > URL: https://issues.apache.org/jira/browse/BEAM-7933 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Affects Versions: 2.14.0 >Reporter: Enrico Canzonieri >Assignee: Enrico Canzonieri >Priority: Minor > Labels: portability > Time Spent: 2h 10m > Remaining Estimate: 0h > > grpc calls to the JobServer from the Python SDK do not have timeouts. That > means that the call to pipeline.run()could hang forever if the JobServer is > not running (or failing to start). > E.g. > [https://github.com/apache/beam/blob/master/sdks/python/apache_beam/runners/portability/portable_runner.py#L307] > the call to Prepare() doesn't provide any timeout value and the same applies > to other JobServer requests. > As part of this ticket we could add a default timeout of 60 seconds as the > default timeout for http client. > Additionally, we could consider adding a --job-server-request-timeout to the > [PortableOptions|https://github.com/apache/beam/blob/master/sdks/python/apache_beam/options/pipeline_options.py#L805] > class to be used in the JobServer interactions inside probable_runner.py. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-7933) Adding timeout to JobServer grpc calls
[ https://issues.apache.org/jira/browse/BEAM-7933?focusedWorklogId=321589=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-321589 ] ASF GitHub Bot logged work on BEAM-7933: Author: ASF GitHub Bot Created on: 01/Oct/19 22:21 Start Date: 01/Oct/19 22:21 Worklog Time Spent: 10m Work Description: ecanzonieri commented on pull request #9673: [BEAM-7933] Add job server request timeout (default to 60 seconds) URL: https://github.com/apache/beam/pull/9673#discussion_r330302508 ## File path: sdks/python/apache_beam/runners/portability/portable_runner.py ## @@ -251,7 +253,11 @@ def send_options_request(max_retries=5): # This reports channel is READY but connections may fail # Seems to be only an issue on Mac with port forwardings return job_service.DescribePipelineOptions( - beam_job_api_pb2.DescribePipelineOptionsRequest()) + beam_job_api_pb2.DescribePipelineOptionsRequest(), + timeout=portable_options.job_server_timeout) Review comment: hmmm, I may be missing something but the timeout is never added to the request. My understanding is that timeout itself is not a request parameter but rather a grpc method call parameter. @ibzib can you point me to the code where you see me adding the timeout to the request? Perhaps, I accidentally added it to the wrong place. I'm no grpc expert, but I've ran some manual tests on this feature: 1) run python driver without job server running and default timeout -> The driver times out as expected after 60 seconds. 2) run python driver without job server running and change timeout -> The driver times out as expected after the specified timeout. 3) run python driver with job server running but code changed to slow down Prepare() -> The driver times out as expected. 4) run python driver with long running streaming job (Run() method) -> the driver program never times out (note this is the expected behavior since the driver is stuck waiting for completion. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 321589) Time Spent: 2h (was: 1h 50m) > Adding timeout to JobServer grpc calls > -- > > Key: BEAM-7933 > URL: https://issues.apache.org/jira/browse/BEAM-7933 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Affects Versions: 2.14.0 >Reporter: Enrico Canzonieri >Assignee: Enrico Canzonieri >Priority: Minor > Labels: portability > Time Spent: 2h > Remaining Estimate: 0h > > grpc calls to the JobServer from the Python SDK do not have timeouts. That > means that the call to pipeline.run()could hang forever if the JobServer is > not running (or failing to start). > E.g. > [https://github.com/apache/beam/blob/master/sdks/python/apache_beam/runners/portability/portable_runner.py#L307] > the call to Prepare() doesn't provide any timeout value and the same applies > to other JobServer requests. > As part of this ticket we could add a default timeout of 60 seconds as the > default timeout for http client. > Additionally, we could consider adding a --job-server-request-timeout to the > [PortableOptions|https://github.com/apache/beam/blob/master/sdks/python/apache_beam/options/pipeline_options.py#L805] > class to be used in the JobServer interactions inside probable_runner.py. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-7933) Adding timeout to JobServer grpc calls
[ https://issues.apache.org/jira/browse/BEAM-7933?focusedWorklogId=321436=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-321436 ] ASF GitHub Bot logged work on BEAM-7933: Author: ASF GitHub Bot Created on: 01/Oct/19 18:37 Start Date: 01/Oct/19 18:37 Worklog Time Spent: 10m Work Description: ibzib commented on pull request #9673: [BEAM-7933] Add job server request timeout (default to 60 seconds) URL: https://github.com/apache/beam/pull/9673#discussion_r330209685 ## File path: sdks/python/apache_beam/runners/portability/portable_runner.py ## @@ -251,7 +253,11 @@ def send_options_request(max_retries=5): # This reports channel is READY but connections may fail # Seems to be only an issue on Mac with port forwardings return job_service.DescribePipelineOptions( - beam_job_api_pb2.DescribePipelineOptionsRequest()) + beam_job_api_pb2.DescribePipelineOptionsRequest(), + timeout=portable_options.job_server_timeout) Review comment: Adding the timeout to the stubs fixes the tests, but are you sure adding the timeout like that is actually effective? In all the other places you changed, you put timeout in the requests, but here you put it in the service method, which I assumed was the problem. ie `DescribePipelineOptions(DescribePipelineOptionsRequest(), timeout=...)` should be `DescribePipelineOptions(DescribePipelineOptionsRequest(timeout=...))` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 321436) Time Spent: 1h 50m (was: 1h 40m) > Adding timeout to JobServer grpc calls > -- > > Key: BEAM-7933 > URL: https://issues.apache.org/jira/browse/BEAM-7933 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Affects Versions: 2.14.0 >Reporter: Enrico Canzonieri >Assignee: Enrico Canzonieri >Priority: Minor > Labels: portability > Time Spent: 1h 50m > Remaining Estimate: 0h > > grpc calls to the JobServer from the Python SDK do not have timeouts. That > means that the call to pipeline.run()could hang forever if the JobServer is > not running (or failing to start). > E.g. > [https://github.com/apache/beam/blob/master/sdks/python/apache_beam/runners/portability/portable_runner.py#L307] > the call to Prepare() doesn't provide any timeout value and the same applies > to other JobServer requests. > As part of this ticket we could add a default timeout of 60 seconds as the > default timeout for http client. > Additionally, we could consider adding a --job-server-request-timeout to the > [PortableOptions|https://github.com/apache/beam/blob/master/sdks/python/apache_beam/options/pipeline_options.py#L805] > class to be used in the JobServer interactions inside probable_runner.py. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-7933) Adding timeout to JobServer grpc calls
[ https://issues.apache.org/jira/browse/BEAM-7933?focusedWorklogId=321435=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-321435 ] ASF GitHub Bot logged work on BEAM-7933: Author: ASF GitHub Bot Created on: 01/Oct/19 18:37 Start Date: 01/Oct/19 18:37 Worklog Time Spent: 10m Work Description: ibzib commented on pull request #9673: [BEAM-7933] Add job server request timeout (default to 60 seconds) URL: https://github.com/apache/beam/pull/9673#discussion_r330206095 ## File path: sdks/python/apache_beam/runners/portability/portable_runner.py ## @@ -251,7 +253,11 @@ def send_options_request(max_retries=5): # This reports channel is READY but connections may fail # Seems to be only an issue on Mac with port forwardings return job_service.DescribePipelineOptions( - beam_job_api_pb2.DescribePipelineOptionsRequest()) + beam_job_api_pb2.DescribePipelineOptionsRequest(), + timeout=portable_options.job_server_timeout) +except grpc.FutureTimeoutError: + # no retry for timeout errors Review comment: Okay. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 321435) Time Spent: 1h 50m (was: 1h 40m) > Adding timeout to JobServer grpc calls > -- > > Key: BEAM-7933 > URL: https://issues.apache.org/jira/browse/BEAM-7933 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Affects Versions: 2.14.0 >Reporter: Enrico Canzonieri >Assignee: Enrico Canzonieri >Priority: Minor > Labels: portability > Time Spent: 1h 50m > Remaining Estimate: 0h > > grpc calls to the JobServer from the Python SDK do not have timeouts. That > means that the call to pipeline.run()could hang forever if the JobServer is > not running (or failing to start). > E.g. > [https://github.com/apache/beam/blob/master/sdks/python/apache_beam/runners/portability/portable_runner.py#L307] > the call to Prepare() doesn't provide any timeout value and the same applies > to other JobServer requests. > As part of this ticket we could add a default timeout of 60 seconds as the > default timeout for http client. > Additionally, we could consider adding a --job-server-request-timeout to the > [PortableOptions|https://github.com/apache/beam/blob/master/sdks/python/apache_beam/options/pipeline_options.py#L805] > class to be used in the JobServer interactions inside probable_runner.py. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-7933) Adding timeout to JobServer grpc calls
[ https://issues.apache.org/jira/browse/BEAM-7933?focusedWorklogId=321437=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-321437 ] ASF GitHub Bot logged work on BEAM-7933: Author: ASF GitHub Bot Created on: 01/Oct/19 18:37 Start Date: 01/Oct/19 18:37 Worklog Time Spent: 10m Work Description: ibzib commented on pull request #9673: [BEAM-7933] Add job server request timeout (default to 60 seconds) URL: https://github.com/apache/beam/pull/9673#discussion_r330211671 ## File path: sdks/python/apache_beam/runners/portability/local_job_service.py ## @@ -169,7 +169,7 @@ def GetMessageStream(self, request, context=None): resp = beam_job_api_pb2.JobMessagesResponse(message_response=msg) yield resp - def DescribePipelineOptions(self, request, context=None): + def DescribePipelineOptions(self, request, context=None, timeout=None): Review comment: It doesn't look like the `timeout` argument is actually used -- see other comment below This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 321437) Time Spent: 1h 50m (was: 1h 40m) > Adding timeout to JobServer grpc calls > -- > > Key: BEAM-7933 > URL: https://issues.apache.org/jira/browse/BEAM-7933 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Affects Versions: 2.14.0 >Reporter: Enrico Canzonieri >Assignee: Enrico Canzonieri >Priority: Minor > Labels: portability > Time Spent: 1h 50m > Remaining Estimate: 0h > > grpc calls to the JobServer from the Python SDK do not have timeouts. That > means that the call to pipeline.run()could hang forever if the JobServer is > not running (or failing to start). > E.g. > [https://github.com/apache/beam/blob/master/sdks/python/apache_beam/runners/portability/portable_runner.py#L307] > the call to Prepare() doesn't provide any timeout value and the same applies > to other JobServer requests. > As part of this ticket we could add a default timeout of 60 seconds as the > default timeout for http client. > Additionally, we could consider adding a --job-server-request-timeout to the > [PortableOptions|https://github.com/apache/beam/blob/master/sdks/python/apache_beam/options/pipeline_options.py#L805] > class to be used in the JobServer interactions inside probable_runner.py. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-7933) Adding timeout to JobServer grpc calls
[ https://issues.apache.org/jira/browse/BEAM-7933?focusedWorklogId=321346=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-321346 ] ASF GitHub Bot logged work on BEAM-7933: Author: ASF GitHub Bot Created on: 01/Oct/19 16:16 Start Date: 01/Oct/19 16:16 Worklog Time Spent: 10m Work Description: ecanzonieri commented on pull request #9673: [BEAM-7933] Add job server request timeout (default to 60 seconds) URL: https://github.com/apache/beam/pull/9673#discussion_r330147713 ## File path: sdks/python/apache_beam/runners/portability/portable_runner.py ## @@ -251,7 +253,11 @@ def send_options_request(max_retries=5): # This reports channel is READY but connections may fail # Seems to be only an issue on Mac with port forwardings return job_service.DescribePipelineOptions( - beam_job_api_pb2.DescribePipelineOptionsRequest()) + beam_job_api_pb2.DescribePipelineOptionsRequest(), + timeout=portable_options.job_server_timeout) Review comment: I fixed the tests by adding the timeout to the stubs of the LocalJobServicer. It works locally now, let's see if it passes on jenkins. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 321346) Time Spent: 1h 40m (was: 1.5h) > Adding timeout to JobServer grpc calls > -- > > Key: BEAM-7933 > URL: https://issues.apache.org/jira/browse/BEAM-7933 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Affects Versions: 2.14.0 >Reporter: Enrico Canzonieri >Assignee: Enrico Canzonieri >Priority: Minor > Labels: portability > Time Spent: 1h 40m > Remaining Estimate: 0h > > grpc calls to the JobServer from the Python SDK do not have timeouts. That > means that the call to pipeline.run()could hang forever if the JobServer is > not running (or failing to start). > E.g. > [https://github.com/apache/beam/blob/master/sdks/python/apache_beam/runners/portability/portable_runner.py#L307] > the call to Prepare() doesn't provide any timeout value and the same applies > to other JobServer requests. > As part of this ticket we could add a default timeout of 60 seconds as the > default timeout for http client. > Additionally, we could consider adding a --job-server-request-timeout to the > [PortableOptions|https://github.com/apache/beam/blob/master/sdks/python/apache_beam/options/pipeline_options.py#L805] > class to be used in the JobServer interactions inside probable_runner.py. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-7933) Adding timeout to JobServer grpc calls
[
https://issues.apache.org/jira/browse/BEAM-7933?focusedWorklogId=321344=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-321344
]
ASF GitHub Bot logged work on BEAM-7933:
Author: ASF GitHub Bot
Created on: 01/Oct/19 16:15
Start Date: 01/Oct/19 16:15
Worklog Time Spent: 10m
Work Description: ecanzonieri commented on pull request #9673:
[BEAM-7933] Add job server request timeout (default to 60 seconds)
URL: https://github.com/apache/beam/pull/9673#discussion_r330147442
##
File path: sdks/python/apache_beam/runners/portability/portable_runner.py
##
@@ -141,23 +141,25 @@ def _create_environment(options):
payload=(portable_options.environment_config.encode('ascii')
if portable_options.environment_config else None))
- def default_job_server(self, options):
+ def default_job_server(self, portable_options):
Review comment:
I reverted it.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 321344)
Time Spent: 1h 20m (was: 1h 10m)
> Adding timeout to JobServer grpc calls
> --
>
> Key: BEAM-7933
> URL: https://issues.apache.org/jira/browse/BEAM-7933
> Project: Beam
> Issue Type: Improvement
> Components: sdk-py-core
>Affects Versions: 2.14.0
>Reporter: Enrico Canzonieri
>Assignee: Enrico Canzonieri
>Priority: Minor
> Labels: portability
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> grpc calls to the JobServer from the Python SDK do not have timeouts. That
> means that the call to pipeline.run()could hang forever if the JobServer is
> not running (or failing to start).
> E.g.
> [https://github.com/apache/beam/blob/master/sdks/python/apache_beam/runners/portability/portable_runner.py#L307]
> the call to Prepare() doesn't provide any timeout value and the same applies
> to other JobServer requests.
> As part of this ticket we could add a default timeout of 60 seconds as the
> default timeout for http client.
> Additionally, we could consider adding a --job-server-request-timeout to the
> [PortableOptions|https://github.com/apache/beam/blob/master/sdks/python/apache_beam/options/pipeline_options.py#L805]
> class to be used in the JobServer interactions inside probable_runner.py.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-7933) Adding timeout to JobServer grpc calls
[ https://issues.apache.org/jira/browse/BEAM-7933?focusedWorklogId=320919=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-320919 ] ASF GitHub Bot logged work on BEAM-7933: Author: ASF GitHub Bot Created on: 30/Sep/19 22:36 Start Date: 30/Sep/19 22:36 Worklog Time Spent: 10m Work Description: ecanzonieri commented on pull request #9673: [BEAM-7933] Add job server request timeout (default to 60 seconds) URL: https://github.com/apache/beam/pull/9673#discussion_r329815849 ## File path: sdks/python/apache_beam/runners/portability/portable_runner.py ## @@ -251,7 +253,11 @@ def send_options_request(max_retries=5): # This reports channel is READY but connections may fail # Seems to be only an issue on Mac with port forwardings return job_service.DescribePipelineOptions( - beam_job_api_pb2.DescribePipelineOptionsRequest()) + beam_job_api_pb2.DescribePipelineOptionsRequest(), + timeout=portable_options.job_server_timeout) +except grpc.FutureTimeoutError: + # no retry for timeout errors Review comment: My guess is that we want to retry for other errors, but we should treat timeout as unretriable, so that we never wait for longer than the specified timeout (including retries). We could implement exponential backoff so that it will overall still wait for no longer than the timeout, but I'm not sure it's worth the additional complexity. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 320919) Time Spent: 1h 10m (was: 1h) > Adding timeout to JobServer grpc calls > -- > > Key: BEAM-7933 > URL: https://issues.apache.org/jira/browse/BEAM-7933 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Affects Versions: 2.14.0 >Reporter: Enrico Canzonieri >Assignee: Enrico Canzonieri >Priority: Minor > Labels: portability > Time Spent: 1h 10m > Remaining Estimate: 0h > > grpc calls to the JobServer from the Python SDK do not have timeouts. That > means that the call to pipeline.run()could hang forever if the JobServer is > not running (or failing to start). > E.g. > [https://github.com/apache/beam/blob/master/sdks/python/apache_beam/runners/portability/portable_runner.py#L307] > the call to Prepare() doesn't provide any timeout value and the same applies > to other JobServer requests. > As part of this ticket we could add a default timeout of 60 seconds as the > default timeout for http client. > Additionally, we could consider adding a --job-server-request-timeout to the > [PortableOptions|https://github.com/apache/beam/blob/master/sdks/python/apache_beam/options/pipeline_options.py#L805] > class to be used in the JobServer interactions inside probable_runner.py. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-7933) Adding timeout to JobServer grpc calls
[ https://issues.apache.org/jira/browse/BEAM-7933?focusedWorklogId=320900=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-320900 ] ASF GitHub Bot logged work on BEAM-7933: Author: ASF GitHub Bot Created on: 30/Sep/19 21:53 Start Date: 30/Sep/19 21:53 Worklog Time Spent: 10m Work Description: ecanzonieri commented on pull request #9673: [BEAM-7933] Add job server request timeout (default to 60 seconds) URL: https://github.com/apache/beam/pull/9673#discussion_r329803384 ## File path: sdks/python/apache_beam/runners/portability/portable_runner.py ## @@ -251,7 +253,11 @@ def send_options_request(max_retries=5): # This reports channel is READY but connections may fail # Seems to be only an issue on Mac with port forwardings return job_service.DescribePipelineOptions( - beam_job_api_pb2.DescribePipelineOptionsRequest()) + beam_job_api_pb2.DescribePipelineOptionsRequest(), + timeout=portable_options.job_server_timeout) Review comment: My guess is that this is invoking the LocalJobServicer that doesn't have the timeout argument. I'll try to repro locally to verify. If that's the case I'd like to add the timeout argument to all LocalJobServicer methods. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 320900) Time Spent: 1h (was: 50m) > Adding timeout to JobServer grpc calls > -- > > Key: BEAM-7933 > URL: https://issues.apache.org/jira/browse/BEAM-7933 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Affects Versions: 2.14.0 >Reporter: Enrico Canzonieri >Assignee: Enrico Canzonieri >Priority: Minor > Labels: portability > Time Spent: 1h > Remaining Estimate: 0h > > grpc calls to the JobServer from the Python SDK do not have timeouts. That > means that the call to pipeline.run()could hang forever if the JobServer is > not running (or failing to start). > E.g. > [https://github.com/apache/beam/blob/master/sdks/python/apache_beam/runners/portability/portable_runner.py#L307] > the call to Prepare() doesn't provide any timeout value and the same applies > to other JobServer requests. > As part of this ticket we could add a default timeout of 60 seconds as the > default timeout for http client. > Additionally, we could consider adding a --job-server-request-timeout to the > [PortableOptions|https://github.com/apache/beam/blob/master/sdks/python/apache_beam/options/pipeline_options.py#L805] > class to be used in the JobServer interactions inside probable_runner.py. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-7933) Adding timeout to JobServer grpc calls
[
https://issues.apache.org/jira/browse/BEAM-7933?focusedWorklogId=320886=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-320886
]
ASF GitHub Bot logged work on BEAM-7933:
Author: ASF GitHub Bot
Created on: 30/Sep/19 21:29
Start Date: 30/Sep/19 21:29
Worklog Time Spent: 10m
Work Description: ecanzonieri commented on pull request #9673:
[BEAM-7933] Add job server request timeout (default to 60 seconds)
URL: https://github.com/apache/beam/pull/9673#discussion_r329794942
##
File path: sdks/python/apache_beam/runners/portability/portable_runner.py
##
@@ -141,23 +141,25 @@ def _create_environment(options):
payload=(portable_options.environment_config.encode('ascii')
if portable_options.environment_config else None))
- def default_job_server(self, options):
+ def default_job_server(self, portable_options):
# TODO Provide a way to specify a container Docker URL
# https://issues.apache.org/jira/browse/BEAM-6328
if not self._dockerized_job_server:
self._dockerized_job_server = job_server.StopOnExitJobServer(
job_server.DockerizedJobServer())
return self._dockerized_job_server
- def create_job_service(self, options):
-job_endpoint = options.view_as(PortableOptions).job_endpoint
+ def create_job_service(self, portable_options):
Review comment:
I changed it because it is actually immediately converted back to
portable_options. Basically the caller takes options and converts it to
portable_options, then to the create_job_service method it passes options which
is again converted to portable_options. It seems to be that we could pass
portable_options in the first place and simplify the code a bit. However, I
don't have a strong option, I'll revert back to options.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 320886)
Time Spent: 50m (was: 40m)
> Adding timeout to JobServer grpc calls
> --
>
> Key: BEAM-7933
> URL: https://issues.apache.org/jira/browse/BEAM-7933
> Project: Beam
> Issue Type: Improvement
> Components: sdk-py-core
>Affects Versions: 2.14.0
>Reporter: Enrico Canzonieri
>Assignee: Enrico Canzonieri
>Priority: Minor
> Labels: portability
> Time Spent: 50m
> Remaining Estimate: 0h
>
> grpc calls to the JobServer from the Python SDK do not have timeouts. That
> means that the call to pipeline.run()could hang forever if the JobServer is
> not running (or failing to start).
> E.g.
> [https://github.com/apache/beam/blob/master/sdks/python/apache_beam/runners/portability/portable_runner.py#L307]
> the call to Prepare() doesn't provide any timeout value and the same applies
> to other JobServer requests.
> As part of this ticket we could add a default timeout of 60 seconds as the
> default timeout for http client.
> Additionally, we could consider adding a --job-server-request-timeout to the
> [PortableOptions|https://github.com/apache/beam/blob/master/sdks/python/apache_beam/options/pipeline_options.py#L805]
> class to be used in the JobServer interactions inside probable_runner.py.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-7933) Adding timeout to JobServer grpc calls
[ https://issues.apache.org/jira/browse/BEAM-7933?focusedWorklogId=319810=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-319810 ] ASF GitHub Bot logged work on BEAM-7933: Author: ASF GitHub Bot Created on: 27/Sep/19 20:50 Start Date: 27/Sep/19 20:50 Worklog Time Spent: 10m Work Description: ibzib commented on pull request #9673: [BEAM-7933] Add job server request timeout (default to 60 seconds) URL: https://github.com/apache/beam/pull/9673#discussion_r329245933 ## File path: sdks/python/apache_beam/runners/portability/portable_runner.py ## @@ -251,7 +253,11 @@ def send_options_request(max_retries=5): # This reports channel is READY but connections may fail # Seems to be only an issue on Mac with port forwardings return job_service.DescribePipelineOptions( - beam_job_api_pb2.DescribePipelineOptionsRequest()) + beam_job_api_pb2.DescribePipelineOptionsRequest(), + timeout=portable_options.job_server_timeout) Review comment: I think this line is causing tests to fail. Also, we're already retrying this request. So I would either keep the existing logic or add the new timeout, but not both. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 319810) Time Spent: 0.5h (was: 20m) > Adding timeout to JobServer grpc calls > -- > > Key: BEAM-7933 > URL: https://issues.apache.org/jira/browse/BEAM-7933 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Affects Versions: 2.14.0 >Reporter: Enrico Canzonieri >Assignee: Enrico Canzonieri >Priority: Minor > Labels: portability > Time Spent: 0.5h > Remaining Estimate: 0h > > grpc calls to the JobServer from the Python SDK do not have timeouts. That > means that the call to pipeline.run()could hang forever if the JobServer is > not running (or failing to start). > E.g. > [https://github.com/apache/beam/blob/master/sdks/python/apache_beam/runners/portability/portable_runner.py#L307] > the call to Prepare() doesn't provide any timeout value and the same applies > to other JobServer requests. > As part of this ticket we could add a default timeout of 60 seconds as the > default timeout for http client. > Additionally, we could consider adding a --job-server-request-timeout to the > [PortableOptions|https://github.com/apache/beam/blob/master/sdks/python/apache_beam/options/pipeline_options.py#L805] > class to be used in the JobServer interactions inside probable_runner.py. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-7933) Adding timeout to JobServer grpc calls
[
https://issues.apache.org/jira/browse/BEAM-7933?focusedWorklogId=319809=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-319809
]
ASF GitHub Bot logged work on BEAM-7933:
Author: ASF GitHub Bot
Created on: 27/Sep/19 20:50
Start Date: 27/Sep/19 20:50
Worklog Time Spent: 10m
Work Description: ibzib commented on pull request #9673: [BEAM-7933] Add
job server request timeout (default to 60 seconds)
URL: https://github.com/apache/beam/pull/9673#discussion_r329242301
##
File path: sdks/python/apache_beam/runners/portability/portable_runner.py
##
@@ -141,23 +141,25 @@ def _create_environment(options):
payload=(portable_options.environment_config.encode('ascii')
if portable_options.environment_config else None))
- def default_job_server(self, options):
+ def default_job_server(self, portable_options):
Review comment:
why this change?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 319809)
Time Spent: 0.5h (was: 20m)
> Adding timeout to JobServer grpc calls
> --
>
> Key: BEAM-7933
> URL: https://issues.apache.org/jira/browse/BEAM-7933
> Project: Beam
> Issue Type: Improvement
> Components: sdk-py-core
>Affects Versions: 2.14.0
>Reporter: Enrico Canzonieri
>Assignee: Enrico Canzonieri
>Priority: Minor
> Labels: portability
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> grpc calls to the JobServer from the Python SDK do not have timeouts. That
> means that the call to pipeline.run()could hang forever if the JobServer is
> not running (or failing to start).
> E.g.
> [https://github.com/apache/beam/blob/master/sdks/python/apache_beam/runners/portability/portable_runner.py#L307]
> the call to Prepare() doesn't provide any timeout value and the same applies
> to other JobServer requests.
> As part of this ticket we could add a default timeout of 60 seconds as the
> default timeout for http client.
> Additionally, we could consider adding a --job-server-request-timeout to the
> [PortableOptions|https://github.com/apache/beam/blob/master/sdks/python/apache_beam/options/pipeline_options.py#L805]
> class to be used in the JobServer interactions inside probable_runner.py.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-7933) Adding timeout to JobServer grpc calls
[
https://issues.apache.org/jira/browse/BEAM-7933?focusedWorklogId=319811=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-319811
]
ASF GitHub Bot logged work on BEAM-7933:
Author: ASF GitHub Bot
Created on: 27/Sep/19 20:50
Start Date: 27/Sep/19 20:50
Worklog Time Spent: 10m
Work Description: ibzib commented on pull request #9673: [BEAM-7933] Add
job server request timeout (default to 60 seconds)
URL: https://github.com/apache/beam/pull/9673#discussion_r329234451
##
File path: sdks/python/apache_beam/options/pipeline_options.py
##
@@ -808,11 +808,15 @@ class PortableOptions(PipelineOptions):
"""
@classmethod
def _add_argparse_args(cls, parser):
-parser.add_argument('--job_endpoint',
-default=None,
-help=
-('Job service endpoint to use. Should be in the form '
- 'of address and port, e.g. localhost:3000'))
+parser.add_argument(
+'--job_endpoint', default=None,
+help=('Job service endpoint to use. Should be in the form of address '
+ 'and port, e.g. localhost:3000'))
+parser.add_argument(
+'--job_server_timeout', default=60, type=int,
Review comment:
indentation here looks off
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 319811)
Time Spent: 0.5h (was: 20m)
> Adding timeout to JobServer grpc calls
> --
>
> Key: BEAM-7933
> URL: https://issues.apache.org/jira/browse/BEAM-7933
> Project: Beam
> Issue Type: Improvement
> Components: sdk-py-core
>Affects Versions: 2.14.0
>Reporter: Enrico Canzonieri
>Assignee: Enrico Canzonieri
>Priority: Minor
> Labels: portability
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> grpc calls to the JobServer from the Python SDK do not have timeouts. That
> means that the call to pipeline.run()could hang forever if the JobServer is
> not running (or failing to start).
> E.g.
> [https://github.com/apache/beam/blob/master/sdks/python/apache_beam/runners/portability/portable_runner.py#L307]
> the call to Prepare() doesn't provide any timeout value and the same applies
> to other JobServer requests.
> As part of this ticket we could add a default timeout of 60 seconds as the
> default timeout for http client.
> Additionally, we could consider adding a --job-server-request-timeout to the
> [PortableOptions|https://github.com/apache/beam/blob/master/sdks/python/apache_beam/options/pipeline_options.py#L805]
> class to be used in the JobServer interactions inside probable_runner.py.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-7933) Adding timeout to JobServer grpc calls
[ https://issues.apache.org/jira/browse/BEAM-7933?focusedWorklogId=319812=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-319812 ] ASF GitHub Bot logged work on BEAM-7933: Author: ASF GitHub Bot Created on: 27/Sep/19 20:50 Start Date: 27/Sep/19 20:50 Worklog Time Spent: 10m Work Description: ibzib commented on pull request #9673: [BEAM-7933] Add job server request timeout (default to 60 seconds) URL: https://github.com/apache/beam/pull/9673#discussion_r329240622 ## File path: sdks/python/apache_beam/runners/portability/portable_runner.py ## @@ -251,7 +253,11 @@ def send_options_request(max_retries=5): # This reports channel is READY but connections may fail # Seems to be only an issue on Mac with port forwardings return job_service.DescribePipelineOptions( - beam_job_api_pb2.DescribePipelineOptionsRequest()) + beam_job_api_pb2.DescribePipelineOptionsRequest(), + timeout=portable_options.job_server_timeout) +except grpc.FutureTimeoutError: + # no retry for timeout errors Review comment: Are we sure we don't want retries? We could do exponential backoff, similar to what SubprocessJobServer already does. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 319812) Time Spent: 0.5h (was: 20m) > Adding timeout to JobServer grpc calls > -- > > Key: BEAM-7933 > URL: https://issues.apache.org/jira/browse/BEAM-7933 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Affects Versions: 2.14.0 >Reporter: Enrico Canzonieri >Assignee: Enrico Canzonieri >Priority: Minor > Labels: portability > Time Spent: 0.5h > Remaining Estimate: 0h > > grpc calls to the JobServer from the Python SDK do not have timeouts. That > means that the call to pipeline.run()could hang forever if the JobServer is > not running (or failing to start). > E.g. > [https://github.com/apache/beam/blob/master/sdks/python/apache_beam/runners/portability/portable_runner.py#L307] > the call to Prepare() doesn't provide any timeout value and the same applies > to other JobServer requests. > As part of this ticket we could add a default timeout of 60 seconds as the > default timeout for http client. > Additionally, we could consider adding a --job-server-request-timeout to the > [PortableOptions|https://github.com/apache/beam/blob/master/sdks/python/apache_beam/options/pipeline_options.py#L805] > class to be used in the JobServer interactions inside probable_runner.py. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-7933) Adding timeout to JobServer grpc calls
[ https://issues.apache.org/jira/browse/BEAM-7933?focusedWorklogId=319813=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-319813 ] ASF GitHub Bot logged work on BEAM-7933: Author: ASF GitHub Bot Created on: 27/Sep/19 20:50 Start Date: 27/Sep/19 20:50 Worklog Time Spent: 10m Work Description: ibzib commented on pull request #9673: [BEAM-7933] Add job server request timeout (default to 60 seconds) URL: https://github.com/apache/beam/pull/9673#discussion_r329244609 ## File path: sdks/python/apache_beam/runners/portability/portable_runner.py ## @@ -240,7 +242,7 @@ def run_pipeline(self, pipeline, options): known_runner_urns=flink_known_urns, partial=True) -job_service = self.create_job_service(options) +job_service = self.create_job_service(portable_options) Review comment: Not sure this is necessary? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 319813) > Adding timeout to JobServer grpc calls > -- > > Key: BEAM-7933 > URL: https://issues.apache.org/jira/browse/BEAM-7933 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Affects Versions: 2.14.0 >Reporter: Enrico Canzonieri >Assignee: Enrico Canzonieri >Priority: Minor > Labels: portability > Time Spent: 0.5h > Remaining Estimate: 0h > > grpc calls to the JobServer from the Python SDK do not have timeouts. That > means that the call to pipeline.run()could hang forever if the JobServer is > not running (or failing to start). > E.g. > [https://github.com/apache/beam/blob/master/sdks/python/apache_beam/runners/portability/portable_runner.py#L307] > the call to Prepare() doesn't provide any timeout value and the same applies > to other JobServer requests. > As part of this ticket we could add a default timeout of 60 seconds as the > default timeout for http client. > Additionally, we could consider adding a --job-server-request-timeout to the > [PortableOptions|https://github.com/apache/beam/blob/master/sdks/python/apache_beam/options/pipeline_options.py#L805] > class to be used in the JobServer interactions inside probable_runner.py. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-7933) Adding timeout to JobServer grpc calls
[
https://issues.apache.org/jira/browse/BEAM-7933?focusedWorklogId=319814=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-319814
]
ASF GitHub Bot logged work on BEAM-7933:
Author: ASF GitHub Bot
Created on: 27/Sep/19 20:50
Start Date: 27/Sep/19 20:50
Worklog Time Spent: 10m
Work Description: ibzib commented on pull request #9673: [BEAM-7933] Add
job server request timeout (default to 60 seconds)
URL: https://github.com/apache/beam/pull/9673#discussion_r329244408
##
File path: sdks/python/apache_beam/runners/portability/portable_runner.py
##
@@ -141,23 +141,25 @@ def _create_environment(options):
payload=(portable_options.environment_config.encode('ascii')
if portable_options.environment_config else None))
- def default_job_server(self, options):
+ def default_job_server(self, portable_options):
# TODO Provide a way to specify a container Docker URL
# https://issues.apache.org/jira/browse/BEAM-6328
if not self._dockerized_job_server:
self._dockerized_job_server = job_server.StopOnExitJobServer(
job_server.DockerizedJobServer())
return self._dockerized_job_server
- def create_job_service(self, options):
-job_endpoint = options.view_as(PortableOptions).job_endpoint
+ def create_job_service(self, portable_options):
Review comment:
I'd just leave this as `options`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 319814)
Time Spent: 40m (was: 0.5h)
> Adding timeout to JobServer grpc calls
> --
>
> Key: BEAM-7933
> URL: https://issues.apache.org/jira/browse/BEAM-7933
> Project: Beam
> Issue Type: Improvement
> Components: sdk-py-core
>Affects Versions: 2.14.0
>Reporter: Enrico Canzonieri
>Assignee: Enrico Canzonieri
>Priority: Minor
> Labels: portability
> Time Spent: 40m
> Remaining Estimate: 0h
>
> grpc calls to the JobServer from the Python SDK do not have timeouts. That
> means that the call to pipeline.run()could hang forever if the JobServer is
> not running (or failing to start).
> E.g.
> [https://github.com/apache/beam/blob/master/sdks/python/apache_beam/runners/portability/portable_runner.py#L307]
> the call to Prepare() doesn't provide any timeout value and the same applies
> to other JobServer requests.
> As part of this ticket we could add a default timeout of 60 seconds as the
> default timeout for http client.
> Additionally, we could consider adding a --job-server-request-timeout to the
> [PortableOptions|https://github.com/apache/beam/blob/master/sdks/python/apache_beam/options/pipeline_options.py#L805]
> class to be used in the JobServer interactions inside probable_runner.py.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-7933) Adding timeout to JobServer grpc calls
[ https://issues.apache.org/jira/browse/BEAM-7933?focusedWorklogId=319263=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-319263 ] ASF GitHub Bot logged work on BEAM-7933: Author: ASF GitHub Bot Created on: 26/Sep/19 22:51 Start Date: 26/Sep/19 22:51 Worklog Time Spent: 10m Work Description: ecanzonieri commented on issue #9673: [BEAM-7933] Add job server request timeout (default to 60 seconds) URL: https://github.com/apache/beam/pull/9673#issuecomment-535714867 R: @ibzib @aaltay This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 319263) Time Spent: 20m (was: 10m) > Adding timeout to JobServer grpc calls > -- > > Key: BEAM-7933 > URL: https://issues.apache.org/jira/browse/BEAM-7933 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Affects Versions: 2.14.0 >Reporter: Enrico Canzonieri >Assignee: Enrico Canzonieri >Priority: Minor > Labels: portability > Time Spent: 20m > Remaining Estimate: 0h > > grpc calls to the JobServer from the Python SDK do not have timeouts. That > means that the call to pipeline.run()could hang forever if the JobServer is > not running (or failing to start). > E.g. > [https://github.com/apache/beam/blob/master/sdks/python/apache_beam/runners/portability/portable_runner.py#L307] > the call to Prepare() doesn't provide any timeout value and the same applies > to other JobServer requests. > As part of this ticket we could add a default timeout of 60 seconds as the > default timeout for http client. > Additionally, we could consider adding a --job-server-request-timeout to the > [PortableOptions|https://github.com/apache/beam/blob/master/sdks/python/apache_beam/options/pipeline_options.py#L805] > class to be used in the JobServer interactions inside probable_runner.py. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-7933) Adding timeout to JobServer grpc calls
[ https://issues.apache.org/jira/browse/BEAM-7933?focusedWorklogId=319262=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-319262 ] ASF GitHub Bot logged work on BEAM-7933: Author: ASF GitHub Bot Created on: 26/Sep/19 22:48 Start Date: 26/Sep/19 22:48 Worklog Time Spent: 10m Work Description: ecanzonieri commented on pull request #9673: [BEAM-7933] Add job server request timeout (default to 60 seconds) URL: https://github.com/apache/beam/pull/9673 Add a new option --job_server_timeout that default to 60 seconds. The request timeout is user for all job server requests (with exception to the ones that are expected to last long or hang). The request timeout is also used upon channel creation so that the Beam driver will fail if the job server is not available. Let me know if the option name looks fine or it needs renaming. Finally, there are other grpc requests (non job service) that I'm not covering as part of this pr. Let me know if I should add the timeout to those as well. Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/) Python | [