Default plus a flag to override sounds reasonable. Although from Dataflow experience I do not remember timeouts causing issues and each new added flag adds complexity. What do others think?
On Thu, Aug 8, 2019 at 11:38 AM Kyle Weaver <[email protected]> wrote: > If we do make a default, I still think it should be configurable via a > flag. I can't think of why the prepare, stage artifact, job state, or job > message requests might take more than 60 seconds, but you never know, > particularly with artifact staging, which might be uploading artifacts to > distributed storage. > > I assume the run request itself would not be subject to timeouts, as > running the pipeline can be assumed to take significantly longer than the > setup work. > > Kyle Weaver | Software Engineer | github.com/ibzib | [email protected] > > > On Thu, Aug 8, 2019 at 11:20 AM Enrico Canzonieri <[email protected]> wrote: > >> Default timeout with no flag may work as well. >> The main consideration here is whether some api calls may take longer >> than 60 seconds because of the complexity of the users' Beam pipeline. E.g. >> Could job_service.Prepare() take longer than 60 seconds if the given Beam >> pipeline is extremely complex? >> >> Basically if there are cases when the user code may cause the call >> duration to increase to the point the timeout prevents submitting the app >> itself then we should consider having a flag. >> >> On 2019/08/07 20:13:12, Ahmet Altay wrote: >> > Could we pick a default timeout value instead of introducing a flag? We >> use> >> > 60 seconds as the default timeout for http client [1], we can do the >> same> >> > here.> >> > >> > [1]> >> > >> https://github.com/apache/beam/blob/3a182d64c86ad038692800f5c343659ab0b935b0/sdks/python/apache_beam/internal/http_client.py#L32> >> >> > >> > On Wed, Aug 7, 2019 at 11:53 AM enrico canzonieri > >> > wrote:> >> > >> > > Hello,> >> > >> >> > > I noticed that the calls to the JobServer from the Python SDK do not >> have> >> > > timeouts. If I'm not mistaken that means that the call to >> pipeline.run()> >> > > could hang forever if the JobServer is not running (or failing to >> start).> >> > > E.g.> >> > > >> https://github.com/apache/beam/blob/master/sdks/python/apache_beam/runners/portability/portable_runner.py#L307> >> >> > > the call to Prepare() doesn't provide any timeout value and the same> >> > > applies to other JobServer requests.> >> > > I was considering adding a --job-server-request-timeout to the> >> > > PortableOptions> >> > > > >> > > class to be used in the JobServer interactions inside >> probable_runner.py.> >> > > Is there any specific reason why the timeout is not currently >> supported?> >> > > Does anybody have any objection adding the jobserver timeout? I >> could> >> > > volunteer to file a ticket and submit a pr for this.> >> > >> >> > > Cheers,> >> > > Enrico Canzonieri> >> > >> >> > >> >>
