[ https://issues.apache.org/jira/browse/BEAM-6033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Udi Meiri updated BEAM-6033: ---------------------------- Description: Ideally solve both issues below in one PR, but issue 1 has priority as it can halt a pipeline. Issue 1: Datastore client (and other httplib2-based clients for GCS, Dataflow, BigQuery, etc.) doesn't set a socket timeout. This can cause _flush_batch() in datastoreio.py to block forever waiting for a response. This issue is very similar to https://issues.apache.org/jira/browse/BEAM-5915 and the solution should be similar. Issue 2: Standardize use of proxy environment settings, as in gcsio: https://github.com/apache/beam/blob/8d3389df78aa2e0a0de06b7c5743ca3530dec4ac/sdks/python/apache_beam/io/gcp/gcsio.py#L136 Issue for proxy settings: https://issues.apache.org/jira/browse/BEAM-3184 was: Ideally solve both issues below in one PR, but issue 1 has priority as it can halt a pipeline. Issue 1: Datastore client (and other httplib2-based clients for GCS, Dataflow, BigQuery, etc.) doesn't set a socket timeout. This can cause _flush_batch() in datastoreio.py to block forever waiting for a response. This issue is very similar to https://issues.apache.org/jira/browse/BEAM-5915 and the solution should be similar. Issue 2: Standardize use of proxy environment settings, as in gcsio: https://github.com/apache/beam/blob/8d3389df78aa2e0a0de06b7c5743ca3530dec4ac/sdks/python/apache_beam/io/gcp/gcsio.py#L136 > normalize httplib2.Http initialization and usage > ------------------------------------------------ > > Key: BEAM-6033 > URL: https://issues.apache.org/jira/browse/BEAM-6033 > Project: Beam > Issue Type: Bug > Components: sdk-py-core > Reporter: Udi Meiri > Assignee: Ahmet Altay > Priority: Major > > Ideally solve both issues below in one PR, but issue 1 has priority as it can > halt a pipeline. > Issue 1: > Datastore client (and other httplib2-based clients for GCS, Dataflow, > BigQuery, etc.) doesn't set a socket timeout. > This can cause _flush_batch() in datastoreio.py to block forever waiting for > a response. > This issue is very similar to https://issues.apache.org/jira/browse/BEAM-5915 > and the solution should be similar. > Issue 2: > Standardize use of proxy environment settings, as in gcsio: > https://github.com/apache/beam/blob/8d3389df78aa2e0a0de06b7c5743ca3530dec4ac/sdks/python/apache_beam/io/gcp/gcsio.py#L136 > Issue for proxy settings: https://issues.apache.org/jira/browse/BEAM-3184 -- This message was sent by Atlassian JIRA (v7.6.3#76005)