chamikaramj commented on a change in pull request #4136: [BEAM-3184] Added
ProxyInfoFromEnvironmentVar() & GetNewHttp() methods for GCS
URL: https://github.com/apache/beam/pull/4136#discussion_r153058069
##########
File path: sdks/python/apache_beam/io/gcp/gcsio.py
##########
@@ -87,6 +87,50 @@
MAX_BATCH_OPERATION_SIZE = 100
+def ProxyInfoFromEnvironmentVar(proxy_env_var):
+ """Reads proxy info from the environment and converts to httplib2.ProxyInfo.
+ Args:
+ proxy_env_var: Environment variable string to read, such as http_proxy or
+ https_proxy.
+ Returns:
+ httplib2.ProxyInfo constructed from the environment string.
+ """
+ proxy_url = os.environ.get(proxy_env_var)
+ if not proxy_url or not proxy_env_var.lower().startswith('http'):
+ return httplib2.ProxyInfo(httplib2.socks.PROXY_TYPE_HTTP, None, 0)
+ proxy_protocol = proxy_env_var.lower().split('_')[0]
+ if not proxy_url.lower().startswith('http'):
+ # proxy_info_from_url requires a protocol, which is always http or https.
+ proxy_url = proxy_protocol + '://' + proxy_url
+ return httplib2.proxy_info_from_url(proxy_url, method=proxy_protocol)
+
+def GetNewHttp(http_class=httplib2.Http, **kwargs):
+ """Creates and returns a new httplib2.Http instance.
+ Args:
+ http_class: Optional custom Http class to use.
+ **kwargs: Arguments to pass to http_class constructor.
+ Returns:
+ An initialized httplib2.Http instance.
+ """
+ proxy_info = httplib2.ProxyInfo(
+ proxy_type=3,
+ proxy_host=None,
+ proxy_port=None,
+ proxy_user=None,
+ proxy_pass=None,
+ proxy_rdns=None
+ )
+
+ for proxy_env_var in ['http_proxy', 'HTTP_PROXY', 'https_proxy',
'HTTPS_PROXY']:
Review comment:
Why do we have to mention all these values here ? Isn't it enough to use a
single variable (say, http_proxy) and ask users to always use that ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
