Re: PUT with proxy-support
Running pycurl-7.19.0/setup.py -q bdist_egg --dist-dir
/tmp/easy_install-2ZCa8v/pycurl-7.19.0/egg-dist-tmp-DyHFls
Using curl-config (libcurl 7.12.1)
src/pycurl.c:42:20: Python.h: No such file or directory
src/pycurl.c:43:22: pythread.h: No such file or directory
src/pycurl.c:58:4: #error Need Python version 2.2 or greater to
compile pycurl.
src/pycurl.c:61:4: #error Need libcurl version 7.19.0 or greater to
compile pycurl.
[... Error Clipped]
error: Setup script exited with error: command '/usr/bin/gcc' failed
with exit status 1
I'm not familiar with red hat. But looks like this is now a sysadmin
problem. I could install pycurl under freebsd and ubuntu too.
I have some code that might be able to PUT through a proxy for you.
Please look at the attached file.
You can setup a proxy this way:
import MozzillaEmulator
MozzillaEmulator.DEFAULT_PROXIES = {
'http':'http://user:passw...@proxy.host.com:3128',
'https':'http://user:passw...@proxy.host.com:3128',
}
And here is how you use it:
dl = MozillaEmulator.MozillaEmulator()
put_url = http://some_url_to_put_to;
headers = {'Content-Type':'application/xml;charset=UTF-8'}
data = open(some_file,rb).read()
response = dl.download(put_url,data,headers,put_method=True)
If you don't set the put_method flag, then it will POST instead. (But in
that case, post data must be a valid post data field fields and values.)
If you don't give post_data then it will GET. The PUT method was tested
without a proxy, and the GET and POST methods were tested with and
without proxy.
Actually I have tried to do PUT through a proxy, but it didn't work. But
it was a restriction on the proxy side. (As far as I can recall, the
error message came from the proxy server, something about unsupported
request?). So it *might* work for you, with the right proxy server that
supports PUT requests.
Also be aware that this version uses urllib2. Although it can use https
protocol, it doesn't check the server's certificate.
Good luck,
Laszlo
#--
#
# Author: Laszlo Nagy
#
# Copyright: (c) 2005 by Szoftver Messias Bt.
# Licence: BSD style
#
# Setup urllib so it will use Mozilla user agent settings.
#
#
#--
import os
import hashlib
import urllib
import urllib2
import mimetypes
#from gzip import GzipFile
import cStringIO
from cPickle import loads,dumps
import cookielib
DEFAULT_PROXIES = None
class MozillaCacher(object):
A dictionary like object, that can cache results on a storage device.
def __init__(self,cachedir='.cache'):
self.cachedir = cachedir
if not os.path.isdir(cachedir):
os.mkdir(cachedir)
def name2fname(self,name):
return os.path.join(self.cachedir,name)
def __getitem__(self,name):
if not isinstance(name,str):
raise TypeError()
fname = self.name2fname(name)
if os.path.isfile(fname):
return file(fname,'rb').read()
else:
raise IndexError()
def __setitem__(self,name,value):
if not isinstance(name,str):
raise TypeError()
fname = self.name2fname(name)
if os.path.isfile(fname):
os.unlink(fname)
f = file(fname,'wb+')
try:
f.write(value)
finally:
f.close()
def __delitem__(self,name):
if not isinstance(name,str):
raise TypeError()
fname = self.name2fname(name)
if os.path.isfile(fname):
os.unlink(fname)
def __iter__(self):
raise NotImplementedError()
def has_key(self,name):
return os.path.isfile(self.name2fname(name))
class MozillaEmulator(object):
def __init__(self,cacher={},trycount=0,proxies=None):
Create a new MozillaEmulator object.
@param cacher: A dictionary like object, that can cache search results on a storage device.
You can use a simple dictionary here, but it is not recommended.
You can also put None here to disable caching completely.
@param trycount: The download() method will retry the operation if it fails. You can specify -1 for infinite retrying.
A value of 0 means no retrying. A value of 1 means one retry. etc.
:param proxies: It must be a dict like this:
{
'http': 'http://user:pas...@www.example.com:3128/',
'https': 'http://user:pas...@www.example.com:3128/',
}
If you pass None then DEFAULT_PROXIES will be used. To force
direct connection, use an empty dict.
This is a simplified version of urllib2 proxy handling.
To use direct connection, call with proxies=None.
Look at the example here:
http://docs.python.org/release/2.5.2/lib/urllib2-examples.html
Re: PUT with proxy-support
On Fri, Aug 26, 2011 at 3:15 PM, Laszlo Nagy gand...@shopzeus.com wrote:
Running pycurl-7.19.0/setup.py -q bdist_egg --dist-dir
/tmp/easy_install-2ZCa8v/**pycurl-7.19.0/egg-dist-tmp-**DyHFls
Using curl-config (libcurl 7.12.1)
src/pycurl.c:42:20: Python.h: No such file or directory
src/pycurl.c:43:22: pythread.h: No such file or directory
src/pycurl.c:58:4: #error Need Python version 2.2 or greater to compile
pycurl.
src/pycurl.c:61:4: #error Need libcurl version 7.19.0 or greater to
compile pycurl.
[... Error Clipped]
error: Setup script exited with error: command '/usr/bin/gcc' failed with
exit status 1
I'm not familiar with red hat. But looks like this is now a sysadmin
problem. I could install pycurl under freebsd and ubuntu too.
I have some code that might be able to PUT through a proxy for you. Please
look at the attached file.
You can setup a proxy this way:
import MozzillaEmulator
MozzillaEmulator.DEFAULT_**PROXIES = {
'http':'http://user:password@**proxy.host.com:3128http://user:passw...@proxy.host.com:3128
',
'https':'http://user:password@**proxy.host.com:3128http://user:passw...@proxy.host.com:3128
',
}
And here is how you use it:
dl = MozillaEmulator.**MozillaEmulator()
put_url = http://some_url_to_put_to;
headers = {'Content-Type':'application/**xml;charset=UTF-8'}
data = open(some_file,rb).read()
response = dl.download(put_url,data,**headers,put_method=True)
If you don't set the put_method flag, then it will POST instead. (But in
that case, post data must be a valid post data field fields and values.) If
you don't give post_data then it will GET. The PUT method was tested without
a proxy, and the GET and POST methods were tested with and without proxy.
Actually I have tried to do PUT through a proxy, but it didn't work. But it
was a restriction on the proxy side. (As far as I can recall, the error
message came from the proxy server, something about unsupported request?).
So it *might* work for you, with the right proxy server that supports PUT
requests.
Also be aware that this version uses urllib2. Although it can use https
protocol, it doesn't check the server's certificate.
Mozilla Emulator works like a charm.
Also managed to install pycurl and the same works too.
Thanks. :)
~Shashwat
--
http://mail.python.org/mailman/listinfo/python-list
Re: PUT with proxy-support
On 25/08/11 13:07, Shashwat Anand wrote: I want to make a PUT request. I need some headers of my own ( certificates etc ) and I need to mandatorily use a proxy. Also the url is of the form http://www.xyz.com/abc and I don't have permission to put data on http://www.xyz.com while I do have permission to put data on http://www.xyz.com/abc I tried httplib, httplib2, urllib2 with no avail. What did you try? What problems did you run into? I'm sure there is a way in Python, and chances are you were already close to finding it -- show us what you tried, what actually happened, including any error messages in full, and what you wanted to happen. Thomas I managed to do this via command line curl: $ curl http:/xyz.com/testing/shashwat/test.txt http://xyz.com/testing/shashwat/test.txt -T test.txt -H sw-version: 1.0 -H CA-Cert-Auth:v=1;a=yxyz.prod;h=10.10.0.1;t=1316594650;s=.AeEYJMMfElN74fnWD3GlXJ4J.1KiQFg-- --proxy proxy.xyz.com:3128 http://proxy.xyz.com:3128 -H Content-Type:text/plain Is there a way to do it in python apart from using command line curl in python. The machine is RHEL4 and is giving hard time installing pycurl. -- http://mail.python.org/mailman/listinfo/python-list
Re: PUT with proxy-support
On Thu, Aug 25, 2011 at 4:48 PM, Thomas Jollans t...@jollybox.de wrote:
On 25/08/11 13:07, Shashwat Anand wrote:
I want to make a PUT request.
I need some headers of my own ( certificates etc ) and I need to
mandatorily use a proxy.
Also the url is of the form http://www.xyz.com/abc and I don't have
permission to put data
on http://www.xyz.com while I do have permission to put data
on http://www.xyz.com/abc
I tried httplib, httplib2, urllib2 with no avail.
What did you try? What problems did you run into?
I'm sure there is a way in Python, and chances are you were already
close to finding it -- show us what you tried, what actually happened,
including any error messages in full, and what you wanted to happen.
Thomas
Hi Thomas,
so one of my tries was:
import urllib
import urllib2
import httplib
import httplib2
url = 'http://alatheia.zenfs.com/testing/shashwat'
body_content = 'CONTENT GOES HERE'
proxy = 'ca-proxy.corp.xyz.com:3128'
params = {
'x-sws-version' : '1.0',
'x-sws-access' : 'public',
'User-Agent' : 'CacheSystem',
'Cache-Control' : 'public',
'Content-Type' : 'text/plain',
'App-Auth' :
'v=1;a=client.alatheia.prod;h=10.16.19.23;t=1316594650;s=AeEYJMMfElN74fnWD3GlXJ4J.1KiQFg--',
}
httplib2.debuglevel=4
h = httplib2.Http(proxy_info = httplib2.ProxyInfo(3, '
ca-proxy.corp.xyz.com:3128', 3128))
resp, content = h.request(url, PUT, body=body_content, headers = params)
print resp
print content
Output:
connect: (alatheia.zenfs.com, 80)
Traceback (most recent call last):
File test.py, line 29, in module
resp, content = h.request(url, PUT, body=body_content, headers =
params)
File /home/y/lib/python2.6/site-packages/httplib2/__init__.py, line
1436, in request
(response, content) = self._request(conn, authority, uri, request_uri,
method, body, headers, redirections, cachekey)
File /home/y/lib/python2.6/site-packages/httplib2/__init__.py, line
1188, in _request
(response, content) = self._conn_request(conn, request_uri, method,
body, headers)
File /home/y/lib/python2.6/site-packages/httplib2/__init__.py, line
1123, in _conn_request
conn.connect()
File /home/y/lib/python2.6/site-packages/httplib2/__init__.py, line 786,
in connect
self.sock.connect(sa)
File /home/y/lib/python2.6/site-packages/httplib2/socks.py, line 381, in
connect
self.__negotiatehttp(destpair[0], destpair[1])
File /home/y/lib/python2.6/site-packages/httplib2/socks.py, line 347, in
__negotiatehttp
raise HTTPError((statuscode, statusline[2]))
httplib2.socks.HTTPError: (403, 'Tunnel or SSL Forbidden')
The reason I can trace it is because,
I can use PUT on http://alatheia.zenfs.com/testing/shashwat but not on
http://alatheia.zenfs.com/ however host is resolved.
Again port 80 is used even when I use specific port (3128, in this case).
--
http://mail.python.org/mailman/listinfo/python-list
Re: PUT with proxy-support
On Thu, Aug 25, 2011 at 4:48 PM, Max Countryman m...@me.com wrote: Check out the python Requests module: http://docs.python-requests.org/en/latest/index.html Python request module is not documented very well IMHO. I tried to figure how to make PUT calls, how to add proxy, how to add certificates in headers. Did not managed to find all of it. Am not sure is supports REST calls with proxy support. Sent from my iPhone On Aug 25, 2011, at 7:07, Shashwat Anand anand.shash...@gmail.com wrote: I want to make a PUT request. I need some headers of my own ( certificates etc ) and I need to mandatorily use a proxy. Also the url is of the form http://www.xyz.com/abchttp://www.xyz.com/abcand I don't have permission to put data on http://www.xyz.comhttp://www.xyz.com while I do have permission to put data on http://www.xyz.com/abchttp://www.xyz.com/abc I tried httplib, httplib2, urllib2 with no avail. I managed to do this via command line curl: $ curl http:/ http://xyz.com/testing/shashwat/test.txt xyz.com/testing/shashwat/test.txt -T test.txt -H sw-version: 1.0 -H CA-Cert-Auth:v=1;a=yxyz.prod;h=10.10.0.1;t=1316594650;s=.AeEYJMMfElN74fnWD3GlXJ4J.1KiQFg-- --proxy proxy.xyz.com:3128 -H Content-Type:text/plain Is there a way to do it in python apart from using command line curl in python. The machine is RHEL4 and is giving hard time installing pycurl. -- http://mail.python.org/mailman/listinfo/python-list -- http://mail.python.org/mailman/listinfo/python-list
Re: PUT with proxy-support
I tried httplib, httplib2, urllib2 with no avail. I managed to do this via command line curl: $ curl http:/xyz.com/testing/shashwat/test.txt http://xyz.com/testing/shashwat/test.txt -T test.txt -H sw-version: 1.0 -H CA-Cert-Auth:v=1;a=yxyz.prod;h=10.10.0.1;t=1316594650;s=.AeEYJMMfElN74fnWD3GlXJ4J.1KiQFg-- --proxy proxy.xyz.com:3128 http://proxy.xyz.com:3128 -H Content-Type:text/plain If you can do it with command line curl then probably you can do it with pycurl. http://pycurl.sourceforge.net/ Best, Laszlo -- http://mail.python.org/mailman/listinfo/python-list
Re: PUT with proxy-support
On Thu, Aug 25, 2011 at 5:22 PM, Laszlo Nagy gand...@shopzeus.com wrote: ** I tried httplib, httplib2, urllib2 with no avail. I managed to do this via command line curl: $ curl http:/xyz.com/testing/shashwat/test.txt -T test.txt -H sw-version: 1.0 -H CA-Cert-Auth:v=1;a=yxyz.prod;h=10.10.0.1;t=1316594650;s=.AeEYJMMfElN74fnWD3GlXJ4J.1KiQFg-- --proxy proxy.xyz.com:3128 -H Content-Type:text/plain If you can do it with command line curl then probably you can do it with pycurl. http://pycurl.sourceforge.net/ Yeah. I tried that. The system is RHEL 4. So it gave me this error : src/pycurl.c:42:20: Python.h: No such file or directory src/pycurl.c:43:22: pythread.h: No such file or directory src/pycurl.c:58:4: #error Need Python version 2.2 or greater to compile pycurl. src/pycurl.c:61:4: #error Need libcurl version 7.19.0 or greater to compile pycurl. Apparently we need python-devel package. Following http://fedoraproject.org/wiki/EPEL/FAQ#howtouse I added EPEL software repository. sh-3.00$ yum list | grep -i python-dev sh-3.00$ sudo yum -y install python-dev Password: Setting up Install Process Setting up repositories epel [1/1] epel 100% |=| 3.8 kB00:00 Reading repository metadata in from local files b1f7bfef07466e9561644aba7 100% |=| 841 kB00:06 epel : ## 2583/2583 Added 2583 new packages, deleted 0 old in 4.51 seconds Parsing package install arguments No Match for argument: python-dev Nothing to do Turned out that python-curl is the required package which is already installed. Still no use. sh-3.00$ yum list | grep -i python-curl python-curl.x86_64 7.12.1-1.3.el4.rf installed sh-3.00$ python Python 2.6.4 (r264:75706, Nov 9 2009, 16:32:06) [GCC 3.4.6 20060404 (Red Hat 3.4.6-10)] on linux2 Type help, copyright, credits or license for more information. import pycurl Traceback (most recent call last): File stdin, line 1, in module ImportError: No module named pycurl Tried installing via easy_install sh-3.00$ sudo easy_install pycurl Searching for pycurl Reading http://pypi.python.org/simple/pycurl/ Reading http://pycurl.sourceforge.net/ Reading http://pycurl.sourceforge.net/download/ Best match: pycurl 7.19.0 Downloading http://pycurl.sourceforge.net/download/pycurl-7.19.0.tar.gz Processing pycurl-7.19.0.tar.gz Running pycurl-7.19.0/setup.py -q bdist_egg --dist-dir /tmp/easy_install-2ZCa8v/pycurl-7.19.0/egg-dist-tmp-DyHFls Using curl-config (libcurl 7.12.1) src/pycurl.c:42:20: Python.h: No such file or directory src/pycurl.c:43:22: pythread.h: No such file or directory src/pycurl.c:58:4: #error Need Python version 2.2 or greater to compile pycurl. src/pycurl.c:61:4: #error Need libcurl version 7.19.0 or greater to compile pycurl. [... Error Clipped] error: Setup script exited with error: command '/usr/bin/gcc' failed with exit status 1 Best, Laszlo -- http://mail.python.org/mailman/listinfo/python-list
Re: PUT with proxy-support
Check out the python Requests module: http://docs.python-requests.org/en/latest/index.html Sent from my iPhone On Aug 25, 2011, at 7:07, Shashwat Anand anand.shash...@gmail.com wrote: I want to make a PUT request. I need some headers of my own ( certificates etc ) and I need to mandatorily use a proxy. Also the url is of the form http://www.xyz.com/abc and I don't have permission to put data on http://www.xyz.com while I do have permission to put data on http://www.xyz.com/abc I tried httplib, httplib2, urllib2 with no avail. I managed to do this via command line curl: $ curl http:/xyz.com/testing/shashwat/test.txt -T test.txt -H sw-version: 1.0 -H CA-Cert-Auth:v=1;a=yxyz.prod;h=10.10.0.1;t=1316594650;s=.AeEYJMMfElN74fnWD3GlXJ4J.1KiQFg-- --proxy proxy.xyz.com:3128 -H Content-Type:text/plain Is there a way to do it in python apart from using command line curl in python. The machine is RHEL4 and is giving hard time installing pycurl. -- http://mail.python.org/mailman/listinfo/python-list -- http://mail.python.org/mailman/listinfo/python-list
