urllib.urlopen() with pages that requires cookies.

2006-04-27 Thread Øyvind Østlund 
I am trying to visit a limited amount of web pages that requires
cookies. I will get redirected if my application does not handle them.
I am using urllib.urlopen() to visit the pages right now. And I need a
push in the right direction to find out how to deal with pages that
requires cookies. Anyone have any idea how to go about this?



Thanks in advance.
- ØØ -
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: urllib.urlopen() with pages that requires cookies.

2006-04-27 Thread Fredrik Lundh 
Øyvind Østlund wrote:

> I am trying to visit a limited amount of web pages that requires cookies. I
> will get redirected if my application does not handle them. I am using
> urllib.urlopen() to visit the pages right now. And I need a push in the
> right direction to find out how to deal with pages that requires cookies.
> Anyone have any idea how to go about this?

use urllib2 and cookielib.  here's an outline:

import urllib2, cookielib

# set things up
jar = cookielib.CookieJar()
handler = urllib2.HTTPCookieProcessor(jar)
opener = urllib2.build_opener(handler)
urllib2.install_opener(opener)

data = urllib2.urlopen(someurl).read()

# ...

data = urllib2.urlopen(someotherurl).read()

# ...

# dump cookie jar contents (for debugging)
for cookie in jar:
print cookie





-- 
http://mail.python.org/mailman/listinfo/python-list

Re: urllib.urlopen() with pages that requires cookies.

2006-04-27 Thread mwt 

Fredrik Lundh wrote:
> Øyvind Østlund wrote:
>
> > I am trying to visit a limited amount of web pages that requires cookies. I
> > will get redirected if my application does not handle them. I am using
> > urllib.urlopen() to visit the pages right now. And I need a push in the
> > right direction to find out how to deal with pages that requires cookies.
> > Anyone have any idea how to go about this?
>
> use urllib2 and cookielib.  here's an outline:
>
> import urllib2, cookielib
>
> # set things up
> jar = cookielib.CookieJar()
> handler = urllib2.HTTPCookieProcessor(jar)
> opener = urllib2.build_opener(handler)
> urllib2.install_opener(opener)
>
> data = urllib2.urlopen(someurl).read()
>
> # ...
>
> data = urllib2.urlopen(someotherurl).read()
>
> # ...
>
> # dump cookie jar contents (for debugging)
> for cookie in jar:
> print cookie
>
> 

How would this outline be changed/expanded if it also needed to handle
basic authentication?

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: urllib.urlopen() with pages that requires cookies.

2006-04-29 Thread John J. Lee 
"mwt" <[EMAIL PROTECTED]> writes:

> Fredrik Lundh wrote:
[...]
> > use urllib2 and cookielib.  here's an outline:
> >
> > import urllib2, cookielib
> >
> > # set things up
> > jar = cookielib.CookieJar()
> > handler = urllib2.HTTPCookieProcessor(jar)
> > opener = urllib2.build_opener(handler)
> > urllib2.install_opener(opener)
> >
> > data = urllib2.urlopen(someurl).read()
> >
> > # ...
> >
> > data = urllib2.urlopen(someotherurl).read()
> >
> > # ...
> >
> > # dump cookie jar contents (for debugging)
> > for cookie in jar:
> > print cookie
> >
> > 
> 
> How would this outline be changed/expanded if it also needed to handle
> basic authentication?

...
handler = urllib2.HTTPCookieProcessor(jar)
pm = urllib2.HTTPPasswordMgrWithDefaultRealm()
# use None for realm: send username and password for all realms
pm.add_password(None, "http://example.com/";, "joe", "password")
auth_handler = HTTPBasicAuthHandler(pm)
opener = urllib2.build_opener(handler, auth_handler)
...


Note that (I just noticed that) using both basic auth and digest auth
handlers at the same time is broken, at least in SVN :-((


John

-- 
http://mail.python.org/mailman/listinfo/python-list