Re: [Tutor] python internet archive API?

[Luke Paireepinart]

Switanek, Nick wrote:
>
> I’m a novice Python programmer, and I’ve been looking for a way to 
> collect archived web pages. I would like to use the data on Internet 
> Archive, via the “Wayback Machine”. Look, for example, at 
> http://web.archive.org/web/*/http://www.python.org 
> . I’d like to crawl 
> down the first few levels of links of each of the updated archived 
> pages (the ones with *’s next to them). The site’s robots.txt 
> exclusions are complete, so a screen-scraping strategy doesn’t seem 
> doable.
>
What does the robots.txt have to do with anything?
Just ignore it.
If the robots.txt is telling you not to do something, you know that they 
don't want you to do it.
But if have a valid reason, just do it anyway.
>
> Does anyone have any suggestions for a way to go about this pythonically?
>
> Many thanks,
>
> Nick
>
> 
>
> ___
> Tutor maillist  -  Tutor@python.org
> http://mail.python.org/mailman/listinfo/tutor
>   

___
Tutor maillist  -  Tutor@python.org
http://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] sys.path, managing modules and packages

[Rikard Bosnjakovic]

On 4/26/07, Switanek, Nick <[EMAIL PROTECTED]> wrote:

> Can someone help me better understand how I ought to manage the modules and
> packages that I download? I often find that I can't use the code I've just
> downloaded, despite putting it into Lib/site-packages. Often I've added the
> subdirectory path via sys.path.append, but this seems to go only one level
> down, whereas I thought that when trying to import something python would
> search all subdirectories of the directories in the sys.path. I'd be glad
> for some basic instruction on the steps I need to take between downloading
> code from sourceforge and importing the functions in my scripts or at the
> command line.

You can use the environment-variable $PYTHONPATH for localized modules
you download. I have for example PYTHONPATH=~/hacks/python/modules,
and I put all downloaded modules in there.


-- 
- Rikard - http://bos.hack.org/cv/
___
Tutor maillist  -  Tutor@python.org
http://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] Averaging a list of lists with a listcomp?

[John Fouhy]

On 26/04/07, Terry Carroll <[EMAIL PROTECTED]> wrote:
> I have a list (of arbitrary length) of lists, each sublist having a fixed
> number N of items, all integers.
>
> I would like to produce a list of N items, each item of which is an
> integer which is the average of the elements in the same position of each
> of the sublists of the original list.

I don't think you're going to be able to do it with a single listcomp
based on the original list.  The reason I say this is because you need
to iterate over your _entire_ list (of arbitary length) in order to
compute _each_ element of the result list.

If you want a one-line solution, your best bet is to transform your
start list: instead of [[10, 20, 30], [50, 20, 90], [30, 20, 30]],
you'd like to have [[10,50,30], [20,20,20], [30,90,30]].  Then, it
would be easy.

Well, there's actually a simple way to do this transformation.  You
can use the zip function:

>>> zip([10, 20, 30], [50, 20, 90], [30, 20, 30])
[(10, 50, 30), (20, 20, 20), (30, 90, 30)]

And we can go a step further:

>>> orig = [[10, 20, 30], [50, 20, 90], [30, 20, 30]]
>>> zip(*orig)
[(10, 50, 30), (20, 20, 20), (30, 90, 30)]

So, here is your one-line solution:

>>> [sum(x)/len(x) for x in zip(*orig)]
[30, 20, 50]

HTH! :-)

-- 
John.
___
Tutor maillist  -  Tutor@python.org
http://mail.python.org/mailman/listinfo/tutor

[Tutor] Averaging a list of lists with a listcomp?

[Terry Carroll]

I have a list (of arbitrary length) of lists, each sublist having a fixed
number N of items, all integers.

I would like to produce a list of N items, each item of which is an 
integer which is the average of the elements in the same position of each 
of the sublists of the original list.

I realize the wording above is all hash, so here's the example:

Say the original list is: [[10, 20, 30], [50, 20, 90], [30, 20, 30]]

I want the new list to be: [30, 20, 50]

Because the average of the first elements 10, 50, 30 is 30; second 
elements 20, 20, 20 is 20; and third elements 30, 90, 30 is 50.

Since I'm taking a list and producing another list, I figured this would 
be a good candidate for a list comprehension; but I don't see a good 
approach. 

My attempt is below, but I can see the list comp approach is pretty yucky.  
As you can see, I've got a working solution, but in an effort to think 
more pythonically about list comprehensions, I'd like to see what a good 
list comprehension approach is.

(note: I know there's an integer division issue here; ignore that, I just 
picked easy numbers to keep the example clean.)

My approach:

###
orig = [[10, 20, 30], [50, 20, 90], [30, 20, 30]]
average = [30, 20, 50]

# iterative approach
L1 = [0, 0, 0]
for item in orig:
for i in range(0,3):
L1[i] += item[i]
L1 = [x/len(orig) for x in L1]
print L1

#list comp approach
L2 = [
sum([x[0] for x in orig])/len(orig),
sum([x[1] for x in orig])/len(orig),
sum([x[2] for x in orig])/len(orig)
]
#ew, yuck, hard-coded list indices!
print L2

assert L1 == L2 == average
###


___
Tutor maillist  -  Tutor@python.org
http://mail.python.org/mailman/listinfo/tutor

[Tutor] sys.path, managing modules and packages

[Switanek, Nick]

Can someone help me better understand how I ought to manage the modules
and packages that I download? I often find that I can't use the code
I've just downloaded, despite putting it into Lib/site-packages. Often
I've added the subdirectory path via sys.path.append, but this seems to
go only one level down, whereas I thought that when trying to import
something python would search all subdirectories of the directories in
the sys.path. I'd be glad for some basic instruction on the steps I need
to take between downloading code from sourceforge and importing the
functions in my scripts or at the command line.

 

Thanks,

Nick

 

___
Tutor maillist  -  Tutor@python.org
http://mail.python.org/mailman/listinfo/tutor

[Tutor] python internet archive API?

[Switanek, Nick]

I'm a novice Python programmer, and I've been looking for a way to
collect archived web pages. I would like to use the data on Internet
Archive, via the "Wayback Machine". Look, for example, at
http://web.archive.org/web/*/http://www.python.org
 . I'd like to crawl
down the first few levels of links of each of the updated archived pages
(the ones with *'s next to them). The site's robots.txt exclusions are
complete, so a screen-scraping strategy doesn't seem doable. 

 

Does anyone have any suggestions for a way to go about this
pythonically? 

 

Many thanks,

Nick

 

___
Tutor maillist  -  Tutor@python.org
http://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] best search/replace method?

[John Washakie]

Excellent. Thanks Luke, that seems to be working

On 4/25/07, Luke Paireepinart <[EMAIL PROTECTED]> wrote:
> John Washakie wrote:
> > Tested. Failed...
> >
> > I thought it would be something like that, but that doesn't work..
> > perhaps because 'ImagePathReplaceMe' isn't separate from the rest of
> > the path???
> It doesn't matter what surrounds the string to be replaced in the target
> string.
> Refer to my other e-mail on why Rikard's didn't work.
> -Luke
>
___
Tutor maillist  -  Tutor@python.org
http://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] best search/replace method?

[Luke Paireepinart]

John Washakie wrote:
> Tested. Failed...
>
> I thought it would be something like that, but that doesn't work..
> perhaps because 'ImagePathReplaceMe' isn't separate from the rest of
> the path???
It doesn't matter what surrounds the string to be replaced in the target 
string.
Refer to my other e-mail on why Rikard's didn't work.
-Luke
___
Tutor maillist  -  Tutor@python.org
http://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] best search/replace method?

[Luke Paireepinart]

Rikard Bosnjakovic wrote:
> On 4/25/07, John Washakie <[EMAIL PROTECTED]> wrote:
>
>   
>> cat raw.html |
>> sed 's/ImagePathReplaceMe/NewPathToImage/g' |
>> sed 's/TitleReplaceMe/NewTitle/g' > new.html
>> 
>
> One line's sufficient:
>
> sed -e 's/ImagePathReplaceMe/NewPathToImage/g;s/TitleReplaceMe/NewTitle/g'
> raw.html > new.html
>
>   
>> However, this is going to be part of an Plone product so I want to use
>> Python. What is the best method to accomplish this?
>> 
>
> d = open("raw.html").read()
> d.replace("ImagePathReplaceMe", "NewPathToImage")
> d.replace("TitleReplaceMe", "TitleReplaceMe")
>   
replace isn't a modifying function, so you'd have to do something like this:
d = open("raw.html").read()
x = open("new.html","w")
x.write(d.replace("ImagePathReplaceMe", 
"NewPathToImage").replace("TitleReplaceMe", "TitleReplaceMe"))
x.close()

or just store the strings after replacement in a temporary variable.
> x = open("new.html", "w")
> x.write(d)
> x.close()
>
> Untested.
>
>
>   

___
Tutor maillist  -  Tutor@python.org
http://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] best search/replace method?

[John Washakie]

Tested. Failed...

I thought it would be something like that, but that doesn't work..
perhaps because 'ImagePathReplaceMe' isn't separate from the rest of
the path??? Maybe replace has some options... I'll look into that.

.

On 4/25/07, Rikard Bosnjakovic <[EMAIL PROTECTED]> wrote:
> On 4/25/07, John Washakie <[EMAIL PROTECTED]> wrote:
>
> > cat raw.html |
> > sed 's/ImagePathReplaceMe/NewPathToImage/g' |
> > sed 's/TitleReplaceMe/NewTitle/g' > new.html
>
> One line's sufficient:
>
> sed -e 's/ImagePathReplaceMe/NewPathToImage/g;s/TitleReplaceMe/NewTitle/g'
> raw.html > new.html
>
> > However, this is going to be part of an Plone product so I want to use
> > Python. What is the best method to accomplish this?
>
> d = open("raw.html").read()
> d.replace("ImagePathReplaceMe", "NewPathToImage")
> d.replace("TitleReplaceMe", "TitleReplaceMe")
> x = open("new.html", "w")
> x.write(d)
> x.close()
>
> Untested.
>
>
> --
> - Rikard - http://bos.hack.org/cv/
>
___
Tutor maillist  -  Tutor@python.org
http://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] best search/replace method?

[Rikard Bosnjakovic]

On 4/25/07, John Washakie <[EMAIL PROTECTED]> wrote:

> cat raw.html |
> sed 's/ImagePathReplaceMe/NewPathToImage/g' |
> sed 's/TitleReplaceMe/NewTitle/g' > new.html

One line's sufficient:

sed -e 's/ImagePathReplaceMe/NewPathToImage/g;s/TitleReplaceMe/NewTitle/g'
raw.html > new.html

> However, this is going to be part of an Plone product so I want to use
> Python. What is the best method to accomplish this?

d = open("raw.html").read()
d.replace("ImagePathReplaceMe", "NewPathToImage")
d.replace("TitleReplaceMe", "TitleReplaceMe")
x = open("new.html", "w")
x.write(d)
x.close()

Untested.


-- 
- Rikard - http://bos.hack.org/cv/
___
Tutor maillist  -  Tutor@python.org
http://mail.python.org/mailman/listinfo/tutor

[Tutor] best search/replace method?

[John Washakie]

Folks,

I'm writing a program which will read in an html file, and then
replace certain elements, such as the title, and various paths defined
for images. I can make a 'source' file with ImagePathReplaceMe and
TitleReplaceMe text in it, then search for that and replace it. With
sed my script would be three lines:

cat raw.html |
sed 's/ImagePathReplaceMe/NewPathToImage/g' |
sed 's/TitleReplaceMe/NewTitle/g' > new.html

However, this is going to be part of an Plone product so I want to use
Python. What is the best method to accomplish this?

Thanks,
john
___
Tutor maillist  -  Tutor@python.org
http://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] 100 Programs?

[Rohan Deshpande]

Well there's always pythonchallenge.com, but it's both a riddle and a
programming exercise.  And, they're not entirely python specific (except for
a couple), but it's a start.

-Rohan

On 4/25/07, Alex Dering <[EMAIL PROTECTED]> wrote:


Can anyone recommend a list of progressively harder Python programming
assignments for the self-learner?

My suspicion is that if I arrive at a point where I am proficient in
Python, I still won't trust myself to be the judge of that. So I'm trying to
get a list of specific tasks/problems/projects (with expected time
requirements) that would allow me to say "Well, I did that in 2 hours 20
minutes and the expected time to complete it was 2 hours 30 minutes, so I'm
at least *that* skilled."

Example:
Beginner Level
Each of these should take someone who knows what he or she is doing one
session of less than an hour to complete. These should try to combine
several simple tasks, but not to the point of hopelessly confusing the
programmer. As an example, when I took calculus, the teacher would delight
in saying, "Now, let's take the derivative of a simple function, say, x
cubed raised to the ln of e raised to the negative x times sin x squared." I
don't need to be shown I don't know everything; I already know that.

Intermediate Level
These should take a proficient intermediate between 1 hour and 3 hours
each.

Advanced Level
These should be expected to require at least 3 hours and possibly up to a
week. These should be programs that a single programmer is capable of doing.
And -- yes, and -- these should be portfolio-worthy. Items that, when you
start looking for a job or a freelance assignment, you can point to and
actually get some sort of respect for.

Many thanks for all help.

___
Tutor maillist  -  Tutor@python.org
http://mail.python.org/mailman/listinfo/tutor


___
Tutor maillist  -  Tutor@python.org
http://mail.python.org/mailman/listinfo/tutor

[Tutor] 100 Programs?

[Alex Dering]

Can anyone recommend a list of progressively harder Python programming
assignments for the self-learner?

My suspicion is that if I arrive at a point where I am proficient in Python,
I still won't trust myself to be the judge of that. So I'm trying to get a
list of specific tasks/problems/projects (with expected time requirements)
that would allow me to say "Well, I did that in 2 hours 20 minutes and the
expected time to complete it was 2 hours 30 minutes, so I'm at least
*that*skilled."

Example:
Beginner Level
Each of these should take someone who knows what he or she is doing one
session of less than an hour to complete. These should try to combine
several simple tasks, but not to the point of hopelessly confusing the
programmer. As an example, when I took calculus, the teacher would delight
in saying, "Now, let's take the derivative of a simple function, say, x
cubed raised to the ln of e raised to the negative x times sin x squared." I
don't need to be shown I don't know everything; I already know that.

Intermediate Level
These should take a proficient intermediate between 1 hour and 3 hours each.

Advanced Level
These should be expected to require at least 3 hours and possibly up to a
week. These should be programs that a single programmer is capable of doing.
And -- yes, and -- these should be portfolio-worthy. Items that, when you
start looking for a job or a freelance assignment, you can point to and
actually get some sort of respect for.

Many thanks for all help.
___
Tutor maillist  -  Tutor@python.org
http://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] Custom objects throw what exception?

[Kent Johnson]

Thanos Panousis wrote:
> The tutor list is definately helping me perform my OO-python babysteps.
> 
> I want to ask this. Say I have some objects of my own, and using the
> example I posted in a previous post, say I have a person object with a
> instance variable called hairColor.
> 
> The hairColor property is set via an exotic function that could go
> wrong and produce an exception.
> 
> How should I hanlde this? Should I catch the exception in the
> person.__init__(self,color) "construtor"? and if I do so, what happens
> the code that is waiting for a person object to arrive with a call
> like p = person(). What exception should the person class throw, if
> any?

Don't hide the exception. If you can intelligently *handle* the 
exception and create a person object that is initialized in a reasonable 
way, do so. Otherwise let some kind of exception propagate back to the 
caller so they know there is a problem.

If you catch the exception in person.__init__() then the code that calls 
person() will get whatever person object you create. The caller will not 
know that there was a problem with the color.

A ValueError might be appropriate. You can also define your own 
exception for example
   class HairColorError(Exception): pass

then in your code you can
   raise HairColorError('%s is not a valid hair color' % color)

Kent
___
Tutor maillist  -  Tutor@python.org
http://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] how to stop a function

[Dmitry Dzhus]

Maybe `return`?

-- 
Happy Hacking.

Dmitry "Sphinx" Dzhus
http://sphinx.net.ru
___
Tutor maillist  -  Tutor@python.org
http://mail.python.org/mailman/listinfo/tutor

[Tutor] Custom objects throw what exception?

[Thanos Panousis]

The tutor list is definately helping me perform my OO-python babysteps.

I want to ask this. Say I have some objects of my own, and using the
example I posted in a previous post, say I have a person object with a
instance variable called hairColor.

The hairColor property is set via an exotic function that could go
wrong and produce an exception.

How should I hanlde this? Should I catch the exception in the
person.__init__(self,color) "construtor"? and if I do so, what happens
the code that is waiting for a person object to arrive with a call
like p = person(). What exception should the person class throw, if
any?
___
Tutor maillist  -  Tutor@python.org
http://mail.python.org/mailman/listinfo/tutor