Re: [Tutor] MemoryError !!! Help Required

[Andreas Kostyrka]

Am Montag, den 07.04.2008, 00:32 -0500 schrieb Luke Paireepinart:
> devj wrote:
> > Hi,
> > I am making a web crawler using Python.To avoid dupliacy of urls,i have to
> > maintain lists of downloaded urls and to-be-downloaded urls ,of which the
> > latter grows exponentially,resulting in a MemoryError exception .What are
> > the possible ways to avoid this ??
> >   
> get more RAM, store the list on your hard drive, etc. etc. 
> Why are you trying to do this?  Are you sure you can't use existing 
> tools for this such as wget?
> -Luke

Also traditional solutions involve e.g. remembering a hash value.

Plus if you go for a simple file based solution, you probably should
store it by hostname, e.g.:
http://123.45.67.87/abc/def/text.html => file("127/45/67/87",
"w").write("/abc/def/text.html")
(guess you need to run os.makedirs as needed :-P)

These makes it scaleable (by not storying to many files in one
directory, and by leaving out the common element so the files are
smaller and faster to read), while keeping the code relative simple.

Another solution would be shelve, but you have to keep in mind that if
you are unlucky you might loose the database. (Some of the DBs that
anydbm might not survive power loss, or other problems to well)

Andreas

> ___
> Tutor maillist  -  Tutor@python.org
> http://mail.python.org/mailman/listinfo/tutor


signature.asc
Description: Dies ist ein digital signierter Nachrichtenteil
___
Tutor maillist  -  Tutor@python.org
http://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] MemoryError !!! Help Required

[Luke Paireepinart]

devj wrote:
> Hi,
> I am making a web crawler using Python.To avoid dupliacy of urls,i have to
> maintain lists of downloaded urls and to-be-downloaded urls ,of which the
> latter grows exponentially,resulting in a MemoryError exception .What are
> the possible ways to avoid this ??
>   
get more RAM, store the list on your hard drive, etc. etc. 
Why are you trying to do this?  Are you sure you can't use existing 
tools for this such as wget?
-Luke
___
Tutor maillist  -  Tutor@python.org
http://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] Creating Sudoku

[Luke Paireepinart]

Tom Haynes wrote:
>
> G’day,
>
>  
>
> I am trying to create a simple Sudoku game that takes a simple 
> raw_input of a, r, c or something similar (where a = row, r = column, 
> c = number to place there) and place it in a Sudoku game square. I 
> have tried many times to start it from different angles and just can’t 
> do it. If someone could help step me through what needs to be done 
> that would be greatly appreciated!
>
Sounds like you need to have a 2-dimensional list to store your values.  
The first and second indexes of the list would correspond to the 
row/column variables you have, and the value would be the actual value 
at each location.  You may want to initialize your list to some value 
(fill it with something).  You could also use a dictionary, but it would 
probably not be worthwhile.
You would also probably want to use the int(some_value) conversion on 
your raw_inputs so you can use the values to index your list.
Let us know if this is not enough of a hint!  Hope it goes okay.
-Luke
___
Tutor maillist  -  Tutor@python.org
http://mail.python.org/mailman/listinfo/tutor

[Tutor] Creating Sudoku

[Tom Haynes]

G’day, 

 

I am trying to create a simple Sudoku game that takes a simple raw_input of
a, r, c or something similar (where a = row, r = column, c = number to place
there) and place it in a Sudoku game square. I have tried many times to
start it from different angles and just can’t do it. If someone could help
step me through what needs to be done that would be greatly appreciated!

 

Thanks,

Tom 


No virus found in this outgoing message.
Checked by AVG. 
Version: 7.5.519 / Virus Database: 269.22.8/1362 - Release Date: 6/04/2008
11:12 AM
 
___
Tutor maillist  -  Tutor@python.org
http://mail.python.org/mailman/listinfo/tutor

[Tutor] Question about global variables on modules

[Tiago Katcipis]

I know its not such a pretty thing to have global variables but its only
for an exercise my teacher told to do. Its a function to calculate the
results of a matrix using jacob. I want to inside the module (inside a
function on the module )assign a value to a global variable, but the
only way i found to do this inside the own module function is importing
the module inside himself. Is there another way of doing this? its kind
odd to import the module to himself, i think :-)

here goes the code

<>

import lineares_jacob

*ERRO_FINAL = 0.0*

def obter_respostas(matriz, incognitas, erro_max):
  erro = erro_max * 10
  n = len(matriz)
 
  while(erro >= erro_max):
novas_incognitas = []
y_um = (2.0 + (1.0 * incognitas[1]) - (1.0 * incognitas[4999])) / 3.0
novas_incognitas.append(y_um)
   
for i in range(1 , (n - 1)):
  yi = ( (2.0 * i) + incognitas[i - 1] + incognitas[i + 1] ) / (2.0 + i)
  novas_incognitas.append(yi)
 
y_cinc_mil = (1.0 - incognitas[0] + incognitas[4998]) / 5002.0
novas_incognitas.append(y_cinc_mil)
   
maior = novas_incognitas[0] - incognitas[0]
   
for i in range(1, 5000):
  dif = novas_incognitas[i] - incognitas[i]
  if(dif > maior):
maior = dif
   
erro = maior
incognitas = novas_incognitas
 
  *lineares_jacob.ERRO_FINAL = erro*
  return incognitas
___
Tutor maillist  -  Tutor@python.org
http://mail.python.org/mailman/listinfo/tutor

[Tutor] MemoryError !!! Help Required

[devj]

Hi,
I am making a web crawler using Python.To avoid dupliacy of urls,i have to
maintain lists of downloaded urls and to-be-downloaded urls ,of which the
latter grows exponentially,resulting in a MemoryError exception .What are
the possible ways to avoid this ??
-- 
View this message in context: 
http://www.nabble.com/MemoryError-%21%21%21-Help-Required-tp16510068p16510068.html
Sent from the Python - tutor mailing list archive at Nabble.com.

___
Tutor maillist  -  Tutor@python.org
http://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] socket / over network

[Kim Hawtin]

Hi Nathan,

Nathan McBride wrote:
> Alan Gauld wrote:
>> "Nathan McBride" <[EMAIL PROTECTED]> wrote
>>> I'm pretty tired of the lame backup solution we have at work.
>>> Could anyone point me to a (more or less newbieish) example of how
>>> to
>>> have python open a socket on one box and get data from it, then have
>>> another
>>> box write to it over the network?
>> For a very simple example of using a socket you could try the
>> Network Programming topic in my tutorial.
> 
>> There is also a HowTo or Topic guide on the Python web site
>> that gives a more detailed example.
> 
>> That having been said, backups are usually best done using
>> OS tools or if you must roll your own then using ftp or similar
>> as a file transfer mechanism rather than trying to send a
>> bytestream over a socket. ftp can handle broken connections
>> etc more easily. Detecting and fixing errors over a socket
>> stream is non trivial and for backups is pretty much essential!!
> 
> Going off of wha tyou said, if I choose to use ftp, is there a way i
> could do everything from within python including the server to get the
> files?  Is there like a ftp module for python to help in the passing of
> the files between the computers?

There are number of problems with FTP around security and firewalls, etc.

This might be overkill, but perhaps you could use Twisted with SSH/SCP to get
files around?

See; [Twisted-Python] Twisted SCP
  http://twistedmatrix.com/pipermail/twisted-python/2005-December/012180.html

Perhaps using Rsync and SSH might be more appropriate;
 http://www.howtoforge.com/rsync_incremental_snapshot_backups

regards,

Kim
-- 
Operating Systems, Services and Operations
Information Technology Services, The University of Adelaide
[EMAIL PROTECTED]
___
Tutor maillist  -  Tutor@python.org
http://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] socket / over network

[Nathan McBride]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Alan Gauld wrote:
> "Nathan McBride" <[EMAIL PROTECTED]> wrote
>
> Hi Nathan,
>
> Please don't reply to an existing message to start a new discussion.
> It messes up those of us using threaded mail/news readers and
> increases the likelihood that your message will be missed.
>
>> I'm pretty tired of the lame backup solution we have at work.
>> Could anyone point me to a (more or less newbieish) example of how
>> to
>> have python open a socket on one box and get data from it, then have
>> another
>> box write to it over the network?
>
> For a very simple example of using a socket you could try the
> Network Programming topic in my tutorial.
>
> There is also a HowTo or Topic guide on the Python web site
> that gives a more detailed example.
>
> That having been said, backups are usually best done using
> OS tools or if you must roll your own then using ftp or similar
> as a file transfer mechanism rather than trying to send a
> bytestream over a socket. ftp can handle broken connections
> etc more easily. Detecting and fixing errors over a socket
> stream is non trivial and for backups is pretty much essential!!
>
Going off of wha tyou said, if I choose to use ftp, is there a way i
could do everything from within python including the server to get the
files?  Is there like a ftp module for python to help in the passing of
the files between the computers?

Thanks,

Nate
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.7 (GNU/Linux)
Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org

iD8DBQFH+YH1/n+duykW6K8RAhv+AJoCDvQip6Q1wJSh3dEoRZoC4Gx3oACdF0DK
oQXQTccEnkEz0mf/Qo4Ywqo=
=QRMr
-END PGP SIGNATURE-

___
Tutor maillist  -  Tutor@python.org
http://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] PyMOTW: difflib

[Dick Moores]

At 04:39 PM 4/6/2008, Alan Gauld wrote:
"Dick Moores"
<[EMAIL PROTECTED]> wrote
>> > Copied the data into my demo_pymotw-difflib.py,
>> > deleted the line "from difflib_data import
*",
>> > and ran it
>>
>>I'm curious. Why did you do that?
> OK, I put it back in, changing the filename to 
> demo_pymotw-difflib_error.py.
> See
<
http://py77.python.pastebin.com/f5c3a37ce> for the code and
the
> ImportError: No module named difflib_data.
You still have the copied data which will overwrite the values
imported however to import it you need to have the path set
up correctly. Is the difflib_data.py in the same folder as
difflib.py?
No. The paths are E:\PythonWork\PyMOTW-1.48\difflib\difflib_data.py
and E:\Python25\Lib\difflib.py
Even if there weren't a path problem, my understanding is
that   from difflib_data import *   would only import
the functions in difflib_data.py; There are none.  See
difflib_data.py at
<
http://py77.python.pastebin.com/f5e2c73a5>. Am I wrong?
Ah ha! I AM wrong, aren't I. I just realized that Hellman's intention was
to run difflib_differ.py
(<
http://py77.python.pastebin.com/f7853ef8>), which IS in the same
folder, difflib, as difflib_data.py.  Works fine.  I didn't
need to make my own script. 
I'm still puzzled, though. How can   from difflib_data import
*   work?  
Let's see. Say I have a script, some_variables.py
===some_variables.py=
#!/usr/bin/env python
a = 1
b = 2
c = 3
=end of some_variables.py
And a script, importer.py
=importer.py=
#!/usr/bin/env python
from some_variables.py import *
print a, b, c, a*b*c
end of importer.py===
And by golly, I do get
> "E:\Python25\pythonw.exe" -u
"E:\PythonWork\importer.py" 
1 2 3 6
Sorry, folks, but I'm truly amazed. How come I didn't know this
before??
Thanks, Alan, you've opened my eyes.
Dick

  

UliPad <>:

http://code.google.com/p/ulipad/


___
Tutor maillist  -  Tutor@python.org
http://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] PyMOTW: difflib

[Alan Gauld]

"Dick Moores" <[EMAIL PROTECTED]> wrote
>> > Copied the data into my demo_pymotw-difflib.py,
>> > deleted the line "from difflib_data import *",
>> > and ran it
>>
>>I'm curious. Why did you do that?

> OK, I put it back in, changing the filename to 
> demo_pymotw-difflib_error.py.
> See  for the code and the
> ImportError: No module named difflib_data.

You still have the copied data which will overwrite the values
imported however to import it you need to have the path set
up correctly. Is the difflib_data.py in the same folder as difflib.py?


Your code should only need to be:

a.. import difflib
a.. from difflib_data import *
a..
a.. d = difflib.Differ()
a.. diff = d.compare(text1_lines, text2_lines)
a.. print '\n'.join(list(diff))

And provided both modules are visible to Python it should work.
Copying the data works too but I was just curious why you
needed to...

Alan G.


___
Tutor maillist  -  Tutor@python.org
http://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] PyMOTW: difflib

[Dick Moores]

At 02:22 PM 4/6/2008, Alan Gauld wrote:

>"Dick Moores" <[EMAIL PROTECTED]> wrote
>
> > I downloaded that compressed file and found difflib_data.py (see it
> > at ). Copied the data
> > into
> > my demo_pymotw-difflib.py, deleted the line "from difflib_data
> > import
> > *", and ran it
>
>I'm curious. Why did you do that?
>Why not just leave the import as it was?

OK, I put it back in, changing the filename to demo_pymotw-difflib_error.py.
See  for the code and the 
ImportError: No module named difflib_data.

Or is there something I don't understand here?

Dick



UliPad <>: http://code.google.com/p/ulipad/ 

___
Tutor maillist  -  Tutor@python.org
http://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] PyMOTW: difflib

[Alan Gauld]

"Dick Moores" <[EMAIL PROTECTED]> wrote

> I downloaded that compressed file and found difflib_data.py (see it
> at ). Copied the data 
> into
> my demo_pymotw-difflib.py, deleted the line "from difflib_data 
> import
> *", and ran it

I'm curious. Why did you do that?
Why not just leave the import as it was?

Alan G. 


___
Tutor maillist  -  Tutor@python.org
http://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] PyMOTW: difflib

[Dick Moores]

At 08:24 AM 4/6/2008, Martin Walsh wrote:
>Dick Moores wrote:
> > See < http://blog.doughellmann.com/2007/10/pymotw-difflib.html>
> >
> > And my try with the Differ example,
> > , which also shows the error,
> >
> > "E:\Python25\pythonw.exe" -u "E:\PythonWork\demo_pymotw-difflib.py"
> > Traceback (most recent call last):
> >  File "E:\PythonWork\demo_pymotw-difflib.py", line 12, in 
> >from difflib_data import *
> > ImportError: No module named difflib_data
> >
> > What is difflib_data ?
>
>It is the example data provided with the PyMOTW tutorial.
>
>Near the top of the article (from the link you provided) you'll see the
>heading "Test Data". I assume the author wants you to copy and paste the
>source into a new file named difflib_data.py in your working dir.
>Alternatively, it looks like you can download all the source and example
>data for all PyMOTWs in one compressed file:
>http://www.doughellmann.com/projects/PyMOTW/

Thanks very much for figuring that out for me, and so clearly explaining it.

I downloaded that compressed file and found difflib_data.py (see it 
at ). Copied the data into 
my demo_pymotw-difflib.py, deleted the line "from difflib_data import 
*", and ran it. See it and the output at 


Dick Moores



UliPad <>: http://code.google.com/p/ulipad/ 

___
Tutor maillist  -  Tutor@python.org
http://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] PyMOTW: difflib

[Martin Walsh]

Dick Moores wrote:
> See < http://blog.doughellmann.com/2007/10/pymotw-difflib.html>
> 
> And my try with the Differ example, <
> http://py77.python.pastebin.com/f41ec1ae8>, which also shows the error,
> 
> "E:\Python25\pythonw.exe" -u "E:\PythonWork\demo_pymotw-difflib.py"
> Traceback (most recent call last):
>  File "E:\PythonWork\demo_pymotw-difflib.py", line 12, in 
>from difflib_data import *
> ImportError: No module named difflib_data
> 
> What is difflib_data ?

It is the example data provided with the PyMOTW tutorial.

Near the top of the article (from the link you provided) you'll see the
heading "Test Data". I assume the author wants you to copy and paste the
source into a new file named difflib_data.py in your working dir.
Alternatively, it looks like you can download all the source and example
data for all PyMOTWs in one compressed file:
http://www.doughellmann.com/projects/PyMOTW/

PyMOTW has a cheese shop entry also (http://pypi.python.org/), so one
would assume you could get the source with easy_install as well, but
I've never tried it.

HTH,
Marty


> 
> Thanks,
> 
> Dick Moores

___
Tutor maillist  -  Tutor@python.org
http://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] Diff for Python

[Dick Moores]

I like ExamDiff, and find the free version sufficient for my
purposes.
<
http://www.prestosoft.com/edp_examdiff.asp>
"ExamDiff is a freeware Windows
95/98/Me/NT/2000/XP/2003/Vista tool for visual file comparison. It has a
number of simple and convenient features that many users have been asking
for a long time from a file comparison tool."
from
<
http://www.prestosoft.com/edp_edfeatures.asp>:
==
ExamDiff offers the following features: 
There is no need to specify both filenames -- just enter a directory name
for one of the files to be compared. ExamDiff will try to use an entered
filename with a specified directory name. 
Remembers a user specified number of last compared first and second
files. This allows quick selection of the two files the user wishes to
compare. 
Automatically detects file changes and prompts the user to re-compare
files. 
One push re-compare function which attempts to leave the viewer's focus
in the same place as before the re-compare. 
Drag and drop support for dropping one or two files into the program's
window (e.g. from Windows Explorer). 
Easy editing of the first and second files. ExamDiff will spawn any
editor (configurable by the user) with the first or second file, and line
number option (available for editors that support this, through ExamDiff
variables: $FILE (first or second file name), $CURFIFF (number of line at
which the current difference starts), and $CARET (number of line where
the caret is located). 
Saves the file differences in a standard UNIX DIFF file. 
Easy navigation through the differences via "Previous
Difference"/ "Current difference"/ "Next
Difference" buttons and hot keys or via a drop-down list box of all
the differences 
Allows the copying of text from the comparison panes via drag and drop, a
hot key, or a right button pop up. 
Simple "Search" command to search for strings in the comparison
panes. It also remembers a user specified number of most recent searches.

Customizable text and background colors, font, tab size, and "Show
Differences Only" option as well as options to "Ignore white
spaces in lines", "Ignore changes in amount of white spaces in
lines", "Ignore case",  "Treat files as text
files", "Ignore leading white space in lines" and
"Ignore trailing white space in lines" for comparison.

Fully customizable file extension filter. For example, the user can
choose only .c and .cpp files or any other files  he/she chooses to
be displayed. 
Tooltips which include file properties (when the mouse cursor is placed
over the pane title bars), difference number (when the cursor is over the
yellow triangle marking the current difference), and etc. 
Adjustable pane splitter with smooth synchronized scrolling. Allows easy
toggling between horizontal and vertical splitter orientation as well as
splitter centering and panes hiding. 
Command line options include: Usage: ExamDiff
[Filename1] [Filename2] [Options]
  Filename1, Filename2 are names of files to
be compared
  Options are any
of:  
/i ignore case 
    /w ignore all white space in
lines 
    /b ignore changes in amount of
white space in lines 
    /l ignore leading white space
in lines 
    /e ignore trailing white space
in lines 
    /t treat both files as text
files 
    /d show differences only 
    /n don't show initial
"Compare Files" dialog 
    /aN scroll trough all
differences with N second delay and exit after the last difference 
    /?,/h print this screen

 NOTE: if options /i, /w, /b, /l, /e, /t, or /d
are not set, the last used options remain in effect.

from
<
http://www.prestosoft.com/edp_examdiff.asp#3>
What ExamDiff Cannot Do (But ExamDiff Pro Can) 
 
Compare directories and binary files. 
Highlight file differences down to the level of words or characters in
changed lines. 
Ignore lines and part of lines matching regular expressions. 
Print (and print preview) the diff report. 
Word wrap long lines. 
Edit files inside comparison panes. 
Use named comparison sessions. 
Support Unicode. 
===
The price of ExamDiff Pro is $35.
Dick Moores



___
Tutor maillist  -  Tutor@python.org
http://mail.python.org/mailman/listinfo/tutor

[Tutor] Fw: Diff for Python

[ALAN GAULD]

forwarding to tutor.
Mark, please use Reply-All for tutor messages...

- Forwarded Message 
From: Marc Tompkins <[EMAIL PROTECTED]>
To: Alan Gauld <[EMAIL PROTECTED]>
Sent: Saturday, 5 April, 2008 11:45:47 PM
Subject: Re: [Tutor] Diff for Python

On Sat, Apr 5, 2008 at 3:04 PM, Alan Gauld <[EMAIL PROTECTED]> wrote:
On Windows you can use FC - File Compare.
Its not as powerful as diff but it will highlight differences.



On Windows, I've tried a bunch of diff tools - it's probably the way my brain 
is wired, but I generally find it harder to understand what the diff tool is 
telling me than it would have been to print out the damn files and compare them 
on paper.  I feel like I'm being given clues so I can work out the puzzle 
myself... if I wanted that, I would do a crossword, not use a software tool.

So my tool of choice (since I discovered it about three months ago) is the 
Compare plugin in Notepad++.  It simply displays the files in separate child 
windows, forcibly aligns them with "soft" newlines, and synchronizes the 
windows' scrollbars to keep them lined up side by side.  It also shades the 
lines in different colors depending on whether the lines are the same in both 
files, or one file has a line that the other doesn't, or both files have the 
line but different versions.  None of this is new, of course, but I've never 
used a tool before that got it all so _right_ and made it so simple to use and 
to read.  (Open two or more files in the editor, hit Alt-D, read.  If 
necessary, cut and paste between the windows - hit Alt-D again to resync - 
read.)

I'm sure there are more sophisticated choices.  Honestly, I sometimes feel a 
little guilty using it, 'cause I think I ought to be working harder...  I'm 
sure that both vi and emacs do this in a way that mere mortals such as I cannot 
appreciate, but I think you must have had to start using either vi or emacs at 
a very early age to be able to enjoy the experience.  I'm putting on my 
flame-retardant Nomex suit as I type this.

(Tying this thread in with one from last week...)
As a general-purpose Windows editor, I definitely recommend Notepad++.  (It's 
free, but I moved to it from TextPad, in which I had invested $50.  If you knew 
me, you'd know what high praise this is for Notepad++.)  For Python / wxPython 
development, though, I love me some SPE.

-- 
www.fsrtechnologies.com


___
Tutor maillist  -  Tutor@python.org
http://mail.python.org/mailman/listinfo/tutor