[Tutor] Help : File formatting

[rahool shelke]

Hi,

I am beginner to python and doing coding on UNIX
platforms(AIX,Linux,HP,Solaris)

I want to take the list of files present in packages. On AIX we get the
files list present in packges using command "lslpp -f ".
Now i want to format output of above command such that only list of files
with absolute paths to be taken out and put into separate list/dictionary.
I am considering here the remote connectivity of hosts.

Will you please guide me to get the solution for this ?

-Rahul Shelke
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] pickle problems

[Dave Angel]

On 08/22/2012 07:32 PM, Richard D. Moores wrote:
> 
>
> My code uses gmpy2.is_prime()  (lines 79 and 89). is_prime() is VERY fast.

You do know that this gmpy2 function is only statistically correct ?  it
can false positive.  I don't know what the probs are, but it uses
Miller-Rabin, with a default factor of 25.




-- 

DaveA

___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] subprocess.Popen help...thanks

[Ray Jones]

Thanks to all who responded. I'm deeply into some of the links provided,
and my understanding has greatly increased.


Ray

On 08/22/2012 12:59 AM, Andreas Perstinger wrote:
> On 22.08.2012 03:39, Ray Jones wrote:
>> Does anyone know of a link to a really good tutorial that would help me
>> with subprocess.Popen? a tutorial that uses really small words and more
>> examples than explanation? After 15 years of scripting, I'm ashamed to
>> say that I'm still not all that familiar with input, output, pipes, etc.
>> much beyond a simple 'ls | ws -l' or  &2>/dev/null scenarios. The
>> docs for Popen have left me completely boggled, and I'm not seeing much
>> available on Google search. Any suggestions?
>
> What about this tutorial:
> http://jimmyg.org/blog/2009/working-with-python-subprocess.html
>
> or Doug Hellmann's PyMOTW page about subprocess:
> http://www.doughellmann.com/PyMOTW/subprocess/index.html
>
> Bye, Andreas
> ___
> Tutor maillist  -  Tutor@python.org
> To unsubscribe or change subscription options:
> http://mail.python.org/mailman/listinfo/tutor
>

___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] pickle problems

[Richard D. Moores]

On Wed, Aug 22, 2012 at 11:54 AM, Steven D'Aprano  wrote:
> On 23/08/12 02:17, Richard D. Moores wrote:
>>
>> I've incorporated many of the suggestions I've received here, plus some 
>> important tips from Case Van Horsen, the gmpy2 maintainer, and I believe one 
>> of its developers.
>>
>> Here's a function, factor_integer(), for quickly factoring any integer
>> up to 1e17:.
>
>
> That relies on a pre-existing cache of prime numbers. If you don't use
> those cached prime numbers, do you know how long your code takes?

Much longer.

>> Larger integers will be factored eventually -- the wait can be long or
>> short. Probably long if they require the services of lines 82-91.
>>
>> Examples of extremes, both between 1e25 and 1e26:
>> 2,835,334,663,465,375,591,838,337  [3, 19, 37, 71, 947,
>> 19994908447741489]  1.172 secs
>> 9,349,337,574,247,205,482,125,105  [3, 5, 2027, 2311296661,
>> 133039358281] 402.5 secs
>
>
> I'm astounded by the times you quote there.

Excellent!

> The first example includes the prime 19994908447741489, which is *not*
> in the cache, since it is bigger than 1e17. And yet it only takes about
> a second to find all six primes. If true, that's astonishingly good for
> something written in pure Python.

My code uses gmpy2.is_prime()  (lines 79 and 89). is_prime() is VERY fast.

C:\Windows\System32>python -m timeit -v -r 5 -s "from gmpy2 import
is_prime" "is_prime(19994908447741489)"
10 loops -> 0.0008 secs
100 loops -> 0.0073 secs
1000 loops -> 0.0409 secs
1 loops -> 0.394 secs
raw times: 0.392 0.393 0.394 0.392 0.393
1 loops, best of 5: 39.2 usec per loop

and to show its speed for enormous prime ints:
>>> is_prime(987987987199949084477414897654765465435634564357034523045023453475389457937283)
True

C:\Windows\System32>python -m timeit -v -r 5 -s "from gmpy2 import
is_prime" 
"is_prime(987987987199949084477414897654765465435634564357034523045023453475389457937283)"
10 loops -> 0.0077 secs
100 loops -> 0.0765 secs
1000 loops -> 0.758 secs
raw times: 0.757 0.758 0.757 0.757 0.758
1000 loops, best of 5: 757 usec per loop

Also, of the prime factors of  2,835,334,663,465,375,591,838,337, all
but the last one, 19994908447741489, are small and quickly found.
Once factors 3, 19, 37, 71, 947 are found, 19994908447741489 is simply
the quotient of
 2835334663465375591838337 // (3*19*37*71*947), and is_prime confirms
its primacy.

> The second example includes the prime 133039358281, which is smaller
> than 1e17 and so should be in the cache and therefore found (almost)
> instantly.

No, the cache has only primes up to 100,000,000, the last one being 99,999,989.

> And yet you claim it takes nearly seven minutes to find it.
> If correct, that's astonishingly awful given that you have the prime
> numbers already cached.

The delay is caused by the 2nd largest factor being a big one,
2311296661 which is greater than 100 million, whereas in the first
example, the 2nd largest factor is a puny 947 -- but most important, I
think, is that it smaller than 100 million.

> Can you explain those timings? How did you measure them?

Instead of just line 99 at the end of the pasted script, I had

from time import clock as t

t0 = t()
print(factor_integer(2835334663465375591838337))
t1 = t()
print("time =", round((t1-t0), 3))
"""
OUTPUT:
2,835,334,663,465,375,591,838,337 [3, 19, 37, 71, 947, 19994908447741489]
1.121
"""

Essentially the same algorithm is employed in the other script I
pasted at , factor_random_integers().
I've run factor_random_integers a couple of more times, with
num_integers, min_length, max_length = 10, 25, 25. See
, "More output from
factor_random_integers".

Dick
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] list comprehension, testing for multiple conditions

[Pete O'Connell]

Ok thanks for the advice everyone.

Cheers
Pete

On Thu, Aug 23, 2012 at 10:58 AM, Jerry Hill  wrote:
> On Wed, Aug 22, 2012 at 5:23 PM, Pete O'Connell  
> wrote:
>> OK maybe I am wrong about it being slow (I thought for loops were
>> slower than lis comprehensions). But I do know I need it to be as fast
>> as possible if I need to run it on a thousand files each with hundreds
>> of thousands of lines
>
> You're quite likely wrong about that.  The overall runtime of your
> code is very likely to end up dominated by I/O with the hard drive,
> not the microsecond differences in how you process each line.  The way
> to know for sure is to write code that is easy to read and does the
> correct thing.  Then run the code an see if it's too slow for your
> purposes.  If it is, profile the code and see where it's spending all
> of its time.  Only then can you actually optimize the right places.
>
> Once you've identified where your code needs to be faster, you're
> likely to get more mileage out of a clever algorithm change than a
> micro-optimization to your filtering. Or maybe you'll discover that it
> really is worthwhile to optimize your filtering, because it's inside a
> tight loop, and that's where all of your time is spent.  There are a
> lot of tricks you can use to slightly speed up your python code, but
> none of them are worth doing until you know for sure that you're
> optimizing in the right place.  And if it turns out that the code's
> runtime is fine as first written, you don't have to waste your time
> doing lots of work for little gain.
>
> --
> Jerry
> ___
> Tutor maillist  -  Tutor@python.org
> To unsubscribe or change subscription options:
> http://mail.python.org/mailman/listinfo/tutor



-- 
-
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] list comprehension, testing for multiple conditions

[Jerry Hill]

On Wed, Aug 22, 2012 at 5:23 PM, Pete O'Connell  wrote:
> OK maybe I am wrong about it being slow (I thought for loops were
> slower than lis comprehensions). But I do know I need it to be as fast
> as possible if I need to run it on a thousand files each with hundreds
> of thousands of lines

You're quite likely wrong about that.  The overall runtime of your
code is very likely to end up dominated by I/O with the hard drive,
not the microsecond differences in how you process each line.  The way
to know for sure is to write code that is easy to read and does the
correct thing.  Then run the code an see if it's too slow for your
purposes.  If it is, profile the code and see where it's spending all
of its time.  Only then can you actually optimize the right places.

Once you've identified where your code needs to be faster, you're
likely to get more mileage out of a clever algorithm change than a
micro-optimization to your filtering. Or maybe you'll discover that it
really is worthwhile to optimize your filtering, because it's inside a
tight loop, and that's where all of your time is spent.  There are a
lot of tricks you can use to slightly speed up your python code, but
none of them are worth doing until you know for sure that you're
optimizing in the right place.  And if it turns out that the code's
runtime is fine as first written, you don't have to waste your time
doing lots of work for little gain.

-- 
Jerry
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] list comprehension, testing for multiple conditions

[Alan Gauld]

On 22/08/12 22:23, Pete O'Connell wrote:


What makes you say it is "terribly slow"? Perhaps it is as fast as it
could be under the circumstances. (Maybe it takes a long time because
you have a lot of data, not because it is slow.)


OK maybe I am wrong about it being slow (I thought for loops were
slower than lis comprehensions).


For loops are slightly slower than list comprehensions but depending on
what is happening inside the loop/comprehension the difference may not
be significant.
> But I do know I need it to be as fast
> as possible if I need to run it on a thousand files each with hundreds
> of thousands of lines

If you need it as "fast as possible" you need to run it on a 
supercomputer with a program in assembler after many hours of fine 
tuning. But that will cost a lot of money and you probably don't really 
need it.


For example how often do you need to run this process? Is it a one off - 
set it running over lunch and it will likely be done by the time you get 
back. Is it an hourly event? Set it running in the background while you 
carry on processing the results of the previous run.


Steven's point is that usually absolute speed is not as important as 
people think. Modern computers are absurdly fast. When I started using 
Linux it took about 2 hours to build the kernel. Now I can build it in 
about 2 minutes! Processing a thousand files is largely limited by the 
speed of your hard drive rather than the speed of the processing on the CPU.


HTH
--
Alan G
Author of the Learn to Program web site
http://www.alan-g.me.uk/




___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] Hello Python Tutor - help please!

[Alan Gauld]

On 22/08/12 21:51, Cecilia Chavana-Bryant wrote:


def main(fname, sheet_name):
 wb = xlrd.open_workbook(fname)
 sh = wb.sheet_by_name(sheet_name)
 data1 = sh.col_values(0)
 data2 = sh.col_values(1)

 return data1, data2

fname = "Cal_File_P17.xlsx"
sheet_name = "RefPanelData"
(data1, data2) = main(fname)

print data1, data2

... I do not know where the data is being saved to.



That's because it isn't being saved anywhere, it gets
thrown away after printing it.

If you want to save it you need to augment/replace the print
statement with a file write statement(s)

You could also write it out using the OS redirection facility.
On Unix (ie Your MacOS Terminal) thats done by running the
program like:

$ python myprog.py > myOutputFile.txt

Where you substitute your own desired program name and
output filename of course!

But that will just create a text file that looks like the output 
displayed on the Terminal, which is not, I think, what you are after.

You probably want to do some extra work to save as a csv file.

However, before you do that you could look at using the spreadsheet's 
SaveAs feature to export as a CSV directly, but it depends on how many 
spreadsheets you need to convert!


HTH
--
Alan G
Author of the Learn to Program web site
http://www.alan-g.me.uk/

___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] Hello Python Tutor - help please!

[Alan Gauld]

On 22/08/12 22:51, Cecilia Chavana-Bryant wrote:

Steven, (now from my new account without all the long-winded signature)
can files be attached to posts in this forum?


Yes they can, but we prefer if you just include them in the body if they 
are fairly short (<100 lines?) or put them on a pastebin with a link.


Some mail tools/servers/smartphones don't like mail attachments.


--
Alan G
Author of the Learn to Program web site
http://www.alan-g.me.uk/

___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] list comprehension, testing for multiple conditions

[ALAN GAULD]

Note uits not just that its on multiple lines, 

its the fact it has three distinct if statements. 
All of them must be satisfied to be included 

in the comprehension.

You could do it on 3 lines with one if statement:

theTextAsListNoVnOrVtOrEmptyLine = [x for x in theTextAsListStripped
                                      if "vn" not in x and
                                     "vt" not in x and
                                      x!= ""]

But I still think the three if statements are clearer.
However, if you need to use 'or' instead of 'and' then you 

need to go back to a compound statement but multi-line 

layout still aids legibility.


Alan Gauld
Author of the Learn To Program website
http://www.alan-g.me.uk/



>
> From: Pete O'Connell 
>To: Alan Gauld ; tutor@python.org 
>Sent: Wednesday, 22 August 2012, 22:27
>Subject: Re: [Tutor] list comprehension, testing for multiple conditions
> 
>On Thu, Aug 23, 2012 at 4:16 AM, Alan Gauld  wrote:
>
>> theTextAsListNoVnOrVtOrEmptyLine = [x for x in theTextAsListStripped
>>                                                 if "vn" not in x
>>                                                 if "vt" not in x
>>                                                 if x!= ""]
>>
>> It's slightly more verbose but it makes the rules more explicit, IMHO.
>
>I agree, it seems easier to read when written on multiple lines. I'll
>do it that way,
>Thanks
>Pete
>
>
>___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] list comprehension, testing for multiple conditions

[Mark Lawrence]

On 22/08/2012 22:23, Pete O'Connell wrote:

On Thu, Aug 23, 2012 at 2:53 AM, Steven D'Aprano  wrote:


Further to Steven's sound advice take a look at this 
http://wiki.python.org/moin/PythonSpeed/PerformanceTips  IMHO the part 
on profiling is particulary useful.


--
Cheers.

Mark Lawrence.

___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor

[Tutor] Hello Python Tutor - help please!

[Cecilia Chavana-Bryant]

Steven, (now from my new account without all the long-winded signature) can
files be attached to posts in this forum?

Cecilia
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] list comprehension, testing for multiple conditions

[Pete O'Connell]

On Thu, Aug 23, 2012 at 4:16 AM, Alan Gauld  wrote:

> theTextAsListNoVnOrVtOrEmptyLine = [x for x in theTextAsListStripped
> if "vn" not in x
> if "vt" not in x
> if x!= ""]
>
> It's slightly more verbose but it makes the rules more explicit, IMHO.

I agree, it seems easier to read when written on multiple lines. I'll
do it that way,
Thanks
Pete
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] list comprehension, testing for multiple conditions

[Pete O'Connell]

On Thu, Aug 23, 2012 at 2:53 AM, Steven D'Aprano  wrote:
> On 22/08/12 20:28, Pete O'Connell wrote:
>>
>> Hi. The next step for me to parse the file as I want to is to change
>> lines that look like this:
>> f 21/21/21 22/22/22 24/24/23 23/23/24
>> into lines that look like this:
>> f 21 22 23 24
>
>
> In English, what is the rule you are applying here? My guess is:
>
> "Given three numbers separated by slashes, ignore the first two numbers
> and keep the third."
>
> E.g. "17/25/97" => 97.
>
> Am I close?

Hi Steve, yes that is correct

>
>
>
>> Below is my terribly slow loop for doing this. Any suggestions about
>> how to make this code more efficient would be greatly appreciated
>
>
> What makes you say it is "terribly slow"? Perhaps it is as fast as it
> could be under the circumstances. (Maybe it takes a long time because
> you have a lot of data, not because it is slow.)

OK maybe I am wrong about it being slow (I thought for loops were
slower than lis comprehensions). But I do know I need it to be as fast
as possible if I need to run it on a thousand files each with hundreds
of thousands of lines

>
> The first lesson of programming is not to be too concerned about speed
> until your program is correct.
>
> Like most such guidelines, this is not entirely true -- you don't want
> to write code which is unnecessarily slow. But the question you should
> be asking is, "is it fast enough?" rather than "is it fast?".
>
> Also, the sad truth is that Python tends to be slower than some other
> languages. (It's also faster than some other languages too.) But the
> general process is:
>
> 1) write something that works correctly;
>
> 2) if it is too slow, try to speed it up in Python;
>
> 3) if that's still too slow, try using something like cython or PyPy
>
> 4) if all else fails, now that you have a working prototype, re-write
> it again in C, Java, Lisp or Haskell.
>
> Once they see how much more work is involved in writing fast C code,
> most people decide that "fast enough" is fast enough :)

OK I will keep it as is and see if I can live with it.

Thanks
Pete

>
>
>
>> with open(fileName) as lines:
>>  theGoodLines = [line.strip("\n") for line in lines if "vn" not in
>> line and "vt" not in line and line != "\n"]
>
>
> I prefer to write code in chains of filters.
>
> with open(fileName) as lines:
> # get rid of leading and trailing whitespace, including newlines
> lines = (line.strip() for line in lines)
> # ignore blanks
> lines = (line in lines if line)
> # ignore lines containing "vn" or "vt"
> theGoodLines = [line in lines if not ("vn" in line or "vt" in line)]
>
> Note that only the last step is a list comprehension using [ ], the others
> are generator expressions using ( ) instead.
>
> Will the above be faster than your version? I have no idea. But I think it
> is more readable and understandable. Some people might disagree.
>
>
>
>> for i in range(len(theGoodLines)):
>>  if theGoodLines[i][0] == "f":
>>  aGoodLineAsList = theGoodLines[i].split(" ")
>>  theGoodLines[i] = aGoodLineAsList[0] + " " +
>> aGoodLineAsList[1].split("/")[-1] + " " +
>> aGoodLineAsList[2].split("/")[-1] + " " +
>> aGoodLineAsList[3].split("/")[-1] + " " +
>> aGoodLineAsList[4].split("/")[-1]
>
>
>
> Start with a helper function:
>
> def extract_last_item(term):
> """Extract the item from a term like a/b/c"""
> return term.split("/")[-1]
>
>
> for i, line in enumerate(theGoodLines):
> if line[0] == "f":
> terms = line.split()
> theGoodLines[i] = " ".join([extract_last_item(t) for t in terms])
>
>
>
> See how you go with that.
>
>
>
> --
> Steven
>
> ___
> Tutor maillist  -  Tutor@python.org
> To unsubscribe or change subscription options:
> http://mail.python.org/mailman/listinfo/tutor



-- 
-
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] list comprehension, testing for multiple conditions

[Pete O'Connell]

Thanks eryksun, that is a very clear example.
Cheers
pete
On Wed, Aug 22, 2012 at 11:16 PM, eryksun  wrote:
> On Wed, Aug 22, 2012 at 3:06 AM, Peter Otten <__pete...@web.de> wrote:
>>
>> wanted = [line.strip("\n") for line in lines
>>   if "vn" not in line and "vt" not in line and line != "\n"]
>
> Here's an equivalent expression with the negation factored out:
>
> not ("vn" in line or "vt" in line or line == "\n")
>
> http://en.wikipedia.org/wiki/De_Morgan%27s_laws
>
> If you have a lot of tests all using the same operator (e.g. "in"),
> you can use "any" (OR) or "all" (AND) with a generator expression:
>
> vals = ["vn", "vt", "vu", "vv", "vw", "vx", "vy", "vz"]
> wanted = [line for line in lines if not any(v in line for v in vals)]
> ___
> Tutor maillist  -  Tutor@python.org
> To unsubscribe or change subscription options:
> http://mail.python.org/mailman/listinfo/tutor



-- 
-
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] Hello Python Tutor - help please!

[Ray Jones]

I highly recommend the Google Python class that is found on YouTube.
The first video is found at http://www.youtube.com/watch?v=tKTZoB2Vjuk
The supporting class materials and assignments are found at
http://code.google.com/edu/languages/google-python-class/ . This series
of videos begins at a pretty basic level, but subsequent videos increase
the difficulty level pretty rapidly.

Don't despair - the concept of how to properly manipulate strings,
lists, tuples, and other objects is rather daunting, but if you're
working on your doctoral studies, you already know that complex concepts
aren't simply soaked up like water to a sponge ;).


Ray

On 08/22/2012 03:10 AM, Cecilia Chavana-Bryant wrote:
> Dear all,
>
> I am just returning to my doctoral studies after a 7-month medical
> leave and desperately trying to catch up for lost time. I am
> COMPLETELY new to programming, well, I did try learning C for 3 weeks
> 3 yrs ago (with very little success) but had to stop and then spent 2
> years in the Amazon climbing trees (lots more enjoyable than learning
> to programme!) and collecting loads of field data that I now need to
> post-process and analyse. By the way, the 3 weeks I spent trying to
> learn C really ended up being spent trying to get to grips with using
> a terminal for the first time in my life. 
>
> Since getting back to work, I was advised to try learning Python
> instead of C as it is a much easier first language to learn. I have
> been trying, but again, to not great success. I started following "A
> Primer on Scientific programming with Python" but I kept getting lost
> and stuck, specially on the exercises. I have also been advised that I
> should not try to learn programming by following guides but by trying
> to write the programmes I need to analyse my data. Although I can
> understand the logic behind this last bit of advise (it gives context
> and direction to the learning process) I have also gotten stuck trying
> this approach as "I do not know how to programme!". Thus, I was hoping
> that some of you can remember how you got started and point me towards
> any really good interactive learning guides/materials and/or have a
> good learning strategy for a complete beginner. I have searched the
> web and was overwhelmed by choice of tutorials and guides. I have
> skimmed through a couple of tutorials but then fail to see how all
> that relates to my own work and I get stuck with what seems like basic
> important concepts so I don't progress. I then think I should try to
> make some progress with my own data analysing and go back to trying to
> learn to write a programme for my specific needs and get stuck again
> because this requires more advanced skills then the basic programming
> concepts I have been reading about on the learning guides. So, I am
> now feeling VERY frustrated and have no idea what on Earth I am doing!
> Can anyone please offer guidance in my learning process? I don't know
> how and what I should be spending my time learning first and/or if I
> should focus my learning towards the skill areas I will require to
> write my specific programmes, although I have no idea what these are.
> I would like advise on finding some really good interactive(let you
> know if your solution to an exercise is correct or not) and or video
> tutorials that give you feedback on the solutions you write to
> exercises.   
>
> Many thanks in advance for all your help, it will be much appreciated!
> 
>
>
> Cecilia Chavana-Bryant
> DPhil Candidate - Remote sensing and tropical phenology
> Environmental Change Institute
> School of Geography and the Environment
> University of Oxford
> South Parks Road, Oxford, OX1 3QY
> Web: http://www.eci.ox.ac.uk/teaching/doctoral/chavanabryantcecilia.php
> Tel Direct: +44 (0)1865 275861
> Fax: +44 (0)1865 275885
>
>
> ___
> Tutor maillist  -  Tutor@python.org
> To unsubscribe or change subscription options:
> http://mail.python.org/mailman/listinfo/tutor

___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] pickle problems

[Steven D'Aprano]

On 23/08/12 02:17, Richard D. Moores wrote:

I've incorporated many of the suggestions I've received here.

Here's a function, factor_integer(), for quickly factoring any integer
up to 1e17:.


That relies on a pre-existing cache of prime numbers. If you don't use
those cached prime numbers, do you know how long your code takes?



Larger integers will be factored eventually -- the wait can be long or
short. Probably long if they require the services of lines 82-91.

Examples of extremes, both between 1e25 and 1e26:
2,835,334,663,465,375,591,838,337  [3, 19, 37, 71, 947,
19994908447741489]  1.172 secs
9,349,337,574,247,205,482,125,105  [3, 5, 2027, 2311296661,
133039358281] 402.5 secs


I'm astounded by the times you quote there.

The first example includes the prime 19994908447741489, which is *not*
in the cache, since it is bigger than 1e17. And yet it only takes about
a second to find all six primes. If true, that's astonishingly good for
something written in pure Python.

The second example includes the prime 133039358281, which is smaller
than 1e17 and so should be in the cache and therefore found (almost)
instantly. And yet you claim it takes nearly seven minutes to find it.
If correct, that's astonishingly awful given that you have the prime
numbers already cached.

Can you explain those timings? How did you measure them?



--
Steven
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor

[Tutor] better tools

[Don Jennings]

[slightly OT]

After watching Bret Victor's talk[1], I want **much** better tools for 
programming (and all of the other stuff I do on the computer).

John Resig, creator of jQuery,  claims[2] that Victor's presentation inspired 
the new platform for learning javascript at Khan Academy[3]. I plan to try it 
out.

Take care,
Don

[1] http://vimeo.com/36579366

[2] http://ejohn.org/blog/introducing-khan-cs/

[3] http://www.khanacademy.org/cs
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] Hello Python Tutor - help please!

[Walter Prins]

Hi Cecilia,

You've had a lot of good replies already, but I'd like to add the
following points if I may:

1) You probably should figure out as much as that's possible up front
exactly you're trying to do in terms of data processing first (e.g.
some idea of the stats, graphs, summaries, operations etc), and then
figure out whether Python is in fact the quickest/best way to get
there for you.  Python is very capable, it has many data analysis
libraries and so on and is used by many scientists (NumPy, SciPy,
Pandas comes to mind offhand), but then there are also many other
languages and system also appropriate in this sphere.  (R comes to
mind.)  Your goal (from my point of view) is not to become a
programmer, but to get your research done.  Python may be the way to
achieve that, but from where I'm sitting it may also not be.

2) It may be useful to take some suitable courses from some of the
very good free online courses now available from various sources such
as Coursera.  Some examples that seem relevant:

Computing for Data Analysis:
https://www.coursera.org/course/compdata

Mathematics Biostatistics Boot camp:
https://www.coursera.org/course/biostats

Data Analysis
https://www.coursera.org/course/dataanalysis

There are also courses covering basic programming, including Python,
for example: https://www.coursera.org/course/programming1

HTH,

Walter
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] Hello Python Tutor - help please!

[Alan Gauld]

On 22/08/12 11:10, Cecilia Chavana-Bryant wrote:


"I do not know how to programme!". Thus, I was hoping that some of you
can remember how you got started and point me towards any really good
interactive learning guides/materials and/or have a good learning
strategy for a complete beginner.


At the risk of self promotion you could try the early stages of my 
tutorial (see .sig).


It starts from level zero and explains the concepts of programming 
before getting started writing code. I personally found that to be 
fundamental to my learning when I started (and why I included it!).


It then goes through the basic programming structures using very
simple examples - a multiplication table and address book mainly.
(It includes VBScript and Javascript examples too but feel free to 
ignore them if you find they confuse more than help! The intent is to 
show the common structures present in all programming languages :-)


Whether you progress beyond the first two sections depends on what you 
want to do next. Those would be enough to start writing programs related 
to your work and switching to another tutor at that point

might be effective.

But the concepts topics at the start of my tutor are I think
relatively unique, and if diving straight into writing code isn't 
working maybe a view of the bigger picture will help.


--
Alan G
Author of the Learn to Program web site
http://www.alan-g.me.uk/

___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] Thanks!!

[Mark Lawrence]

On 22/08/2012 16:48, Cecilia Chavana-Bryant wrote:

Hola,

Just a big THANK YOU!!! to everyone that has replied to my post. I have been so 
confused about how to get started and since I am working from home, it has been 
a very frustrating and lonely experience so far. Many, many thanks for your 
empathy, encouragement and availability!!


Cecilia Chavana-Bryant
DPhil Candidate - Remote sensing and tropical phenology
Environmental Change Institute
School of Geography and the Environment
University of Oxford
South Parks Road, Oxford, OX1 3QY
Web: http://www.eci.ox.ac.uk/teaching/doctoral/chavanabryantcecilia.php
Tel Direct: +44 (0)1865 275861
Fax: +44 (0)1865 275885



___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor



Please save your thanks until you're graduated from the tutor mailing 
list to the main Python mailing list :)


For more help take a look here 
http://wiki.python.org/moin/BeginnersGuide/NonProgrammers


--
Cheers.

Mark Lawrence.

___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] pickle problems

[Richard D. Moores]

I've incorporated many of the suggestions I've received here.

Here's a function, factor_integer(), for quickly factoring any integer
up to 1e17: .

Larger integers will be factored eventually -- the wait can be long or
short. Probably long if they require the services of lines 82-91.

Examples of extremes, both between 1e25 and 1e26:
2,835,334,663,465,375,591,838,337  [3, 19, 37, 71, 947,
19994908447741489]  1.172 secs
9,349,337,574,247,205,482,125,105  [3, 5, 2027, 2311296661,
133039358281] 402.5 secs

These were found with the use of a related script,
"factor_random_integers.py", pasted at 


Dick Moores
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] Hello Python Tutor - help please!

[Joel Goldstick]

On Wed, Aug 22, 2012 at 11:33 AM, Cecilia Chavana-Bryant
 wrote:
> Hola Bill,
>
> Many thanks for your reply to my post, you seem to understand the
> predicament I am in very well. Unfortunately, I am currently working from
> home and do not have someone close by to help and this is why I came to this
> space. This is also why I asked for advise about interactive tutorials or
> video tutorials. I have found that I keep getting lost with the more
> traditional tutorials where you just read and then do exercises. Following
> the guide I mentioned on my initial post I got through the first 2 chapters
> but I found them quite hard going. I don't know if this makes me not a
> complete beginner but I certainly do not feel like I learned much from
> reading them. Maybe it is the trying to learn the "computer ecosystem" of
> terminal commands at the same time that is making this learning process so
> tough.
>
> With respect to my field data, during my 2 yrs of fieldwork I collected a
> large amount of data which is currently stored in excel files. My research
> involves remote sensing (data from Earth-observation satellites) and I work
> with data from the MODIS NASA satellite which monitors the health of forest
> canopies using reflectance data. My research is based in the Amazon. I have
> collected field data to monitor the leaf dynamics of canopy leaves during
> the dry season. Dry season is the time of year when many tropical trees
> change their old leaves for new ones. New leaves are more photosynthetically
> active (absorb more carbon from and release more oxygen into the atmosphere)
> so the leaf exchange of such a large forest region as the Amazon can have
> huge effects on regional and global carbon and water cycles and thus on
> global climate (apologies if I'm giving you loads more information than you
> need or requested?!). My data involves a large amount of data on leaf
> demography (we demographically surveyed more than 120,000 leaves), and
> thousands of morphological and reflectance measurements. I will have to
> reorganise this data and create a few easily manipulable datasets so I can
> sort data according to leaf age, canopy position, date, etc. Then I will
> have to do statistical analyses on the data. I will also have to model some
> of the data.
>
> Many thanks for taking the time to respond to my post so comprehensively and
> for your good wishes.
>
>
> Cecilia Chavana-Bryant
> DPhil Candidate - Remote sensing and tropical phenology
> Environmental Change Institute
> School of Geography and the Environment
> University of Oxford
> South Parks Road, Oxford, OX1 3QY
> Web: http://www.eci.ox.ac.uk/teaching/doctoral/chavanabryantcecilia.php
> Tel Direct: +44 (0)1865 275861
> Fax: +44 (0)1865 275885
> 
> From: William R. Wing (Bill Wing) [w...@mac.com]
> Sent: 22 August 2012 15:17
> To: Cecilia Chavana-Bryant
> Cc: William R. Wing (Bill Wing)
> Subject: Re: [Tutor] Hello Python Tutor - help please!
>
> On Aug 22, 2012, at 6:10 AM, Cecilia Chavana-Bryant
>  wrote:
>
> Dear all,
>
> I am just returning to my doctoral studies after a 7-month medical leave and
> desperately trying to catch up for lost time. I am COMPLETELY new to
> programming, well, I did try learning C for 3 weeks 3 yrs ago (with very
> little success) but had to stop and then spent 2 years in the Amazon
> climbing trees (lots more enjoyable than learning to programme!) and
> collecting loads of field data that I now need to post-process and analyse.
> By the way, the 3 weeks I spent trying to learn C really ended up being
> spent trying to get to grips with using a terminal for the first time in my
> life.
>
>
> Could you say a few words about what the field data is, and how you hope to
> analyze it.  That is, are you headed in the direction of plotting species
> density on maps, or the time evolution of something, or doing statistics?
>
> Since getting back to work, I was advised to try learning Python instead of
> C as it is a much easier first language to learn. I have been trying, but
> again, to not great success. I started following "A Primer on Scientific
> programming with Python" but I kept getting lost and stuck, specially on the
> exercises. I have also been advised that I should not try to learn
> programming by following guides but by trying to write the programmes I need
> to analyse my data. Although I can understand the logic behind this last bit
> of advise (it gives context and direction to the learning process) I have
> also gotten stuck trying this approach as "I do not know how to programme!".
> Thus, I was hoping that some of you can remember how you got started and
> point me towards any really good interactive learning guides/materials
> and/or have a good learning strategy for a complete beginner. I have
> searched the web and was overwhelmed by choice of tutorials and guides. I
> have skimmed through a couple of tutorials but then fail to see how all that
> relates to my 

Re: [Tutor] list comprehension, testing for multiple conditions

[Alan Gauld]

On 22/08/12 10:51, Pete O'Connell wrote:


theTextAsListNoVnOrVtOrEmptyLine = [x for x in theTextAsListStripped if "vn" not in x  and 
"vt" not in x and x!= ""]


It works but what I don't understand about this line is why the ands
are not ors


Because 'or' would include x if any one of the conditions was true.

For example if 'vt' existed but not 'vn' then the line would be included 
because vn was not in the line, even though vt was.


We are all assuming that you want to exclude the line if any of the 
phrases is present.


Thats why I actually prefer the multiple if version that Peter posted:

theTextAsListNoVnOrVtOrEmptyLine = [x for x in theTextAsListStripped
if "vn" not in x
if "vt" not in x
if x!= ""]

It's slightly more verbose but it makes the rules more explicit, IMHO.


--
Alan G
Author of the Learn to Program web site
http://www.alan-g.me.uk/

___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor

[Tutor] Thanks!!

[Cecilia Chavana-Bryant]

Hola,

Just a big THANK YOU!!! to everyone that has replied to my post. I have been so 
confused about how to get started and since I am working from home, it has been 
a very frustrating and lonely experience so far. Many, many thanks for your 
empathy, encouragement and availability!!


Cecilia Chavana-Bryant
DPhil Candidate - Remote sensing and tropical phenology
Environmental Change Institute
School of Geography and the Environment
University of Oxford
South Parks Road, Oxford, OX1 3QY
Web: http://www.eci.ox.ac.uk/teaching/doctoral/chavanabryantcecilia.php
Tel Direct: +44 (0)1865 275861
Fax: +44 (0)1865 275885
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] Hello Python Tutor - help please!

[Cecilia Chavana-Bryant]

Hola Bill,

Many thanks for your reply to my post, you seem to understand the predicament I 
am in very well. Unfortunately, I am currently working from home and do not 
have someone close by to help and this is why I came to this space. This is 
also why I asked for advise about interactive tutorials or video tutorials. I 
have found that I keep getting lost with the more traditional tutorials where 
you just read and then do exercises. Following the guide I mentioned on my 
initial post I got through the first 2 chapters but I found them quite hard 
going. I don't know if this makes me not a complete beginner but I certainly do 
not feel like I learned much from reading them. Maybe it is the trying to learn 
the "computer ecosystem" of terminal commands at the same time that is making 
this learning process so tough.

With respect to my field data, during my 2 yrs of fieldwork I collected a large 
amount of data which is currently stored in excel files. My research involves 
remote sensing (data from Earth-observation satellites) and I work with data 
from the MODIS NASA satellite which monitors the health of forest canopies 
using reflectance data. My research is based in the Amazon. I have collected 
field data to monitor the leaf dynamics of canopy leaves during the dry season. 
Dry season is the time of year when many tropical trees change their old leaves 
for new ones. New leaves are more photosynthetically active (absorb more carbon 
from and release more oxygen into the atmosphere) so the leaf exchange of such 
a large forest region as the Amazon can have huge effects on regional and 
global carbon and water cycles and thus on global climate (apologies if I'm 
giving you loads more information than you need or requested?!). My data 
involves a large amount of data on leaf demography (we demographically surveyed 
more than 120,000 leaves), and thousands of morphological and reflectance 
measurements. I will have to reorganise this data and create a few easily 
manipulable datasets so I can sort data according to leaf age, canopy position, 
date, etc. Then I will have to do statistical analyses on the data. I will also 
have to model some of the data.

Many thanks for taking the time to respond to my post so comprehensively and 
for your good wishes.


Cecilia Chavana-Bryant
DPhil Candidate - Remote sensing and tropical phenology
Environmental Change Institute
School of Geography and the Environment
University of Oxford
South Parks Road, Oxford, OX1 3QY
Web: http://www.eci.ox.ac.uk/teaching/doctoral/chavanabryantcecilia.php
Tel Direct: +44 (0)1865 275861
Fax: +44 (0)1865 275885

From: William R. Wing (Bill Wing) [w...@mac.com]
Sent: 22 August 2012 15:17
To: Cecilia Chavana-Bryant
Cc: William R. Wing (Bill Wing)
Subject: Re: [Tutor] Hello Python Tutor - help please!

On Aug 22, 2012, at 6:10 AM, Cecilia Chavana-Bryant 
mailto:cecilia.chavana-bry...@ouce.ox.ac.uk>>
 wrote:

Dear all,

I am just returning to my doctoral studies after a 7-month medical leave and 
desperately trying to catch up for lost time. I am COMPLETELY new to 
programming, well, I did try learning C for 3 weeks 3 yrs ago (with very little 
success) but had to stop and then spent 2 years in the Amazon climbing trees 
(lots more enjoyable than learning to programme!) and collecting loads of field 
data that I now need to post-process and analyse. By the way, the 3 weeks I 
spent trying to learn C really ended up being spent trying to get to grips with 
using a terminal for the first time in my life.


Could you say a few words about what the field data is, and how you hope to 
analyze it.  That is, are you headed in the direction of plotting species 
density on maps, or the time evolution of something, or doing statistics?

Since getting back to work, I was advised to try learning Python instead of C 
as it is a much easier first language to learn. I have been trying, but again, 
to not great success. I started following "A Primer on Scientific programming 
with Python" but I kept getting lost and stuck, specially on the exercises. I 
have also been advised that I should not try to learn programming by following 
guides but by trying to write the programmes I need to analyse my data. 
Although I can understand the logic behind this last bit of advise (it gives 
context and direction to the learning process) I have also gotten stuck trying 
this approach as "I do not know how to programme!". Thus, I was hoping that 
some of you can remember how you got started and point me towards any really 
good interactive learning guides/materials and/or have a good learning strategy 
for a complete beginner. I have searched the web and was overwhelmed by choice 
of tutorials and guides. I have skimmed through a couple of tutorials but then 
fail to see how all that relates to my own work and I get stuck with what seems 
like basic important concepts so I don't progress. I then think I should try to 
make some prog

Re: [Tutor] list comprehension, testing for multiple conditions

[Steven D'Aprano]

On 22/08/12 20:28, Pete O'Connell wrote:

Hi. The next step for me to parse the file as I want to is to change
lines that look like this:
f 21/21/21 22/22/22 24/24/23 23/23/24
into lines that look like this:
f 21 22 23 24


In English, what is the rule you are applying here? My guess is:

"Given three numbers separated by slashes, ignore the first two numbers
and keep the third."

E.g. "17/25/97" => 97.

Am I close?



Below is my terribly slow loop for doing this. Any suggestions about
how to make this code more efficient would be greatly appreciated


What makes you say it is "terribly slow"? Perhaps it is as fast as it
could be under the circumstances. (Maybe it takes a long time because
you have a lot of data, not because it is slow.)

The first lesson of programming is not to be too concerned about speed
until your program is correct.

Like most such guidelines, this is not entirely true -- you don't want
to write code which is unnecessarily slow. But the question you should
be asking is, "is it fast enough?" rather than "is it fast?".

Also, the sad truth is that Python tends to be slower than some other
languages. (It's also faster than some other languages too.) But the
general process is:

1) write something that works correctly;

2) if it is too slow, try to speed it up in Python;

3) if that's still too slow, try using something like cython or PyPy

4) if all else fails, now that you have a working prototype, re-write
it again in C, Java, Lisp or Haskell.

Once they see how much more work is involved in writing fast C code,
most people decide that "fast enough" is fast enough :)



with open(fileName) as lines:
 theGoodLines = [line.strip("\n") for line in lines if "vn" not in
line and "vt" not in line and line != "\n"]


I prefer to write code in chains of filters.

with open(fileName) as lines:
# get rid of leading and trailing whitespace, including newlines
lines = (line.strip() for line in lines)
# ignore blanks
lines = (line in lines if line)
# ignore lines containing "vn" or "vt"
theGoodLines = [line in lines if not ("vn" in line or "vt" in line)]

Note that only the last step is a list comprehension using [ ], the others
are generator expressions using ( ) instead.

Will the above be faster than your version? I have no idea. But I think it
is more readable and understandable. Some people might disagree.



for i in range(len(theGoodLines)):
 if theGoodLines[i][0] == "f":
 aGoodLineAsList = theGoodLines[i].split(" ")
 theGoodLines[i] = aGoodLineAsList[0] + " " +
aGoodLineAsList[1].split("/")[-1] + " " +
aGoodLineAsList[2].split("/")[-1] + " " +
aGoodLineAsList[3].split("/")[-1] + " " +
aGoodLineAsList[4].split("/")[-1]



Start with a helper function:

def extract_last_item(term):
"""Extract the item from a term like a/b/c"""
return term.split("/")[-1]


for i, line in enumerate(theGoodLines):
if line[0] == "f":
terms = line.split()
theGoodLines[i] = " ".join([extract_last_item(t) for t in terms])



See how you go with that.



--
Steven
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] Hello Python Tutor - help please!

[Steven D'Aprano]

Hello Cecilia,

My replies are below, interleaved with your comments, which are
prefixed with > marks.


On 22/08/12 20:10, Cecilia Chavana-Bryant wrote:


By the way, the 3 weeks I spent trying to learn C really ended up
being spent trying to get to grips with using a terminal for the
first time in my life.


Unfortunately, there will be a certain amount of that, or at least
something quite similar to a terminal. Fortunately, using Python in
the terminal is usually MUCH easier than C, and in my experience
using Python's interactive interpreter is one of the best ways to
learn the language.

What sort of computer are you using? Windows, Linux, Macintosh, or
something different? I think that most of the people here use Linux
or Windows, but we can probably help you one way or the other.



Since getting back to work, I was advised to try learning Python
instead of C as it is a much easier first language to learn.


Yes, definitely, but it is still programming. Don't overestimate
the difficulty, if it was hard programmers couldn't learn to do it
*wink*, but on the other hand it's not trivial either.



I have been trying, but again, to not great success. I started
following "A Primer on Scientific programming with Python" but I
kept getting lost and stuck, specially on the exercises.


If you are willing to make a good, honest effort on the exercises
first, we're happy to help you with them. We do like to see your
attempt first, so that we can suggest fixes rather than solve the
problem for you.



I have also been advised that I should not try to learn
programming by following guides but by trying to write the
programmes I need to analyse my data. Although I can understand
the logic behind this last bit of advise (it gives context and
direction to the learning process) I have also gotten stuck
trying this approach as "I do not know how to programme!".


I'm entirely with you there. Having direction in your learning is
a good thing. But until you understand the basic skills you need,
it will be nothing but frustration and pain!

I recommend that, if nothing else, you work through some basic
tutorials so that you at least have some idea of basic language
constructs like:

- strings
- lists
- functions
- ints
- floats
- basic arithmetic
- importing modules
etc.

You could start with the official Python tutorial:

http://docs.python.org/tutorial/index.html

although I find that it is sometimes a bit wordy. (I should
talk...)

If you get stuck, don't hesitate to come back and ask
questions, that's why we're here.



Thus, I was hoping that some of you can remember how you got
started


I learned from the book "Learning Python" by Mark Lutz and
David Ascher, and then by writing oodles and oodles of really
bad code which I have long since thrown away :)


[...]

So, I am now feeling VERY frustrated and have no idea what on
Earth I am doing! Can anyone please offer guidance in my
learning process?


I feel your pain! That's how I feel every time I try to understand
monads in Haskell (don't ask!).

How about if you start off with a simple question you would like
to solve using Python? Something relevant to your work. You may
need to explain some of the concepts to us, since we're not
ecologists. (At least, I'm not, I can't speak for others.)

We can then try to guide you to a solution and introduce concepts
as we go.




--
Steven
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] Hello Python Tutor - help please!

[leon zaat]

If you don't have any prior programmers skills, i would advice first to learn 
the basics. You will need a good foundation, before it is possible to create 
complex functions.
Starting with complex functions is only frustrating if you don't understand the 
basics. 

From: cecilia.chavana-bry...@ouce.ox.ac.uk
To: tutor@python.org
Date: Wed, 22 Aug 2012 10:10:46 +
Subject: [Tutor] Hello Python Tutor - help please!







Dear all,



I am just returning to my doctoral studies after a 7-month medical leave and 
desperately trying to catch up for lost time. I am COMPLETELY new to 
programming, well, I did try learning C for 3 weeks 3 yrs ago (with very little 
success) but had to stop and
 then spent 2 years in the Amazon climbing trees (lots more enjoyable than 
learning to programme!) and collecting loads of field data that I now need to 
post-process and analyse. By the way, the 3 weeks I spent trying to learn C 
really ended up being spent
 trying to get to grips with using a terminal for the first time in my life. 



Since getting back to work, I was advised to try learning Python instead of C 
as it is a much easier first language to learn. I have been trying, but again, 
to not great success. I started following "A Primer on Scientific programming 
with Python" but
 I kept getting lost and stuck, specially on the exercises. I have also been 
advised that I should not try to learn programming by following guides but by 
trying to write the programmes I need to analyse my data. Although I can 
understand the logic behind this
 last bit of advise (it gives context and direction to the learning process) I 
have also gotten stuck trying this approach as "I do not know how to 
programme!". Thus, I was hoping that some of you can remember how you got 
started and point me towards any really
 good interactive learning guides/materials and/or have a good learning 
strategy for a complete beginner. I have searched the web and was overwhelmed 
by choice of tutorials and guides. I have skimmed through a couple of tutorials 
but then fail to see how all
 that relates to my own work and I get stuck with what seems like basic 
important concepts so I don't progress. I then think I should try to make some 
progress with my own data analysing and go back to trying to learn to write a 
programme for my specific needs
 and get stuck again because this requires more advanced skills then the basic 
programming concepts I have been reading about on the learning guides. So, I am 
now feeling VERY frustrated and have no idea what on Earth I am doing! Can 
anyone please offer guidance
 in my learning process? I don't know how and what I should be spending my time 
learning first and/or if I should focus my learning towards the skill areas I 
will require to write my specific programmes, although I have no idea what 
these are. I would like
 advise on finding some really good interactive(let you know if your solution 
to an exercise is correct or not) and or video tutorials that give you feedback 
on the solutions you write to exercises.   




Many thanks in advance for all your help, it will be much appreciated!








Cecilia Chavana-Bryant

DPhil Candidate - Remote sensing and tropical phenology

Environmental Change Institute

School of Geography and the Environment

University of Oxford

South Parks Road, Oxford, OX1 3QY

Web: http://www.eci.ox.ac.uk/teaching/doctoral/chavanabryantcecilia.php

Tel Direct: +44 (0)1865 275861

Fax: +44 (0)1865 275885









___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor   
  ___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] Hello Python Tutor - help please!

[Mario Cacciatore]

Hello,

My highest recommendation for you is to start with a simple hello world
program. Study that program, each line. Think about how and most
importantly, why it works.

Then, extend on it. Make it write to a file instead of a terminal. Then
make it read from a file and print to the terminal. Then make it print one
letter to each file, then read them back in and reconstruct the sentence.

Just take it slow, and one step at a time. So many times people simply go
for complex solutions and get lost in the complexity.

Sorry for top posting. My phone client doesn't support inline replies.

-Mario
--
From: Cecilia Chavana-Bryant
Sent: 8/22/2012 6:35 AM
To: tutor@python.org
Subject: [Tutor] Hello Python Tutor - help please!

   Dear all,

 I am just returning to my doctoral studies after a 7-month medical leave
and desperately trying to catch up for lost time. I am COMPLETELY new to
programming, well, I did try learning C for 3 weeks 3 yrs ago (with very
little success) but had to stop and then spent 2 years in the Amazon
climbing trees (lots more enjoyable than learning to programme!) and
collecting loads of field data that I now need to post-process and analyse.
By the way, the 3 weeks I spent trying to learn C really ended up being
spent trying to get to grips with using a terminal for the first time in my
life.

 Since getting back to work, I was advised to try learning Python instead
of C as it is a much easier first language to learn. I have been trying,
but again, to not great success. I started following "A Primer on
Scientific programming with Python" but I kept getting lost and stuck,
specially on the exercises. I have also been advised that I should not try
to learn programming by following guides but by trying to write the
programmes I need to analyse my data. Although I can understand the logic
behind this last bit of advise (it gives context and direction to the
learning process) I have also gotten stuck trying this approach as "I do
not know how to programme!". Thus, I was hoping that some of you can
remember how you got started and point me towards any really good
interactive learning guides/materials and/or have a good learning strategy
for a complete beginner. I have searched the web and was overwhelmed by
choice of tutorials and guides. I have skimmed through a couple of
tutorials but then fail to see how all that relates to my own work and I
get stuck with what seems like basic important concepts so I don't
progress. I then think I should try to make some progress with my own data
analysing and go back to trying to learn to write a programme for my
specific needs and get stuck again because this requires more advanced
skills then the basic programming concepts I have been reading about on the
learning guides. So, I am now feeling VERY frustrated and have no idea what
on Earth I am doing! Can anyone please offer guidance in my learning
process? I don't know how and what I should be spending my time learning
first and/or if I should focus my learning towards the skill areas I will
require to write my specific programmes, although I have no idea what these
are. I would like advise on finding some really good interactive(let you
know if your solution to an exercise is correct or not) and or video
tutorials that give you feedback on the solutions you write to exercises.

 Many thanks in advance for all your help, it will be much appreciated!



Cecilia Chavana-Bryant
DPhil Candidate - Remote sensing and tropical phenology
Environmental Change Institute
School of Geography and the Environment
University of Oxford
South Parks Road, Oxford, OX1 3QY
Web: http://www.eci.ox.ac.uk/teaching/doctoral/chavanabryantcecilia.php
Tel Direct: +44 (0)1865 275861
Fax: +44 (0)1865 275885
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] list comprehension, testing for multiple conditions

[Peter Otten]

Pete O'Connell wrote:

[Please don't to-post. Clip all text of previous posts except the portion 
relevant to your question]

> Hi. The next step for me to parse the file as I want to is to change
> lines that look like this:
> f 21/21/21 22/22/22 24/24/23 23/23/24
> into lines that look like this:
> f 21 22 23 24
> 
> Below is my terribly slow loop for doing this. 

I'd say you are not yet at the point where you should care for speed too 
much. Build the complete tool-chain with tests to verify correctness and 
then measure execution time to find the bottlenecks.

> Any suggestions about
> how to make this code more efficient would be greatly appreciated
> 
> 

> fileName = '/usr/home/poconnell/Desktop/objCube.obj'
> 
> with open(fileName) as lines:
> theGoodLines = [line.strip("\n") for line in lines if "vn" not in
> line and "vt" not in line and line != "\n"]
> 
> for i in range(len(theGoodLines)):
> if theGoodLines[i][0] == "f":
> aGoodLineAsList = theGoodLines[i].split(" ")
> theGoodLines[i] = aGoodLineAsList[0] + " " +
> aGoodLineAsList[1].split("/")[-1] + " " +
> aGoodLineAsList[2].split("/")[-1] + " " +
> aGoodLineAsList[3].split("/")[-1] + " " +
> aGoodLineAsList[4].split("/")[-1]
> 
> for anItem in theGoodLines:
> print anItem

If your sample data is representative you don't need most of the filtering:

fileName = ...
with open(fileName) as lines:
for line in lines:
if line.startswith("f "):
print " ".join(part.rpartition("/")[-1] for part in 
line.split())

___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] list comprehension, testing for multiple conditions

[eryksun]

On Wed, Aug 22, 2012 at 3:06 AM, Peter Otten <__pete...@web.de> wrote:
>
> wanted = [line.strip("\n") for line in lines
>   if "vn" not in line and "vt" not in line and line != "\n"]

Here's an equivalent expression with the negation factored out:

not ("vn" in line or "vt" in line or line == "\n")

http://en.wikipedia.org/wiki/De_Morgan%27s_laws

If you have a lot of tests all using the same operator (e.g. "in"),
you can use "any" (OR) or "all" (AND) with a generator expression:

vals = ["vn", "vt", "vu", "vv", "vw", "vx", "vy", "vz"]
wanted = [line for line in lines if not any(v in line for v in vals)]
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor

[Tutor] Hello Python Tutor - help please!

[Cecilia Chavana-Bryant]

Dear all,

I am just returning to my doctoral studies after a 7-month medical leave and 
desperately trying to catch up for lost time. I am COMPLETELY new to 
programming, well, I did try learning C for 3 weeks 3 yrs ago (with very little 
success) but had to stop and then spent 2 years in the Amazon climbing trees 
(lots more enjoyable than learning to programme!) and collecting loads of field 
data that I now need to post-process and analyse. By the way, the 3 weeks I 
spent trying to learn C really ended up being spent trying to get to grips with 
using a terminal for the first time in my life.

Since getting back to work, I was advised to try learning Python instead of C 
as it is a much easier first language to learn. I have been trying, but again, 
to not great success. I started following "A Primer on Scientific programming 
with Python" but I kept getting lost and stuck, specially on the exercises. I 
have also been advised that I should not try to learn programming by following 
guides but by trying to write the programmes I need to analyse my data. 
Although I can understand the logic behind this last bit of advise (it gives 
context and direction to the learning process) I have also gotten stuck trying 
this approach as "I do not know how to programme!". Thus, I was hoping that 
some of you can remember how you got started and point me towards any really 
good interactive learning guides/materials and/or have a good learning strategy 
for a complete beginner. I have searched the web and was overwhelmed by choice 
of tutorials and guides. I have skimmed through a couple of tutorials but then 
fail to see how all that relates to my own work and I get stuck with what seems 
like basic important concepts so I don't progress. I then think I should try to 
make some progress with my own data analysing and go back to trying to learn to 
write a programme for my specific needs and get stuck again because this 
requires more advanced skills then the basic programming concepts I have been 
reading about on the learning guides. So, I am now feeling VERY frustrated and 
have no idea what on Earth I am doing! Can anyone please offer guidance in my 
learning process? I don't know how and what I should be spending my time 
learning first and/or if I should focus my learning towards the skill areas I 
will require to write my specific programmes, although I have no idea what 
these are. I would like advise on finding some really good interactive(let you 
know if your solution to an exercise is correct or not) and or video tutorials 
that give you feedback on the solutions you write to exercises.

Many thanks in advance for all your help, it will be much appreciated!



Cecilia Chavana-Bryant
DPhil Candidate - Remote sensing and tropical phenology
Environmental Change Institute
School of Geography and the Environment
University of Oxford
South Parks Road, Oxford, OX1 3QY
Web: http://www.eci.ox.ac.uk/teaching/doctoral/chavanabryantcecilia.php
Tel Direct: +44 (0)1865 275861
Fax: +44 (0)1865 275885
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] list comprehension, testing for multiple conditions

[Pete O'Connell]

Hi. The next step for me to parse the file as I want to is to change
lines that look like this:
f 21/21/21 22/22/22 24/24/23 23/23/24
into lines that look like this:
f 21 22 23 24

Below is my terribly slow loop for doing this. Any suggestions about
how to make this code more efficient would be greatly appreciated


fileName = '/usr/home/poconnell/Desktop/objCube.obj'

with open(fileName) as lines:
theGoodLines = [line.strip("\n") for line in lines if "vn" not in
line and "vt" not in line and line != "\n"]

for i in range(len(theGoodLines)):
if theGoodLines[i][0] == "f":
aGoodLineAsList = theGoodLines[i].split(" ")
theGoodLines[i] = aGoodLineAsList[0] + " " +
aGoodLineAsList[1].split("/")[-1] + " " +
aGoodLineAsList[2].split("/")[-1] + " " +
aGoodLineAsList[3].split("/")[-1] + " " +
aGoodLineAsList[4].split("/")[-1]

for anItem in theGoodLines:
print anItem
##

Thanks!
Pete








On Wed, Aug 22, 2012 at 9:59 PM, Pete O'Connell  wrote:
> Thanks Peter. This looks like what I need:
>
> with open(fileName) as lines:
> wanted = [line.strip("\n") for line in lines if "vn" not in line
> and "vt" not in line and line != "\n"]
>
> Cheers
>
> And in response to Allan's suggestion. I can see using a generator in
> a situation where the if statements were more numerous and complex. I
> am sure that will come in handy.
>
> Thanks
>
>
> On Wed, Aug 22, 2012 at 7:06 PM, Peter Otten <__pete...@web.de> wrote:
>> Pete O'Connell wrote:
>>
>>> Hi I am trying to parse a text file and create a list of all the lines
>>> that don't include: "vn", "vt" or are empty. I want to make this as
>>> fast as possible because I will be parsing many files each containing
>>> thousands of lines. I though I would give list comprehensions a try.
>>> The last 3 lines of the code below have three list comprehensions that
>>> I would like to combine into 1 but I am not sure how to do that.
>>> Any tips would be greatly appreciated
>>>
>>> pete
>>>
>>> #start
>>> fileName = '/usr/home/poconnell/Desktop/objCube.obj'
>>> theFileOpened = open(fileName,'r')
>>> theTextAsList = theFileOpened.readlines()
>>
>> If you have a file with 1,000,000 lines you have now a list of 1,000,000
>> strings of which perhaps 1,000 match your criteria. You are squandering
>> memory. Rule of thumb: never use readlines(), iterate over the file
>> directly.
>>
>>> theTextAsListStripped = []
>>> for aLine in theTextAsList:
>>>
>>> theTextAsListStripped.append(aLine.strip("\n"))
>>>
>>> theTextAsListNoVn = [x for x in theTextAsListStripped if "vn" not in x]
>>> theTextAsListNoVnOrVt = [x for x in theTextAsListNoVn if "vt" not in x]
>>> theTextAsListNoVnOrVtOrEmptyLine = [x for x in theTextAsListNoVn if x !=
>>> ""]
>>
>> I think that should be
>>
>> theTextAsListNoVnOrVtOrEmptyLine = [x for x in theTextAsListNoVnOrVt if x !=
>> ""]
>>
>> You can combine the three if clauses or add them all to one list-comp:
>>
>> with open(filename) as lines:
>> wanted = [line.strip("\n") for line in lines
>>   if "vn" not in line and "vt" not in line and line != "\n"]
>>
>>
>> You can even have multiple if clauses in one list-comp (but that is rarely
>> used):
>>
>> with open(filename) as lines:
>> wanted = [line.strip("\n") for line
>>   if "vn" not in line
>>   if "vt" not in x
>>   if line != "\n"]
>>
>> While your problem is simple enough to combine all filters into one list-
>> comp some problems are not. You can then prevent the intermediate lists from
>> materializing by using generator expressions. The result minimizes memory
>> consumption, too, and should be (almost) as fast. For example:
>>
>> with open(filename) as lines:
>> # use gen-exps to remove empty and whitespace-only lines
>> stripped = (line.strip() for line in lines)
>> nonempty = (line for line in stripped if line)
>>
>> wanted = [line for line in nonempty
>>   if "vt" not in line and "vn" not in line]
>>
>>
>> ___
>> Tutor maillist  -  Tutor@python.org
>> To unsubscribe or change subscription options:
>> http://mail.python.org/mailman/listinfo/tutor
>
>
>
> --
> -



-- 
-
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] list comprehension, testing for multiple conditions

[Pete O'Connell]

Thanks Peter. This looks like what I need:

with open(fileName) as lines:
wanted = [line.strip("\n") for line in lines if "vn" not in line
and "vt" not in line and line != "\n"]

Cheers

And in response to Allan's suggestion. I can see using a generator in
a situation where the if statements were more numerous and complex. I
am sure that will come in handy.

Thanks


On Wed, Aug 22, 2012 at 7:06 PM, Peter Otten <__pete...@web.de> wrote:
> Pete O'Connell wrote:
>
>> Hi I am trying to parse a text file and create a list of all the lines
>> that don't include: "vn", "vt" or are empty. I want to make this as
>> fast as possible because I will be parsing many files each containing
>> thousands of lines. I though I would give list comprehensions a try.
>> The last 3 lines of the code below have three list comprehensions that
>> I would like to combine into 1 but I am not sure how to do that.
>> Any tips would be greatly appreciated
>>
>> pete
>>
>> #start
>> fileName = '/usr/home/poconnell/Desktop/objCube.obj'
>> theFileOpened = open(fileName,'r')
>> theTextAsList = theFileOpened.readlines()
>
> If you have a file with 1,000,000 lines you have now a list of 1,000,000
> strings of which perhaps 1,000 match your criteria. You are squandering
> memory. Rule of thumb: never use readlines(), iterate over the file
> directly.
>
>> theTextAsListStripped = []
>> for aLine in theTextAsList:
>>
>> theTextAsListStripped.append(aLine.strip("\n"))
>>
>> theTextAsListNoVn = [x for x in theTextAsListStripped if "vn" not in x]
>> theTextAsListNoVnOrVt = [x for x in theTextAsListNoVn if "vt" not in x]
>> theTextAsListNoVnOrVtOrEmptyLine = [x for x in theTextAsListNoVn if x !=
>> ""]
>
> I think that should be
>
> theTextAsListNoVnOrVtOrEmptyLine = [x for x in theTextAsListNoVnOrVt if x !=
> ""]
>
> You can combine the three if clauses or add them all to one list-comp:
>
> with open(filename) as lines:
> wanted = [line.strip("\n") for line in lines
>   if "vn" not in line and "vt" not in line and line != "\n"]
>
>
> You can even have multiple if clauses in one list-comp (but that is rarely
> used):
>
> with open(filename) as lines:
> wanted = [line.strip("\n") for line
>   if "vn" not in line
>   if "vt" not in x
>   if line != "\n"]
>
> While your problem is simple enough to combine all filters into one list-
> comp some problems are not. You can then prevent the intermediate lists from
> materializing by using generator expressions. The result minimizes memory
> consumption, too, and should be (almost) as fast. For example:
>
> with open(filename) as lines:
> # use gen-exps to remove empty and whitespace-only lines
> stripped = (line.strip() for line in lines)
> nonempty = (line for line in stripped if line)
>
> wanted = [line for line in nonempty
>   if "vt" not in line and "vn" not in line]
>
>
> ___
> Tutor maillist  -  Tutor@python.org
> To unsubscribe or change subscription options:
> http://mail.python.org/mailman/listinfo/tutor



-- 
-
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] list comprehension, testing for multiple conditions

[Pete O'Connell]

What a great mailing list!
Thanks for all the responses.
I have a few questions, though, first in regards to Puneeth's code. He
writes to use:

>theTextAsListNoVnOrVtOrEmptyLine = [x for x in theTextAsListStripped if "vn" 
>not in x  and "vt" not in x and x!= ""]

It works but what I don't understand about this line is why the ands
are nor ors ("or" doesn't work even though I would have expected it
to)
I am sure I will have a few more questions over the next couple days
as I work my way through the responses.

Thanks
Pete



On Wed, Aug 22, 2012 at 6:23 PM, Puneeth Chaganti  wrote:
> On Wed, Aug 22, 2012 at 11:35 AM, Pete O'Connell
>  wrote:
>> Hi I am trying to parse a text file and create a list of all the lines
>> that don't include: "vn", "vt" or are empty. I want to make this as
>> fast as possible because I will be parsing many files each containing
>> thousands of lines. I though I would give list comprehensions a try.
>> The last 3 lines of the code below have three list comprehensions that
>> I would like to combine into 1 but I am not sure how to do that.
>> Any tips would be greatly appreciated
>>
>> pete
>>
>> #start
>> fileName = '/usr/home/poconnell/Desktop/objCube.obj'
>> theFileOpened = open(fileName,'r')
>> theTextAsList = theFileOpened.readlines()
>>
>> theTextAsListStripped = []
>> for aLine in theTextAsList:
>>
>> theTextAsListStripped.append(aLine.strip("\n"))
>>
>> theTextAsListNoVn = [x for x in theTextAsListStripped if "vn" not in x]
>> theTextAsListNoVnOrVt = [x for x in theTextAsListNoVn if "vt" not in x]
>> theTextAsListNoVnOrVtOrEmptyLine = [x for x in theTextAsListNoVn if x != ""]
>
> Something like this should work :
>
> theTextAsListNoVnOrVtOrEmptyLine = [x for x in theTextAsListStripped
> if "vn" not in x  and "vt" not in x and x!= ""]
>
> HTH,
> Puneeth



-- 
-
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] subprocess.Popen help

[David Abbott]

On Tue, Aug 21, 2012 at 9:39 PM, Ray Jones  wrote:
>
> Does anyone know of a link to a really good tutorial that would help me
> with subprocess.Popen? a tutorial that uses really small words and more
> examples than explanation? After 15 years of scripting, I'm ashamed to
> say that I'm still not all that familiar with input, output, pipes, etc.
> much beyond a simple 'ls | ws -l' or  &2>/dev/null scenarios. The
> docs for Popen have left me completely boggled, and I'm not seeing much
> available on Google search. Any suggestions?
>
> Thanks.
>
>
> Ray
Hi Ray,
http://www.doughellmann.com/PyMOTW/subprocess/

David
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] subprocess.Popen help

[Andreas Perstinger]

On 22.08.2012 03:39, Ray Jones wrote:

Does anyone know of a link to a really good tutorial that would help me
with subprocess.Popen? a tutorial that uses really small words and more
examples than explanation? After 15 years of scripting, I'm ashamed to
say that I'm still not all that familiar with input, output, pipes, etc.
much beyond a simple 'ls | ws -l' or  &2>/dev/null scenarios. The
docs for Popen have left me completely boggled, and I'm not seeing much
available on Google search. Any suggestions?


What about this tutorial:
http://jimmyg.org/blog/2009/working-with-python-subprocess.html

or Doug Hellmann's PyMOTW page about subprocess:
http://www.doughellmann.com/PyMOTW/subprocess/index.html

Bye, Andreas
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] list comprehension, testing for multiple conditions

[Mark Lawrence]

On 22/08/2012 07:29, Johann Spies wrote:

#start
import re
exclude = re.compile('vn|vt|^$|^#')
fileName = '/tmp/x'
theFileOpened = open(fileName,'r')
theTextAsList = theFileOpened.readlines()

theTextAsListStripped = []
for aLine in theTextAsList:

 theTextAsListStripped.append(aLine.strip("\n"))

theTextAsListNoVn = [x for x in theTextAsListStripped if not
re.search(exclude,x)]

print theTextAsListNoVn



Regards
Johann



___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor



Please no, not a regex for something this simple!!!


--
Cheers.

Mark Lawrence.

___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] list comprehension, testing for multiple conditions

[Alan Gauld]

On 22/08/12 08:06, Peter Otten wrote:


You can even have multiple if clauses in one list-comp (but that is rarely
used):

with open(filename) as lines:
 wanted = [line.strip("\n") for line
   if "vn" not in line
   if "vt" not in x
   if line != "\n"]

While your problem is simple enough to combine all filters into one list-
comp some problems are not. You can then prevent the intermediate lists from
materializing by using generator expressions. The result minimizes memory
consumption, too, and should be (almost) as fast. For example:

with open(filename) as lines:
 # use gen-exps to remove empty and whitespace-only lines
 stripped = (line.strip() for line in lines)
 nonempty = (line for line in stripped if line)

 wanted = [line for line in nonempty
   if "vt" not in line and "vn" not in line]


Another option using generators is to roll your own. This would be my 
recomendation for really complex filtering:


def myFilterGenerator(aFile):
for line in aFile:
if 'vn' not in line:   # demo "complexity" not efficiency!
   if 'vt' not in line:
   if '\n' != line:
  yield line.strip()

with open filename as myFile:
result = [line for line in myFilterGenerator(myFile)]


But in this specific case the filter is simple enough that the
list comp from Peter is probably the best solution.

HTH

--
Alan G
Author of the Learn to Program web site
http://www.alan-g.me.uk/

___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] list comprehension, testing for multiple conditions

[Peter Otten]

Pete O'Connell wrote:

> Hi I am trying to parse a text file and create a list of all the lines
> that don't include: "vn", "vt" or are empty. I want to make this as
> fast as possible because I will be parsing many files each containing
> thousands of lines. I though I would give list comprehensions a try.
> The last 3 lines of the code below have three list comprehensions that
> I would like to combine into 1 but I am not sure how to do that.
> Any tips would be greatly appreciated
> 
> pete
> 
> #start
> fileName = '/usr/home/poconnell/Desktop/objCube.obj'
> theFileOpened = open(fileName,'r')
> theTextAsList = theFileOpened.readlines()

If you have a file with 1,000,000 lines you have now a list of 1,000,000
strings of which perhaps 1,000 match your criteria. You are squandering 
memory. Rule of thumb: never use readlines(), iterate over the file 
directly.

> theTextAsListStripped = []
> for aLine in theTextAsList:
> 
> theTextAsListStripped.append(aLine.strip("\n"))
> 
> theTextAsListNoVn = [x for x in theTextAsListStripped if "vn" not in x]
> theTextAsListNoVnOrVt = [x for x in theTextAsListNoVn if "vt" not in x]
> theTextAsListNoVnOrVtOrEmptyLine = [x for x in theTextAsListNoVn if x !=
> ""]

I think that should be

theTextAsListNoVnOrVtOrEmptyLine = [x for x in theTextAsListNoVnOrVt if x != 
""]

You can combine the three if clauses or add them all to one list-comp:

with open(filename) as lines:
wanted = [line.strip("\n") for line in lines
  if "vn" not in line and "vt" not in line and line != "\n"]


You can even have multiple if clauses in one list-comp (but that is rarely 
used):

with open(filename) as lines:
wanted = [line.strip("\n") for line 
  if "vn" not in line
  if "vt" not in x 
  if line != "\n"]

While your problem is simple enough to combine all filters into one list-
comp some problems are not. You can then prevent the intermediate lists from 
materializing by using generator expressions. The result minimizes memory 
consumption, too, and should be (almost) as fast. For example:

with open(filename) as lines:
# use gen-exps to remove empty and whitespace-only lines
stripped = (line.strip() for line in lines)
nonempty = (line for line in stripped if line)

wanted = [line for line in nonempty 
  if "vt" not in line and "vn" not in line]


___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor