Re: [Tutor] list comprehension, testing for multiple conditions
Pete O'Connell wrote:
> Hi, I have tried to simplify things and am running into a bit of trouble.
> What i am really trying to do is: Keep all the lines starting with "v "
> and then delete those lines whose modulus 5 don't equal zero
>
> I have written it like this which seems to take a really long time (a
> couple of minutes when iteration over a folder with 200 files to parse)
> #
> with open(theFilePath) as lines:
> #keep only the lines beginning with "v " (this works)
> theGoodLines = [line.strip("\n") for line in lines if "v " ==
> line[0:2]]
> theLinesAsListSubset = []
> for i in range(len(theGoodLines)):
> nuke.tprint(i)
> if i%5 != 0:
> continue
> elif i%5 == 0:
> theLinesAsListSubset.append(theGoodLines[i])
>
>
> I think it would be better to include the modulud test within the original
> list comprehension but I am not sure how to access the index of "line":
> #something like this is a sketch of what I mean (I know it's wrong)
> theGoodLines = [line.strip("\n") for line in lines if "v " ==
> line[0:2] and line.getIndex() % 5 == 0]
>
>
> Do I need enumerate for this maybe?
With enumerate() you need two steps, but you can use a generator expression
(which doesn't materialize the list as you might remember):
# low memory consumption
good = (line for line in lines if line.startswith("v "))
every_fifth = [line.strip("\n") for index, line in enumerate(prefixed)
if not index % 5]
Alternatively you can build the good list and apply slicing:
# simple/fast
good = [line.strip("\n") for line in lines if line.startswith("v ")]
every_fifth = good[::5]
Finally there's itertools.islice() which helps you combine the advantages of
both:
# low memory consumption, fast
good = (line.strip("\n") for line in lines if line.startswith("v "))
every_fifths= list(itertools.islice(good, 0, None, 5))
___
Tutor maillist - Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] list comprehension, testing for multiple conditions
(you replied off-list, so I'm cc'ing the list here, to keep it public)
On 08/23/2012 10:42 PM, Pete O'Connell wrote:
> On Fri, Aug 24, 2012 at 1:39 PM, Dave Angel wrote:
>
>> On 08/23/2012 09:11 PM, Pete O'Connell wrote:
>>> Hi, I have tried to simplify things and am running into a bit of trouble.
>>> What i am really trying to do is: Keep all the lines starting with "v "
>> and
>>> then delete those lines whose modulus 5 don't equal zero
>>>
>>> I have written it like this which seems to take a really long time (a
>>> couple of minutes when iteration over a folder with 200 files to parse)
>>> #
>>> with open(theFilePath) as lines:
>>> #keep only the lines beginning with "v " (this works)
>>> theGoodLines = [line.strip("\n") for line in lines if "v " ==
>>> line[0:2]]
>>
>> Better to use startswith(), since short lines will cause the if
>> expression above to blow up.
>>
> Thanks that looks much safer.
Sorry about that. Eryksun corrected me on that. Your present code
won't blow up, but I'd still prefer startswith().
>
>>> theLinesAsListSubset = []
>>> for i in range(len(theGoodLines)):
>>
>> When you see a line like this, it's usually clearer to do:
>> for i, line in enumerate(theGoodLines):
>>> nuke.tprint(i)
>>> if i%5 != 0:
>>> continue
>>> elif i%5 == 0:
>>> theLinesAsListSubset.append(theGoodLines[i])
>> It's confusing to me whether you meant to keep only one of every 5 lines
>> of the filtered input, or to keep only those lines of the filtered input
>> that came from the appropriate indices of the original data. You need a
>> more precise spec before you can safely combine the two loops. (You may
>> have it precise in your head; I'm just saying it isn't clear to me)
>>
> Sorry that wasn't clear. I want to keep every fifth line.
Fifth of which list? The one you start with, or the one you get after
throwing out the lines that don't begin with v ? You have presently
coded the latter, and if that's what you want, I can't see any
reasonable way to make it a single list comprehension.
>
>>
>>>
>>>
>>> I think it would be better to include the modulud test within the
>> original
>>> list comprehension but I am not sure how to access the index of "line":
>>> #something like this is a sketch of what I mean (I know it's wrong)
>>> theGoodLines = [line.strip("\n") for line in lines if "v " ==
>>> line[0:2] and line.getIndex() % 5 == 0]
>>>
>>>
>>> Do I need enumerate for this maybe?
>> Good call. Question is whether to do the enumerate on the original
>> list, or on the list you get after. That decision would be based on the
>> question above.
>>
>> I have noticed that part of the slowness comes from the feedback I am
> getting on the command line with print statements. When I streamline those
> it is much faster.
> It is useable even in its current state, still out of curiosity and for my
> python understanding, it would be nice to know if it is possible to write
> it all within one list comprehension.
> Thanks
> Pete
>
>>
--
DaveA
___
Tutor maillist - Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] list comprehension, testing for multiple conditions
On 08/23/2012 10:34 PM, eryksun wrote:
> On Thu, Aug 23, 2012 at 9:39 PM, Dave Angel wrote:
>>
>>> theGoodLines = [line.strip("\n") for line in lines if "v " ==
>>> line[0:2]]
>>
>> Better to use startswith(), since short lines will cause the if
>> expression above to blow up.
>
> A slice won't blow up.
You're right of course. For some reason I was looking at it as though
it were simple subscripting, which can blow up (once the newline has
been discarded, there might not be a [0] item.
> At worst you get an empty string. But the slice
> does create a temporary string, and subsequently the expression uses a
> high-level compare.
>
> startswith is more efficient and flexible in CPython. It does a
> low-level memory compare (memcmp). You can pass it a single string or
> a tuple of strings to match against, and you can set a start/stop
> position.
>
>
--
DaveA
___
Tutor maillist - Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] list comprehension, testing for multiple conditions
On Thu, Aug 23, 2012 at 9:39 PM, Dave Angel wrote:
>
>> theGoodLines = [line.strip("\n") for line in lines if "v " ==
>> line[0:2]]
>
> Better to use startswith(), since short lines will cause the if
> expression above to blow up.
A slice won't blow up. At worst you get an empty string. But the slice
does create a temporary string, and subsequently the expression uses a
high-level compare.
startswith is more efficient and flexible in CPython. It does a
low-level memory compare (memcmp). You can pass it a single string or
a tuple of strings to match against, and you can set a start/stop
position.
___
Tutor maillist - Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] list comprehension, testing for multiple conditions
On 08/23/2012 09:11 PM, Pete O'Connell wrote:
> Hi, I have tried to simplify things and am running into a bit of trouble.
> What i am really trying to do is: Keep all the lines starting with "v " and
> then delete those lines whose modulus 5 don't equal zero
>
> I have written it like this which seems to take a really long time (a
> couple of minutes when iteration over a folder with 200 files to parse)
> #
> with open(theFilePath) as lines:
> #keep only the lines beginning with "v " (this works)
> theGoodLines = [line.strip("\n") for line in lines if "v " ==
> line[0:2]]
Better to use startswith(), since short lines will cause the if
expression above to blow up.
> theLinesAsListSubset = []
> for i in range(len(theGoodLines)):
When you see a line like this, it's usually clearer to do:
for i, line in enumerate(theGoodLines):
> nuke.tprint(i)
> if i%5 != 0:
> continue
> elif i%5 == 0:
> theLinesAsListSubset.append(theGoodLines[i])
It's confusing to me whether you meant to keep only one of every 5 lines
of the filtered input, or to keep only those lines of the filtered input
that came from the appropriate indices of the original data. You need a
more precise spec before you can safely combine the two loops. (You may
have it precise in your head; I'm just saying it isn't clear to me)
>
>
> I think it would be better to include the modulud test within the original
> list comprehension but I am not sure how to access the index of "line":
> #something like this is a sketch of what I mean (I know it's wrong)
> theGoodLines = [line.strip("\n") for line in lines if "v " ==
> line[0:2] and line.getIndex() % 5 == 0]
>
>
> Do I need enumerate for this maybe?
Good call. Question is whether to do the enumerate on the original
list, or on the list you get after. That decision would be based on the
question above.
To be honest, when I'm doing a non-trivial list comprehension, i tend to
write it as a for loop first, get it correct, then reconsider if it can
(& if it should be) rewritten as a comprehension.
--
DaveA
___
Tutor maillist - Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] list comprehension, testing for multiple conditions
Hi, I have tried to simplify things and am running into a bit of trouble.
What i am really trying to do is: Keep all the lines starting with "v " and
then delete those lines whose modulus 5 don't equal zero
I have written it like this which seems to take a really long time (a
couple of minutes when iteration over a folder with 200 files to parse)
#
with open(theFilePath) as lines:
#keep only the lines beginning with "v " (this works)
theGoodLines = [line.strip("\n") for line in lines if "v " ==
line[0:2]]
theLinesAsListSubset = []
for i in range(len(theGoodLines)):
nuke.tprint(i)
if i%5 != 0:
continue
elif i%5 == 0:
theLinesAsListSubset.append(theGoodLines[i])
I think it would be better to include the modulud test within the original
list comprehension but I am not sure how to access the index of "line":
#something like this is a sketch of what I mean (I know it's wrong)
theGoodLines = [line.strip("\n") for line in lines if "v " ==
line[0:2] and line.getIndex() % 5 == 0]
Do I need enumerate for this maybe?
Thanks
Pete
___
Tutor maillist - Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] list comprehension, testing for multiple conditions
On Thu, Aug 23, 2012 at 2:57 AM, Pete O'Connell wrote: > On Thu, Aug 23, 2012 at 4:16 AM, Alan Gauld wrote: > >> theTextAsListNoVnOrVtOrEmptyLine = [x for x in theTextAsListStripped >> if "vn" not in x >> if "vt" not in x >> if x!= ""] >> >> It's slightly more verbose but it makes the rules more explicit, IMHO. > > I agree, it seems easier to read when written on multiple lines. I'll > do it that way, > Thanks > Pete You can look at the option of writing a function to do the selection part. If the function is named well and also corresponds to some domain operation or concept it can be very useful. def select(p): if p == "": return False return "vt" In p or "vn" in p .. selectedLines = [ x for x in TextAsListStripped if select(x) ] .. Asokan Pichai If a language is designed for non-programmers, soon only non-programs get written in it. --- Anonymouse ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] list comprehension, testing for multiple conditions
Ok thanks for the advice everyone. Cheers Pete On Thu, Aug 23, 2012 at 10:58 AM, Jerry Hill wrote: > On Wed, Aug 22, 2012 at 5:23 PM, Pete O'Connell > wrote: >> OK maybe I am wrong about it being slow (I thought for loops were >> slower than lis comprehensions). But I do know I need it to be as fast >> as possible if I need to run it on a thousand files each with hundreds >> of thousands of lines > > You're quite likely wrong about that. The overall runtime of your > code is very likely to end up dominated by I/O with the hard drive, > not the microsecond differences in how you process each line. The way > to know for sure is to write code that is easy to read and does the > correct thing. Then run the code an see if it's too slow for your > purposes. If it is, profile the code and see where it's spending all > of its time. Only then can you actually optimize the right places. > > Once you've identified where your code needs to be faster, you're > likely to get more mileage out of a clever algorithm change than a > micro-optimization to your filtering. Or maybe you'll discover that it > really is worthwhile to optimize your filtering, because it's inside a > tight loop, and that's where all of your time is spent. There are a > lot of tricks you can use to slightly speed up your python code, but > none of them are worth doing until you know for sure that you're > optimizing in the right place. And if it turns out that the code's > runtime is fine as first written, you don't have to waste your time > doing lots of work for little gain. > > -- > Jerry > ___ > Tutor maillist - Tutor@python.org > To unsubscribe or change subscription options: > http://mail.python.org/mailman/listinfo/tutor -- - ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] list comprehension, testing for multiple conditions
On Wed, Aug 22, 2012 at 5:23 PM, Pete O'Connell wrote: > OK maybe I am wrong about it being slow (I thought for loops were > slower than lis comprehensions). But I do know I need it to be as fast > as possible if I need to run it on a thousand files each with hundreds > of thousands of lines You're quite likely wrong about that. The overall runtime of your code is very likely to end up dominated by I/O with the hard drive, not the microsecond differences in how you process each line. The way to know for sure is to write code that is easy to read and does the correct thing. Then run the code an see if it's too slow for your purposes. If it is, profile the code and see where it's spending all of its time. Only then can you actually optimize the right places. Once you've identified where your code needs to be faster, you're likely to get more mileage out of a clever algorithm change than a micro-optimization to your filtering. Or maybe you'll discover that it really is worthwhile to optimize your filtering, because it's inside a tight loop, and that's where all of your time is spent. There are a lot of tricks you can use to slightly speed up your python code, but none of them are worth doing until you know for sure that you're optimizing in the right place. And if it turns out that the code's runtime is fine as first written, you don't have to waste your time doing lots of work for little gain. -- Jerry ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] list comprehension, testing for multiple conditions
On 22/08/12 22:23, Pete O'Connell wrote: What makes you say it is "terribly slow"? Perhaps it is as fast as it could be under the circumstances. (Maybe it takes a long time because you have a lot of data, not because it is slow.) OK maybe I am wrong about it being slow (I thought for loops were slower than lis comprehensions). For loops are slightly slower than list comprehensions but depending on what is happening inside the loop/comprehension the difference may not be significant. > But I do know I need it to be as fast > as possible if I need to run it on a thousand files each with hundreds > of thousands of lines If you need it as "fast as possible" you need to run it on a supercomputer with a program in assembler after many hours of fine tuning. But that will cost a lot of money and you probably don't really need it. For example how often do you need to run this process? Is it a one off - set it running over lunch and it will likely be done by the time you get back. Is it an hourly event? Set it running in the background while you carry on processing the results of the previous run. Steven's point is that usually absolute speed is not as important as people think. Modern computers are absurdly fast. When I started using Linux it took about 2 hours to build the kernel. Now I can build it in about 2 minutes! Processing a thousand files is largely limited by the speed of your hard drive rather than the speed of the processing on the CPU. HTH -- Alan G Author of the Learn to Program web site http://www.alan-g.me.uk/ ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] list comprehension, testing for multiple conditions
Note uits not just that its on multiple lines, its the fact it has three distinct if statements. All of them must be satisfied to be included in the comprehension. You could do it on 3 lines with one if statement: theTextAsListNoVnOrVtOrEmptyLine = [x for x in theTextAsListStripped if "vn" not in x and "vt" not in x and x!= ""] But I still think the three if statements are clearer. However, if you need to use 'or' instead of 'and' then you need to go back to a compound statement but multi-line layout still aids legibility. Alan Gauld Author of the Learn To Program website http://www.alan-g.me.uk/ > > From: Pete O'Connell >To: Alan Gauld ; tutor@python.org >Sent: Wednesday, 22 August 2012, 22:27 >Subject: Re: [Tutor] list comprehension, testing for multiple conditions > >On Thu, Aug 23, 2012 at 4:16 AM, Alan Gauld wrote: > >> theTextAsListNoVnOrVtOrEmptyLine = [x for x in theTextAsListStripped >> if "vn" not in x >> if "vt" not in x >> if x!= ""] >> >> It's slightly more verbose but it makes the rules more explicit, IMHO. > >I agree, it seems easier to read when written on multiple lines. I'll >do it that way, >Thanks >Pete > > >___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] list comprehension, testing for multiple conditions
On 22/08/2012 22:23, Pete O'Connell wrote: On Thu, Aug 23, 2012 at 2:53 AM, Steven D'Aprano wrote: Further to Steven's sound advice take a look at this http://wiki.python.org/moin/PythonSpeed/PerformanceTips IMHO the part on profiling is particulary useful. -- Cheers. Mark Lawrence. ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] list comprehension, testing for multiple conditions
On Thu, Aug 23, 2012 at 4:16 AM, Alan Gauld wrote: > theTextAsListNoVnOrVtOrEmptyLine = [x for x in theTextAsListStripped > if "vn" not in x > if "vt" not in x > if x!= ""] > > It's slightly more verbose but it makes the rules more explicit, IMHO. I agree, it seems easier to read when written on multiple lines. I'll do it that way, Thanks Pete ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] list comprehension, testing for multiple conditions
On Thu, Aug 23, 2012 at 2:53 AM, Steven D'Aprano wrote:
> On 22/08/12 20:28, Pete O'Connell wrote:
>>
>> Hi. The next step for me to parse the file as I want to is to change
>> lines that look like this:
>> f 21/21/21 22/22/22 24/24/23 23/23/24
>> into lines that look like this:
>> f 21 22 23 24
>
>
> In English, what is the rule you are applying here? My guess is:
>
> "Given three numbers separated by slashes, ignore the first two numbers
> and keep the third."
>
> E.g. "17/25/97" => 97.
>
> Am I close?
Hi Steve, yes that is correct
>
>
>
>> Below is my terribly slow loop for doing this. Any suggestions about
>> how to make this code more efficient would be greatly appreciated
>
>
> What makes you say it is "terribly slow"? Perhaps it is as fast as it
> could be under the circumstances. (Maybe it takes a long time because
> you have a lot of data, not because it is slow.)
OK maybe I am wrong about it being slow (I thought for loops were
slower than lis comprehensions). But I do know I need it to be as fast
as possible if I need to run it on a thousand files each with hundreds
of thousands of lines
>
> The first lesson of programming is not to be too concerned about speed
> until your program is correct.
>
> Like most such guidelines, this is not entirely true -- you don't want
> to write code which is unnecessarily slow. But the question you should
> be asking is, "is it fast enough?" rather than "is it fast?".
>
> Also, the sad truth is that Python tends to be slower than some other
> languages. (It's also faster than some other languages too.) But the
> general process is:
>
> 1) write something that works correctly;
>
> 2) if it is too slow, try to speed it up in Python;
>
> 3) if that's still too slow, try using something like cython or PyPy
>
> 4) if all else fails, now that you have a working prototype, re-write
> it again in C, Java, Lisp or Haskell.
>
> Once they see how much more work is involved in writing fast C code,
> most people decide that "fast enough" is fast enough :)
OK I will keep it as is and see if I can live with it.
Thanks
Pete
>
>
>
>> with open(fileName) as lines:
>> theGoodLines = [line.strip("\n") for line in lines if "vn" not in
>> line and "vt" not in line and line != "\n"]
>
>
> I prefer to write code in chains of filters.
>
> with open(fileName) as lines:
> # get rid of leading and trailing whitespace, including newlines
> lines = (line.strip() for line in lines)
> # ignore blanks
> lines = (line in lines if line)
> # ignore lines containing "vn" or "vt"
> theGoodLines = [line in lines if not ("vn" in line or "vt" in line)]
>
> Note that only the last step is a list comprehension using [ ], the others
> are generator expressions using ( ) instead.
>
> Will the above be faster than your version? I have no idea. But I think it
> is more readable and understandable. Some people might disagree.
>
>
>
>> for i in range(len(theGoodLines)):
>> if theGoodLines[i][0] == "f":
>> aGoodLineAsList = theGoodLines[i].split(" ")
>> theGoodLines[i] = aGoodLineAsList[0] + " " +
>> aGoodLineAsList[1].split("/")[-1] + " " +
>> aGoodLineAsList[2].split("/")[-1] + " " +
>> aGoodLineAsList[3].split("/")[-1] + " " +
>> aGoodLineAsList[4].split("/")[-1]
>
>
>
> Start with a helper function:
>
> def extract_last_item(term):
> """Extract the item from a term like a/b/c"""
> return term.split("/")[-1]
>
>
> for i, line in enumerate(theGoodLines):
> if line[0] == "f":
> terms = line.split()
> theGoodLines[i] = " ".join([extract_last_item(t) for t in terms])
>
>
>
> See how you go with that.
>
>
>
> --
> Steven
>
> ___
> Tutor maillist - Tutor@python.org
> To unsubscribe or change subscription options:
> http://mail.python.org/mailman/listinfo/tutor
--
-
___
Tutor maillist - Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] list comprehension, testing for multiple conditions
Thanks eryksun, that is a very clear example.
Cheers
pete
On Wed, Aug 22, 2012 at 11:16 PM, eryksun wrote:
> On Wed, Aug 22, 2012 at 3:06 AM, Peter Otten <__pete...@web.de> wrote:
>>
>> wanted = [line.strip("\n") for line in lines
>> if "vn" not in line and "vt" not in line and line != "\n"]
>
> Here's an equivalent expression with the negation factored out:
>
> not ("vn" in line or "vt" in line or line == "\n")
>
> http://en.wikipedia.org/wiki/De_Morgan%27s_laws
>
> If you have a lot of tests all using the same operator (e.g. "in"),
> you can use "any" (OR) or "all" (AND) with a generator expression:
>
> vals = ["vn", "vt", "vu", "vv", "vw", "vx", "vy", "vz"]
> wanted = [line for line in lines if not any(v in line for v in vals)]
> ___
> Tutor maillist - Tutor@python.org
> To unsubscribe or change subscription options:
> http://mail.python.org/mailman/listinfo/tutor
--
-
___
Tutor maillist - Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] list comprehension, testing for multiple conditions
On 22/08/12 10:51, Pete O'Connell wrote: theTextAsListNoVnOrVtOrEmptyLine = [x for x in theTextAsListStripped if "vn" not in x and "vt" not in x and x!= ""] It works but what I don't understand about this line is why the ands are not ors Because 'or' would include x if any one of the conditions was true. For example if 'vt' existed but not 'vn' then the line would be included because vn was not in the line, even though vt was. We are all assuming that you want to exclude the line if any of the phrases is present. Thats why I actually prefer the multiple if version that Peter posted: theTextAsListNoVnOrVtOrEmptyLine = [x for x in theTextAsListStripped if "vn" not in x if "vt" not in x if x!= ""] It's slightly more verbose but it makes the rules more explicit, IMHO. -- Alan G Author of the Learn to Program web site http://www.alan-g.me.uk/ ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] list comprehension, testing for multiple conditions
On 22/08/12 20:28, Pete O'Connell wrote:
Hi. The next step for me to parse the file as I want to is to change
lines that look like this:
f 21/21/21 22/22/22 24/24/23 23/23/24
into lines that look like this:
f 21 22 23 24
In English, what is the rule you are applying here? My guess is:
"Given three numbers separated by slashes, ignore the first two numbers
and keep the third."
E.g. "17/25/97" => 97.
Am I close?
Below is my terribly slow loop for doing this. Any suggestions about
how to make this code more efficient would be greatly appreciated
What makes you say it is "terribly slow"? Perhaps it is as fast as it
could be under the circumstances. (Maybe it takes a long time because
you have a lot of data, not because it is slow.)
The first lesson of programming is not to be too concerned about speed
until your program is correct.
Like most such guidelines, this is not entirely true -- you don't want
to write code which is unnecessarily slow. But the question you should
be asking is, "is it fast enough?" rather than "is it fast?".
Also, the sad truth is that Python tends to be slower than some other
languages. (It's also faster than some other languages too.) But the
general process is:
1) write something that works correctly;
2) if it is too slow, try to speed it up in Python;
3) if that's still too slow, try using something like cython or PyPy
4) if all else fails, now that you have a working prototype, re-write
it again in C, Java, Lisp or Haskell.
Once they see how much more work is involved in writing fast C code,
most people decide that "fast enough" is fast enough :)
with open(fileName) as lines:
theGoodLines = [line.strip("\n") for line in lines if "vn" not in
line and "vt" not in line and line != "\n"]
I prefer to write code in chains of filters.
with open(fileName) as lines:
# get rid of leading and trailing whitespace, including newlines
lines = (line.strip() for line in lines)
# ignore blanks
lines = (line in lines if line)
# ignore lines containing "vn" or "vt"
theGoodLines = [line in lines if not ("vn" in line or "vt" in line)]
Note that only the last step is a list comprehension using [ ], the others
are generator expressions using ( ) instead.
Will the above be faster than your version? I have no idea. But I think it
is more readable and understandable. Some people might disagree.
for i in range(len(theGoodLines)):
if theGoodLines[i][0] == "f":
aGoodLineAsList = theGoodLines[i].split(" ")
theGoodLines[i] = aGoodLineAsList[0] + " " +
aGoodLineAsList[1].split("/")[-1] + " " +
aGoodLineAsList[2].split("/")[-1] + " " +
aGoodLineAsList[3].split("/")[-1] + " " +
aGoodLineAsList[4].split("/")[-1]
Start with a helper function:
def extract_last_item(term):
"""Extract the item from a term like a/b/c"""
return term.split("/")[-1]
for i, line in enumerate(theGoodLines):
if line[0] == "f":
terms = line.split()
theGoodLines[i] = " ".join([extract_last_item(t) for t in terms])
See how you go with that.
--
Steven
___
Tutor maillist - Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] list comprehension, testing for multiple conditions
Pete O'Connell wrote:
[Please don't to-post. Clip all text of previous posts except the portion
relevant to your question]
> Hi. The next step for me to parse the file as I want to is to change
> lines that look like this:
> f 21/21/21 22/22/22 24/24/23 23/23/24
> into lines that look like this:
> f 21 22 23 24
>
> Below is my terribly slow loop for doing this.
I'd say you are not yet at the point where you should care for speed too
much. Build the complete tool-chain with tests to verify correctness and
then measure execution time to find the bottlenecks.
> Any suggestions about
> how to make this code more efficient would be greatly appreciated
>
>
> fileName = '/usr/home/poconnell/Desktop/objCube.obj'
>
> with open(fileName) as lines:
> theGoodLines = [line.strip("\n") for line in lines if "vn" not in
> line and "vt" not in line and line != "\n"]
>
> for i in range(len(theGoodLines)):
> if theGoodLines[i][0] == "f":
> aGoodLineAsList = theGoodLines[i].split(" ")
> theGoodLines[i] = aGoodLineAsList[0] + " " +
> aGoodLineAsList[1].split("/")[-1] + " " +
> aGoodLineAsList[2].split("/")[-1] + " " +
> aGoodLineAsList[3].split("/")[-1] + " " +
> aGoodLineAsList[4].split("/")[-1]
>
> for anItem in theGoodLines:
> print anItem
If your sample data is representative you don't need most of the filtering:
fileName = ...
with open(fileName) as lines:
for line in lines:
if line.startswith("f "):
print " ".join(part.rpartition("/")[-1] for part in
line.split())
___
Tutor maillist - Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] list comprehension, testing for multiple conditions
On Wed, Aug 22, 2012 at 3:06 AM, Peter Otten <__pete...@web.de> wrote:
>
> wanted = [line.strip("\n") for line in lines
> if "vn" not in line and "vt" not in line and line != "\n"]
Here's an equivalent expression with the negation factored out:
not ("vn" in line or "vt" in line or line == "\n")
http://en.wikipedia.org/wiki/De_Morgan%27s_laws
If you have a lot of tests all using the same operator (e.g. "in"),
you can use "any" (OR) or "all" (AND) with a generator expression:
vals = ["vn", "vt", "vu", "vv", "vw", "vx", "vy", "vz"]
wanted = [line for line in lines if not any(v in line for v in vals)]
___
Tutor maillist - Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] list comprehension, testing for multiple conditions
Hi. The next step for me to parse the file as I want to is to change
lines that look like this:
f 21/21/21 22/22/22 24/24/23 23/23/24
into lines that look like this:
f 21 22 23 24
Below is my terribly slow loop for doing this. Any suggestions about
how to make this code more efficient would be greatly appreciated
fileName = '/usr/home/poconnell/Desktop/objCube.obj'
with open(fileName) as lines:
theGoodLines = [line.strip("\n") for line in lines if "vn" not in
line and "vt" not in line and line != "\n"]
for i in range(len(theGoodLines)):
if theGoodLines[i][0] == "f":
aGoodLineAsList = theGoodLines[i].split(" ")
theGoodLines[i] = aGoodLineAsList[0] + " " +
aGoodLineAsList[1].split("/")[-1] + " " +
aGoodLineAsList[2].split("/")[-1] + " " +
aGoodLineAsList[3].split("/")[-1] + " " +
aGoodLineAsList[4].split("/")[-1]
for anItem in theGoodLines:
print anItem
##
Thanks!
Pete
On Wed, Aug 22, 2012 at 9:59 PM, Pete O'Connell wrote:
> Thanks Peter. This looks like what I need:
>
> with open(fileName) as lines:
> wanted = [line.strip("\n") for line in lines if "vn" not in line
> and "vt" not in line and line != "\n"]
>
> Cheers
>
> And in response to Allan's suggestion. I can see using a generator in
> a situation where the if statements were more numerous and complex. I
> am sure that will come in handy.
>
> Thanks
>
>
> On Wed, Aug 22, 2012 at 7:06 PM, Peter Otten <__pete...@web.de> wrote:
>> Pete O'Connell wrote:
>>
>>> Hi I am trying to parse a text file and create a list of all the lines
>>> that don't include: "vn", "vt" or are empty. I want to make this as
>>> fast as possible because I will be parsing many files each containing
>>> thousands of lines. I though I would give list comprehensions a try.
>>> The last 3 lines of the code below have three list comprehensions that
>>> I would like to combine into 1 but I am not sure how to do that.
>>> Any tips would be greatly appreciated
>>>
>>> pete
>>>
>>> #start
>>> fileName = '/usr/home/poconnell/Desktop/objCube.obj'
>>> theFileOpened = open(fileName,'r')
>>> theTextAsList = theFileOpened.readlines()
>>
>> If you have a file with 1,000,000 lines you have now a list of 1,000,000
>> strings of which perhaps 1,000 match your criteria. You are squandering
>> memory. Rule of thumb: never use readlines(), iterate over the file
>> directly.
>>
>>> theTextAsListStripped = []
>>> for aLine in theTextAsList:
>>>
>>> theTextAsListStripped.append(aLine.strip("\n"))
>>>
>>> theTextAsListNoVn = [x for x in theTextAsListStripped if "vn" not in x]
>>> theTextAsListNoVnOrVt = [x for x in theTextAsListNoVn if "vt" not in x]
>>> theTextAsListNoVnOrVtOrEmptyLine = [x for x in theTextAsListNoVn if x !=
>>> ""]
>>
>> I think that should be
>>
>> theTextAsListNoVnOrVtOrEmptyLine = [x for x in theTextAsListNoVnOrVt if x !=
>> ""]
>>
>> You can combine the three if clauses or add them all to one list-comp:
>>
>> with open(filename) as lines:
>> wanted = [line.strip("\n") for line in lines
>> if "vn" not in line and "vt" not in line and line != "\n"]
>>
>>
>> You can even have multiple if clauses in one list-comp (but that is rarely
>> used):
>>
>> with open(filename) as lines:
>> wanted = [line.strip("\n") for line
>> if "vn" not in line
>> if "vt" not in x
>> if line != "\n"]
>>
>> While your problem is simple enough to combine all filters into one list-
>> comp some problems are not. You can then prevent the intermediate lists from
>> materializing by using generator expressions. The result minimizes memory
>> consumption, too, and should be (almost) as fast. For example:
>>
>> with open(filename) as lines:
>> # use gen-exps to remove empty and whitespace-only lines
>> stripped = (line.strip() for line in lines)
>> nonempty = (line for line in stripped if line)
>>
>> wanted = [line for line in nonempty
>> if "vt" not in line and "vn" not in line]
>>
>>
>> ___
>> Tutor maillist - Tutor@python.org
>> To unsubscribe or change subscription options:
>> http://mail.python.org/mailman/listinfo/tutor
>
>
>
> --
> -
--
-
___
Tutor maillist - Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] list comprehension, testing for multiple conditions
Thanks Peter. This looks like what I need:
with open(fileName) as lines:
wanted = [line.strip("\n") for line in lines if "vn" not in line
and "vt" not in line and line != "\n"]
Cheers
And in response to Allan's suggestion. I can see using a generator in
a situation where the if statements were more numerous and complex. I
am sure that will come in handy.
Thanks
On Wed, Aug 22, 2012 at 7:06 PM, Peter Otten <__pete...@web.de> wrote:
> Pete O'Connell wrote:
>
>> Hi I am trying to parse a text file and create a list of all the lines
>> that don't include: "vn", "vt" or are empty. I want to make this as
>> fast as possible because I will be parsing many files each containing
>> thousands of lines. I though I would give list comprehensions a try.
>> The last 3 lines of the code below have three list comprehensions that
>> I would like to combine into 1 but I am not sure how to do that.
>> Any tips would be greatly appreciated
>>
>> pete
>>
>> #start
>> fileName = '/usr/home/poconnell/Desktop/objCube.obj'
>> theFileOpened = open(fileName,'r')
>> theTextAsList = theFileOpened.readlines()
>
> If you have a file with 1,000,000 lines you have now a list of 1,000,000
> strings of which perhaps 1,000 match your criteria. You are squandering
> memory. Rule of thumb: never use readlines(), iterate over the file
> directly.
>
>> theTextAsListStripped = []
>> for aLine in theTextAsList:
>>
>> theTextAsListStripped.append(aLine.strip("\n"))
>>
>> theTextAsListNoVn = [x for x in theTextAsListStripped if "vn" not in x]
>> theTextAsListNoVnOrVt = [x for x in theTextAsListNoVn if "vt" not in x]
>> theTextAsListNoVnOrVtOrEmptyLine = [x for x in theTextAsListNoVn if x !=
>> ""]
>
> I think that should be
>
> theTextAsListNoVnOrVtOrEmptyLine = [x for x in theTextAsListNoVnOrVt if x !=
> ""]
>
> You can combine the three if clauses or add them all to one list-comp:
>
> with open(filename) as lines:
> wanted = [line.strip("\n") for line in lines
> if "vn" not in line and "vt" not in line and line != "\n"]
>
>
> You can even have multiple if clauses in one list-comp (but that is rarely
> used):
>
> with open(filename) as lines:
> wanted = [line.strip("\n") for line
> if "vn" not in line
> if "vt" not in x
> if line != "\n"]
>
> While your problem is simple enough to combine all filters into one list-
> comp some problems are not. You can then prevent the intermediate lists from
> materializing by using generator expressions. The result minimizes memory
> consumption, too, and should be (almost) as fast. For example:
>
> with open(filename) as lines:
> # use gen-exps to remove empty and whitespace-only lines
> stripped = (line.strip() for line in lines)
> nonempty = (line for line in stripped if line)
>
> wanted = [line for line in nonempty
> if "vt" not in line and "vn" not in line]
>
>
> ___
> Tutor maillist - Tutor@python.org
> To unsubscribe or change subscription options:
> http://mail.python.org/mailman/listinfo/tutor
--
-
___
Tutor maillist - Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] list comprehension, testing for multiple conditions
What a great mailing list!
Thanks for all the responses.
I have a few questions, though, first in regards to Puneeth's code. He
writes to use:
>theTextAsListNoVnOrVtOrEmptyLine = [x for x in theTextAsListStripped if "vn"
>not in x and "vt" not in x and x!= ""]
It works but what I don't understand about this line is why the ands
are nor ors ("or" doesn't work even though I would have expected it
to)
I am sure I will have a few more questions over the next couple days
as I work my way through the responses.
Thanks
Pete
On Wed, Aug 22, 2012 at 6:23 PM, Puneeth Chaganti wrote:
> On Wed, Aug 22, 2012 at 11:35 AM, Pete O'Connell
> wrote:
>> Hi I am trying to parse a text file and create a list of all the lines
>> that don't include: "vn", "vt" or are empty. I want to make this as
>> fast as possible because I will be parsing many files each containing
>> thousands of lines. I though I would give list comprehensions a try.
>> The last 3 lines of the code below have three list comprehensions that
>> I would like to combine into 1 but I am not sure how to do that.
>> Any tips would be greatly appreciated
>>
>> pete
>>
>> #start
>> fileName = '/usr/home/poconnell/Desktop/objCube.obj'
>> theFileOpened = open(fileName,'r')
>> theTextAsList = theFileOpened.readlines()
>>
>> theTextAsListStripped = []
>> for aLine in theTextAsList:
>>
>> theTextAsListStripped.append(aLine.strip("\n"))
>>
>> theTextAsListNoVn = [x for x in theTextAsListStripped if "vn" not in x]
>> theTextAsListNoVnOrVt = [x for x in theTextAsListNoVn if "vt" not in x]
>> theTextAsListNoVnOrVtOrEmptyLine = [x for x in theTextAsListNoVn if x != ""]
>
> Something like this should work :
>
> theTextAsListNoVnOrVtOrEmptyLine = [x for x in theTextAsListStripped
> if "vn" not in x and "vt" not in x and x!= ""]
>
> HTH,
> Puneeth
--
-
___
Tutor maillist - Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] list comprehension, testing for multiple conditions
On 22/08/2012 07:29, Johann Spies wrote:
#start
import re
exclude = re.compile('vn|vt|^$|^#')
fileName = '/tmp/x'
theFileOpened = open(fileName,'r')
theTextAsList = theFileOpened.readlines()
theTextAsListStripped = []
for aLine in theTextAsList:
theTextAsListStripped.append(aLine.strip("\n"))
theTextAsListNoVn = [x for x in theTextAsListStripped if not
re.search(exclude,x)]
print theTextAsListNoVn
Regards
Johann
___
Tutor maillist - Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor
Please no, not a regex for something this simple!!!
--
Cheers.
Mark Lawrence.
___
Tutor maillist - Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] list comprehension, testing for multiple conditions
On 22/08/12 08:06, Peter Otten wrote:
You can even have multiple if clauses in one list-comp (but that is rarely
used):
with open(filename) as lines:
wanted = [line.strip("\n") for line
if "vn" not in line
if "vt" not in x
if line != "\n"]
While your problem is simple enough to combine all filters into one list-
comp some problems are not. You can then prevent the intermediate lists from
materializing by using generator expressions. The result minimizes memory
consumption, too, and should be (almost) as fast. For example:
with open(filename) as lines:
# use gen-exps to remove empty and whitespace-only lines
stripped = (line.strip() for line in lines)
nonempty = (line for line in stripped if line)
wanted = [line for line in nonempty
if "vt" not in line and "vn" not in line]
Another option using generators is to roll your own. This would be my
recomendation for really complex filtering:
def myFilterGenerator(aFile):
for line in aFile:
if 'vn' not in line: # demo "complexity" not efficiency!
if 'vt' not in line:
if '\n' != line:
yield line.strip()
with open filename as myFile:
result = [line for line in myFilterGenerator(myFile)]
But in this specific case the filter is simple enough that the
list comp from Peter is probably the best solution.
HTH
--
Alan G
Author of the Learn to Program web site
http://www.alan-g.me.uk/
___
Tutor maillist - Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] list comprehension, testing for multiple conditions
Pete O'Connell wrote:
> Hi I am trying to parse a text file and create a list of all the lines
> that don't include: "vn", "vt" or are empty. I want to make this as
> fast as possible because I will be parsing many files each containing
> thousands of lines. I though I would give list comprehensions a try.
> The last 3 lines of the code below have three list comprehensions that
> I would like to combine into 1 but I am not sure how to do that.
> Any tips would be greatly appreciated
>
> pete
>
> #start
> fileName = '/usr/home/poconnell/Desktop/objCube.obj'
> theFileOpened = open(fileName,'r')
> theTextAsList = theFileOpened.readlines()
If you have a file with 1,000,000 lines you have now a list of 1,000,000
strings of which perhaps 1,000 match your criteria. You are squandering
memory. Rule of thumb: never use readlines(), iterate over the file
directly.
> theTextAsListStripped = []
> for aLine in theTextAsList:
>
> theTextAsListStripped.append(aLine.strip("\n"))
>
> theTextAsListNoVn = [x for x in theTextAsListStripped if "vn" not in x]
> theTextAsListNoVnOrVt = [x for x in theTextAsListNoVn if "vt" not in x]
> theTextAsListNoVnOrVtOrEmptyLine = [x for x in theTextAsListNoVn if x !=
> ""]
I think that should be
theTextAsListNoVnOrVtOrEmptyLine = [x for x in theTextAsListNoVnOrVt if x !=
""]
You can combine the three if clauses or add them all to one list-comp:
with open(filename) as lines:
wanted = [line.strip("\n") for line in lines
if "vn" not in line and "vt" not in line and line != "\n"]
You can even have multiple if clauses in one list-comp (but that is rarely
used):
with open(filename) as lines:
wanted = [line.strip("\n") for line
if "vn" not in line
if "vt" not in x
if line != "\n"]
While your problem is simple enough to combine all filters into one list-
comp some problems are not. You can then prevent the intermediate lists from
materializing by using generator expressions. The result minimizes memory
consumption, too, and should be (almost) as fast. For example:
with open(filename) as lines:
# use gen-exps to remove empty and whitespace-only lines
stripped = (line.strip() for line in lines)
nonempty = (line for line in stripped if line)
wanted = [line for line in nonempty
if "vt" not in line and "vn" not in line]
___
Tutor maillist - Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] list comprehension, testing for multiple conditions
#start
import re
exclude = re.compile('vn|vt|^$|^#')
fileName = '/tmp/x'
theFileOpened = open(fileName,'r')
theTextAsList = theFileOpened.readlines()
theTextAsListStripped = []
for aLine in theTextAsList:
theTextAsListStripped.append(aLine.strip("\n"))
theTextAsListNoVn = [x for x in theTextAsListStripped if not
re.search(exclude,x)]
print theTextAsListNoVn
Regards
Johann
--
Because experiencing your loyal love is better than life itself,
my lips will praise you. (Psalm 63:3)
___
Tutor maillist - Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor
[Tutor] list comprehension, testing for multiple conditions
Hi I am trying to parse a text file and create a list of all the lines
that don't include: "vn", "vt" or are empty. I want to make this as
fast as possible because I will be parsing many files each containing
thousands of lines. I though I would give list comprehensions a try.
The last 3 lines of the code below have three list comprehensions that
I would like to combine into 1 but I am not sure how to do that.
Any tips would be greatly appreciated
pete
#start
fileName = '/usr/home/poconnell/Desktop/objCube.obj'
theFileOpened = open(fileName,'r')
theTextAsList = theFileOpened.readlines()
theTextAsListStripped = []
for aLine in theTextAsList:
theTextAsListStripped.append(aLine.strip("\n"))
theTextAsListNoVn = [x for x in theTextAsListStripped if "vn" not in x]
theTextAsListNoVnOrVt = [x for x in theTextAsListNoVn if "vt" not in x]
theTextAsListNoVnOrVtOrEmptyLine = [x for x in theTextAsListNoVn if x != ""]
#end
and here is the simple file I am parsing as a test:
#start##
## OBJ file generated by Nuke ##
# vertex list - offset=0
v 0.00 0.00 1.00
v 1.00 0.00 1.00
v 0.00 1.00 1.00
v 1.00 1.00 1.00
v 1.00 0.00 0.00
v 0.00 0.00 0.00
v 1.00 1.00 0.00
v 0.00 1.00 0.00
v 1.00 0.00 1.00
v 1.00 0.00 0.00
v 1.00 1.00 1.00
v 1.00 1.00 0.00
v 0.00 0.00 0.00
v 0.00 0.00 1.00
v 0.00 1.00 0.00
v 0.00 1.00 1.00
v 0.00 1.00 1.00
v 1.00 1.00 1.00
v 0.00 1.00 0.00
v 1.00 1.00 0.00
v 0.00 0.00 0.00
v 1.00 0.00 0.00
v 0.00 0.00 1.00
v 1.00 0.00 1.00
# point texture coordinates - offset=0
vt 0.00 0.00
vt 1.00 0.00
vt 0.00 1.00
vt 1.00 1.00
vt 0.00 0.00
vt 1.00 0.00
vt 0.00 1.00
vt 1.00 1.00
vt 0.00 0.00
vt 1.00 0.00
vt 0.00 1.00
vt 1.00 1.00
vt 0.00 0.00
vt 1.00 0.00
vt 0.00 1.00
vt 1.00 1.00
vt 0.00 0.00
vt 1.00 0.00
vt 0.00 1.00
vt 1.00 1.00
vt 0.00 0.00
vt 1.00 0.00
vt 0.00 1.00
vt 1.00 1.00
# vertex normals - offset=0
vn 0.00 0.00 1.00
vn 0.00 0.00 1.00
vn 0.00 0.00 1.00
vn 0.00 0.00 1.00
vn 0.00 0.00 -1.00
vn 0.00 0.00 -1.00
vn 0.00 0.00 -1.00
vn 0.00 0.00 -1.00
vn 1.00 0.00 0.00
vn 1.00 0.00 0.00
vn 1.00 0.00 0.00
vn 1.00 0.00 0.00
vn -1.00 0.00 0.00
vn -1.00 0.00 0.00
vn -1.00 0.00 0.00
vn -1.00 0.00 0.00
vn 0.00 1.00 0.00
vn 0.00 1.00 0.00
vn 0.00 1.00 0.00
vn 0.00 1.00 0.00
vn 0.00 -1.00 0.00
vn 0.00 -1.00 0.00
vn 0.00 -1.00 0.00
vn 0.00 -1.00 0.00
f 1/1/1 2/2/2 4/4/3 3/3/4
f 5/5/5 6/6/6 8/8/7 7/7/8
f 9/9/9 10/10/10 12/12/11 11/11/12
f 13/13/13 14/14/14 16/16/15 15/15/16
f 17/17/17 18/18/18 20/20/19 19/19/20
f 21/21/21 22/22/22 24/24/23 23/23/24
# end of file
___
Tutor maillist - Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor
