On Thursday, December 20, 2012 6:17:04 PM UTC-7, Dave Angel wrote:
On 12/20/2012 07:19 PM, larry.mart...@gmail.com wrote:
I have a list of tuples that contains a tool_id, a time, and a
message. I want to select from this list all the elements where the
message matches some string, and all the other elements where the
time is within some diff of any matching message for that tool.
Here is how I am currently doing this:
No, it's not. This is a fragment of code, without enough clues as to
what else is going. We can guess, but that's likely to make a mess.
Of course it's a fragment - it's part of a large program and I was
just showing the relevant parts.
First question is whether this code works exactly correctly?
Yes, the code works. I end up with just the rows I want.
Are you only concerned about speed, not fixing features?
Don't know what you mean by 'fixing features'. The code does what I
want, it just takes too long.
As far as I can tell, the logic that includes the time comparison is
bogus.
Not at all.
You don't do anything there to worry about the value of tup[2],
just whether some
item has a nearby time. Of course, I could misunderstand the spec.
The data comes from a database. tup[2] is a datetime column. tdiff
comes from a datetime.timedelta()
Are you making a global called 'self' ? That name is by convention only
used in methods to designate the instance object. What's the attribute
self?
Yes, self is my instance object. self.message contains the string of
interest that I need to look for.
Can cdata have duplicates, and are they significant?
No, it will not have duplicates.
Are you including the time building that as part of your 2 hour
measurement?
No, the 2 hours is just the time to run the
cdata[:] = [tup for tup in cdata if determine(tup)]
Is the list sorted in any way?
Yes, the list is sorted by tool and datetime.
Chances are your performance bottleneck is the doubly-nested loop. You
have a list comprehension at top-level code, and inside it calls a
function that also loops over the 600,000 items. So the inner loop
gets
executed 360 billion times. You can cut this down drastically by some
judicious sorting, as well as by having a map of lists, where the
map is
keyed by the tool.
Thanks. I will try that.
# record time for each message matching the specified message for
each tool
messageTimes = {}
You're building a dictionary; are you actually using the value (1), or
is only the key relevant?
Only the keys.
A set is a dict without a value.
Yes, I could use a set, but I don't think that would make it
measurably faster.
But more mportantly, you never look up anything in this dictionary.
So why
isn't it a list? For that matter, why don't you just use the
messageTimes list?
Yes, it could be a list too.
for row in cdata: # tool, time, message
if self.message in row[2]:
messageTimes[row[0], row[1]] = 1
# now pull out each message that is within the time diff for each
matched message
# as well as the matched messages themselves
def determine(tup):
if self.message in tup[2]: return True # matched message
for (tool, date_time) in messageTimes:
if tool == tup[0]:
if abs(date_time-tup[1]) <= tdiff:
return True
return False
cdata[:] = [tup for tup in cdata if determine(tup)]
As the code exists, there's no need to copy the list. Just do a simple
bind.
This statement is to remove the items from cdata that I don't want. I
don't know what you mean by bind. I'm not familiar with that python
function.
This code works, but it takes way too long to run - e.g. when cdata
has 600,000 elements (which is typical for my app) it takes 2 hours
for this to run.
Can anyone give me some suggestions on speeding this up?
