On Sat, Sep 15, 2012 at 10:18 AM, Albert-Jan Roskam <fo...@yahoo.com> wrote:
> Thanks, I hadn't noticed this yet. I am refactoring some of the rest of my > code > and I hadn't run anything yet. My code has two methods that return record(s): > an iterator (__getitem__) and a generator (readFile, which is also called by > __enter__). Shouldn't I also take the possibility of a MemoryError into > account when the caller does something like data[:10**8]? It may no longer fit > into memory, esp. when the dataset is also wide. The issue with c_long isn't a problem for a slice since key.indices(self.nCases) limits the upper bound. For the individual index you had it right the first time by raising IndexError before it even gets to the c_long conversion. I'm sorry for wasting your time on a non-problem. However, your test there is a bit off. A negative index can be -nCases since counting from the end starts at -1. If you first do the ternary check to add the offset to a negative index, afterward you can raise an IndexError if "not 0 <= value < nCases". As to MemoryError, dealing with gigabytes of data in main memory is not a problem I've come up against in practice. You might still want a reasonable upper bound for slices. Often when the process runs out of memory it won't even see a MemoryError. The OS simply kills it. On the other hand, while bugs like a c_long wrapping around need to be caught to prevent silent corruption of data, there's nothing at all silent about crashing the process. It's up to you how much you want to micromanage the situation. You might want to check out psutil as a cross-platform way to monitor the process memory usage: http://code.google.com/p/psutil If you're also supporting the iterator protocol with the __iter__ method, then I think a helper _items(start, stop, step) generator function would be a good idea. Here's an updated example (not tested however; it's just a suggestion): import operator def _items(self, start=0, stop=None, step=1): if stop is None: stop = self.nCases for i in range(start, stop, step): retcode1 = self.iomodule.SeekNextCase(self.fh, ctypes.c_long(i)) self.caseBuffer, self.caseBufferPtr = self.getCaseBuffer() retcode2 = self.iomodule.WholeCaseIn(self.fh, self.caseBufferPtr) record = struct.unpack(self.structFmt, self.caseBuffer.raw) if any([retcode1, retcode2]): raise RuntimeError("Error retrieving record %d [%s, %s]" % (i, retcodes[retcode1], retcodes[retcode2])) yield record def __iter__(self): return self._items() def __getitem__(self, key): is_slice = isinstance(key, slice) if is_slice: start, stop, step = key.indices(self.nCases) else: key = operator.index(key) start = key + self.nCases if key < 0 else key if not 0 <= start < self.nCases: raise IndexError stop = start + 1 step = 1 records = self._items(start, stop, step) if is_slice: return list(records) return next(records) > but I didn't know about LOAD_ATTR. That's the bytecode operation to fetch an attribute. Whether or not bypassing it will provide a significant speedup depends on what else you're doing in the loop. If the the single LOAD_ATTR is only a small fraction of the total processing time, or you're not looping thousands of times, then this little change is insignificant. > Is a list comprehension still faster than this? I think list comprehensions or generator expressions are best if the evaluated expression isn't too complex and uses built-in types and functions. I won't typically write a function just to use a list comprehension for a single statement. Compared to a regular for loop (especially if append is cached in a fast local), the function call overhead makes it a wash or worse, even given the comprehension's efficiency at building the list. If the main work of the loop is the most significant factor, then the choice of for loop vs list comprehension doesn't matter much with regard to performance, but I still think it's simpler to just use a regular for loop. You can also write a generator function if you need to reuse an iteration in multiple statements. > Does it also mean that e.g. "from ctypes import *" (--> c_long()) is > faster than "import ctypes" (--> ctypes.c_long()). I am now putting as much as > possible in __init__. I don't like the first way of importing at all. It's not a good idea to pollute your namespace with "import *" statements. In a function, you can cache an attribute locally if doing so will provide a significant speedup. Or you can use a default argument like this: def f(x, c_long=ctypes.c_long): return c_long(x) _______________________________________________ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor