On Jan 15, 6:39 pm, Per Freem wrote:
> hello
>
> i have an optimization questions about python. i am iterating through
> a file and counting the number of repeated elements. the file has on
> the order
> of tens of millions elements...
>
> i create a dictionary that maps elements of the file that
Per Freem writes:
> the only 'twist' is that my elt is an instance of a class (MyClass)
> with 3 fields, all numeric. the class is hashable, and so
> my_dict[elt] works well. the __repr__ and __hash__ methods of my
> class simply return str() representation of self,
which just calls __str__().
On Jan 15, 4:39 pm, Per Freem wrote:
> hello
>
> i have an optimization questions about python. i am iterating through
> a file and counting the number of repeated elements. the file has on
> the order
> of tens of millions elements...
>
> i create a dictionary that maps elements of the file that
Paul Rubin wrote:
Per Freem writes:
2] is there an easy way to have nested defaultdicts? ie i want to say
that my_dict = defaultdict(defaultdict(int)) -- to reflect the fact
that my_dict is a dictionary, whose values are dictionary that map to
ints. but that syntax is not valid.
my_dict = def
On Jan 15, 5:31 pm, Per Freem wrote:
> ...the aKeys are very small (less than 100) where
> as the bKeys are the ones that are in the millions. so in that case,
> doing a Try-Except on aKey should be very efficient, since often it
> will not fail, ...
Do you know the aKeys in advance? If so, the
Per Freem schrieb:
> 1] is Try-Except really slower? my dict actually has two layers, so
> my_dict[aKey][bKeys]. the aKeys are very small (less than 100) where
> as the bKeys are the ones that are in the millions. so in that case,
> doing a Try-Except on aKey should be very efficient, since often
On Thu, 15 Jan 2009 14:49:29 -0800, bearophileHUGS wrote:
> Matimus, your suggestions are all good.
>
> Try-except is slower than:
> if x in adict: ... else: ...
Not according to my tests.
>>> def tryexcept(D, key):
... try:
... return D[key]
... except KeyError:
...
Per Freem writes:
> 2] is there an easy way to have nested defaultdicts? ie i want to say
> that my_dict = defaultdict(defaultdict(int)) -- to reflect the fact
> that my_dict is a dictionary, whose values are dictionary that map to
> ints. but that syntax is not valid.
my_dict = defaultdict(lambd
thanks to everyone for the excellent suggestions. a few follow up q's:
1] is Try-Except really slower? my dict actually has two layers, so
my_dict[aKey][bKeys]. the aKeys are very small (less than 100) where
as the bKeys are the ones that are in the millions. so in that case,
doing a Try-Except o
On Thu, 15 Jan 2009 23:22:48 +0100, Christian Heimes wrote:
>> is there anything that can be done to speed up this simply code? right
>> now it is taking well over 15 minutes to process, on a 3 Ghz machine
>> with lots of RAM (though this is all taking CPU power, not RAM at this
>> point.)
>
> cl
Matimus, your suggestions are all good.
Try-except is slower than:
if x in adict: ... else: ...
A defaultdict is generally faster (there are some conditions when it's
not faster, but they aren't much common. I think it's when the ratio
of duplicates is really low), creating just a tuple instead of
> class MyClass
>
> def __str__(self):
> return "%s-%s-%s" %(self.field1, self.field2, self.field3)
>
> def __repr__(self):
> return str(self)
>
> def __hash__(self):
> return hash(str(self))
>
>
> is there anything that can be done to speed up this simply code? right
> now i
On Fri, Jan 16, 2009 at 8:39 AM, Per Freem wrote:
> hello
>
> i have an optimization questions about python. i am iterating through
> a file and counting the number of repeated elements. the file has on
> the order
> of tens of millions elements...
>
>
> for line in file:
> try:
>elt = MyCla
On Jan 15, 1:39 pm, Per Freem wrote:
> hello
>
> i have an optimization questions about python. i am iterating through
> a file and counting the number of repeated elements. the file has on
> the order
> of tens of millions elements...
>
> i create a dictionary that maps elements of the file that
hello
i have an optimization questions about python. i am iterating through
a file and counting the number of repeated elements. the file has on
the order
of tens of millions elements...
i create a dictionary that maps elements of the file that i want to
count
to their number of occurs. so i iter
15 matches
Mail list logo
