Re: [pygtk] push huge array into liststore
Hi List, hi Zyphos,
cl.
Detaching the model from the treeview already speeded up things from 50
seconds to unbelievable 20 seconds!
I will also take a look at the other hint.
Thanks a lot and kind regards
Cornelius
Am 13.07.2010 15:35, schrieb Zyphos:
Hi,
a solution is written into the FAQ:
http://faq.pygtk.org/index.py?req=showfile=faq13.043.htp
treeview.freeze_child_notify()
treeview.set_model(None)
# Add rows to the model
# ...
treeview.set_model(model)
treeview.thaw_child_notify()
Regards,
Zyphos
Tim Evans a écrit :
On 2010-07-13 3:48, Cornelius Kölbel wrote:
Dear list,
I got an array of dictionaries, that I want to add to a GTKListStore,
that is displayed an a treeview.
This is very slow. I already replaced the liststore.append by
liststore.insert, which is much much faster.
But still, filling the 10.000 values takes about 50 seconds.
Is there any cool mapping function, to push the array to the liststore?
I used a for loop to iterate over the array...
I also tried while an to pop the array elements...
It is something like this:
data: array of dictionaries
data = array( { 'v1' : 'something', 'v2': 'something else' } ,
{ 'v1' : 'another', 'v2': 'something completely diff' }
)
for d in data:
self.liststore.insert( 0, ( d.get(v1), d.get(v2) ))
...is there a better way than doing a for loop?
...or a way to not have to interate over the 10.000 dicts?
...or is there a cool reading on performance tuning pyton?
Some general ideas:
- Make sure the store isn't being viewed when you insert data, that
will slow it down.
- Do attribute lookups outside big loops:
append = self.liststore.append
get = d.get
for d in data:
append(0, (get('v1'), get('v2'), ...))
- Put the treeview into fixed height and width mode. Automatic sizing
is the enemy of treeview performance. This only works if all your
rows are the same height. The function calls you need are:
gtk.TreeViewColumn.set_sizing
with the gtk.TREE_VIEW_COLUMN_FIXED value
gtk.TreeViewColumn.set_fixed_width
gtk.TreeView.set_fixed_height_mode
- If your list is large enough it may be worth subclassing
gtk.TreeModel and overriding the required methods. It's complex, but
it can avoid referencing all your data at tree build time, instead
loading as the user scrolls.
- Write it in C. Always valid, even if as a last resort.
___
pygtk mailing list pygtk@daa.com.au
http://www.daa.com.au/mailman/listinfo/pygtk
Read the PyGTK FAQ: http://faq.pygtk.org/
signature.asc
Description: OpenPGP digital signature
___
pygtk mailing list pygtk@daa.com.au
http://www.daa.com.au/mailman/listinfo/pygtk
Read the PyGTK FAQ: http://faq.pygtk.org/
Re: [pygtk] push huge array into liststore
Hello List, hello Gerald,
this reduced the time from roughly 20 sec to 19.5 sec.
So real evil thing i was doing was sorting and filtering on the model.
So detaching the model from the treeview while filling it, is far the
best idea ,-)
Thanks a lot and kind regards
Cornelius
Am 13.07.2010 20:21, schrieb Gerald Britton:
2010/7/12 Gerald Britton gerald.brit...@gmail.com:
First off, why are you using an array? Not that it's bad or anything,
but how does it help you where a simple list would not?
Second, what would be nice is if liststore.insert would take an
interable. Baring that you could simplify a little like this:
keys = ['v1', 'v1', ..., 'vn']
for d in data:
selt.liststore.insert(0, tuple(d[k] for k in keys))
whether it is worth it or not depends on how many keys you have.
Here's another way, assuming that there
are not too many keys to be looked up:
append = self.liststore.append# lookup the method once
from operator import itemgetter
v1 = itemgetter('v1') # special getter for 'v1'
v2 = itemgetter('v2') # ...
v3 = itemgetter('v3')
...
for d in data:
append((v1(d), v2(d), v3(d), ...))
Yet another approach: If you can use OrderedDict (new in Python 2.7
and 3.1), you can do this:
append = self.liststore.append
for d in data:
append(d.values())
2010/7/12 Pietro Battiston m...@pietrobattiston.it:
Il giorno lun, 12/07/2010 alle 17.48 +0200, Cornelius Kölbel ha scritto:
Dear list,
I got an array of dictionaries, that I want to add to a GTKListStore,
that is displayed an a treeview.
This is very slow. I already replaced the liststore.append by
liststore.insert, which is much much faster.
But still, filling the 10.000 values takes about 50 seconds.
Is there any cool mapping function, to push the array to the liststore?
I used a for loop to iterate over the array...
I also tried while an to pop the array elements...
It is something like this:
data: array of dictionaries
data = array( { 'v1' : 'something', 'v2': 'something else' } ,
{ 'v1' : 'another', 'v2': 'something completely diff' }
)
for d in data:
self.liststore.insert( 0, ( d.get(v1), d.get(v2) ))
...is there a better way than doing a for loop?
...or a way to not have to interate over the 10.000 dicts?
...or is there a cool reading on performance tuning pyton?
I really don't know, but... already read
http://faq.pygtk.org/index.py?req=showfile=faq13.043.htp
?
Pietro
___
pygtk mailing list pygtk@daa.com.au
http://www.daa.com.au/mailman/listinfo/pygtk
Read the PyGTK FAQ: http://faq.pygtk.org/
--
Gerald Britton
--
Cornelius Kölbel
(Senior Security Consultant, Head of Product Management)
http://www.lsexperts.de
LSE Leading Security Experts GmbH, Postfach 100121, 64201 Darmstadt
Tel: +49 6151 9067-252, Fax: -299, Mobil: +49 160 96307089
Unternehmenssitz: Weiterstadt, Amtsgericht Darmstadt: HRB8649
Geschaeftsfuehrer: Oliver Michel, Sven Walther
signature.asc
Description: OpenPGP digital signature
___
pygtk mailing list pygtk@daa.com.au
http://www.daa.com.au/mailman/listinfo/pygtk
Read the PyGTK FAQ: http://faq.pygtk.org/
Re: [pygtk] push huge array into liststore
On 14-07-10 11:19, Cornelius Kölbel wrote:
Hello List, hello Gerald,
this reduced the time from roughly 20 sec to 19.5 sec.
So real evil thing i was doing was sorting and filtering on the model.
So detaching the model from the treeview while filling it, is far the
best idea ,-)
I improved my fill-treeview-code upon reading this subject, the
following gave good results:
treeview.freeze_child_notify()
treeview.set_model(None)
liststore.set_default_sort_func(lambda *args: -1)
liststore.set_sort_column_id(-1, gtk.SORT_ASCENDING)
for d in data:
liststore.insert(0, [d])
treeview.set_model(liststore)
treeview.thaw_child_notify()
The disconnection of the model and remove the sort function gave great
speed performance. But using insert() instead of append() helped too.
Cheers,
Timo
Thanks a lot and kind regards
Cornelius
Am 13.07.2010 20:21, schrieb Gerald Britton:
2010/7/12 Gerald Brittongerald.brit...@gmail.com:
First off, why are you using an array? Not that it's bad or anything,
but how does it help you where a simple list would not?
Second, what would be nice is if liststore.insert would take an
interable. Baring that you could simplify a little like this:
keys = ['v1', 'v1', ..., 'vn']
for d in data:
selt.liststore.insert(0, tuple(d[k] for k in keys))
whether it is worth it or not depends on how many keys you have.
Here's another way, assuming that there
are not too many keys to be looked up:
append = self.liststore.append# lookup the method once
from operator import itemgetter
v1 = itemgetter('v1') # special getter for 'v1'
v2 = itemgetter('v2') # ...
v3 = itemgetter('v3')
...
for d in data:
append((v1(d), v2(d), v3(d), ...))
Yet another approach: If you can use OrderedDict (new in Python 2.7
and 3.1), you can do this:
append = self.liststore.append
for d in data:
append(d.values())
2010/7/12 Pietro Battistonm...@pietrobattiston.it:
Il giorno lun, 12/07/2010 alle 17.48 +0200, Cornelius Kölbel ha scritto:
Dear list,
I got an array of dictionaries, that I want to add to a GTKListStore,
that is displayed an a treeview.
This is very slow. I already replaced the liststore.append by
liststore.insert, which is much much faster.
But still, filling the 10.000 values takes about 50 seconds.
Is there any cool mapping function, to push the array to the liststore?
I used a for loop to iterate over the array...
I also tried while an to pop the array elements...
It is something like this:
data: array of dictionaries
data = array( { 'v1' : 'something', 'v2': 'something else' } ,
{ 'v1' : 'another', 'v2': 'something completely diff' }
)
for d in data:
self.liststore.insert( 0, ( d.get(v1), d.get(v2) ))
...is there a better way than doing a for loop?
...or a way to not have to interate over the 10.000 dicts?
...or is there a cool reading on performance tuning pyton?
I really don't know, but... already read
http://faq.pygtk.org/index.py?req=showfile=faq13.043.htp
?
Pietro
___
pygtk mailing list pygtk@daa.com.au
http://www.daa.com.au/mailman/listinfo/pygtk
Read the PyGTK FAQ: http://faq.pygtk.org/
--
Gerald Britton
___
pygtk mailing list pygtk@daa.com.au
http://www.daa.com.au/mailman/listinfo/pygtk
Read the PyGTK FAQ: http://faq.pygtk.org/
___
pygtk mailing list pygtk@daa.com.au
http://www.daa.com.au/mailman/listinfo/pygtk
Read the PyGTK FAQ: http://faq.pygtk.org/
Re: [pygtk] push huge array into liststore
Hi,
a solution is written into the FAQ:
http://faq.pygtk.org/index.py?req=showfile=faq13.043.htp
treeview.freeze_child_notify()
treeview.set_model(None)
# Add rows to the model
# ...
treeview.set_model(model)
treeview.thaw_child_notify()
Regards,
Zyphos
Tim Evans a écrit :
On 2010-07-13 3:48, Cornelius Kölbel wrote:
Dear list,
I got an array of dictionaries, that I want to add to a GTKListStore,
that is displayed an a treeview.
This is very slow. I already replaced the liststore.append by
liststore.insert, which is much much faster.
But still, filling the 10.000 values takes about 50 seconds.
Is there any cool mapping function, to push the array to the liststore?
I used a for loop to iterate over the array...
I also tried while an to pop the array elements...
It is something like this:
data: array of dictionaries
data = array( { 'v1' : 'something', 'v2': 'something else' } ,
{ 'v1' : 'another', 'v2': 'something completely diff' }
)
for d in data:
self.liststore.insert( 0, ( d.get(v1), d.get(v2) ))
...is there a better way than doing a for loop?
...or a way to not have to interate over the 10.000 dicts?
...or is there a cool reading on performance tuning pyton?
Some general ideas:
- Make sure the store isn't being viewed when you insert data, that
will slow it down.
- Do attribute lookups outside big loops:
append = self.liststore.append
get = d.get
for d in data:
append(0, (get('v1'), get('v2'), ...))
- Put the treeview into fixed height and width mode. Automatic sizing
is the enemy of treeview performance. This only works if all your
rows are the same height. The function calls you need are:
gtk.TreeViewColumn.set_sizing
with the gtk.TREE_VIEW_COLUMN_FIXED value
gtk.TreeViewColumn.set_fixed_width
gtk.TreeView.set_fixed_height_mode
- If your list is large enough it may be worth subclassing
gtk.TreeModel and overriding the required methods. It's complex, but
it can avoid referencing all your data at tree build time, instead
loading as the user scrolls.
- Write it in C. Always valid, even if as a last resort.
___
pygtk mailing list pygtk@daa.com.au
http://www.daa.com.au/mailman/listinfo/pygtk
Read the PyGTK FAQ: http://faq.pygtk.org/
Re: [pygtk] push huge array into liststore
On Mon, Jul 12, 2010 at 9:09 PM, Tim Evans t.ev...@aranz.com wrote:
On 2010-07-13 3:48, Cornelius Kölbel wrote:
Dear list,
I got an array of dictionaries, that I want to add to a GTKListStore,
that is displayed an a treeview.
This is very slow. I already replaced the liststore.append by
liststore.insert, which is much much faster.
But still, filling the 10.000 values takes about 50 seconds.
Is there any cool mapping function, to push the array to the liststore?
I used a for loop to iterate over the array...
I also tried while an to pop the array elements...
It is something like this:
data: array of dictionaries
data = array( { 'v1' : 'something', 'v2': 'something else' } ,
{ 'v1' : 'another', 'v2': 'something completely diff' }
)
for d in data:
self.liststore.insert( 0, ( d.get(v1), d.get(v2) ))
...is there a better way than doing a for loop?
...or a way to not have to interate over the 10.000 dicts?
...or is there a cool reading on performance tuning pyton?
Some general ideas:
- Make sure the store isn't being viewed when you insert data, that
will slow it down.
- Do attribute lookups outside big loops:
append = self.liststore.append
get = d.get
for d in data:
append(0, (get('v1'), get('v2'), ...))
This sample won't work since you are using d before it is bound in
the for loop. Also, the append method only takes one argument: the
row to be added to the store. Here's another way, assuming that there
are not too many keys to be looked up:
append = self.liststore.append
from operator import itemgetter
v1 = itemgetter('v1')
v2 = itemgetter('v2')
v3 = itemgetter('v3')
...
for d in data:
append((v1(d), v2(d), v3(d), ...))
Yet another approach: If you can use OrderedDict (new in Python 2.7
and 3.1), you can do this:
append = self.liststore.append
for d in data:
append(d.values())
- Put the treeview into fixed height and width mode. Automatic sizing
is the enemy of treeview performance. This only works if all your
rows are the same height. The function calls you need are:
gtk.TreeViewColumn.set_sizing
with the gtk.TREE_VIEW_COLUMN_FIXED value
gtk.TreeViewColumn.set_fixed_width
gtk.TreeView.set_fixed_height_mode
- If your list is large enough it may be worth subclassing
gtk.TreeModel and overriding the required methods. It's complex, but
it can avoid referencing all your data at tree build time, instead
loading as the user scrolls.
- Write it in C. Always valid, even if as a last resort.
--
Tim Evans
Applied Research Associates NZ
http://www.aranz.com/
___
pygtk mailing list pygtk@daa.com.au
http://www.daa.com.au/mailman/listinfo/pygtk
Read the PyGTK FAQ: http://faq.pygtk.org/
--
Gerald Britton
___
pygtk mailing list pygtk@daa.com.au
http://www.daa.com.au/mailman/listinfo/pygtk
Read the PyGTK FAQ: http://faq.pygtk.org/
Re: [pygtk] push huge array into liststore
2010/7/12 Gerald Britton gerald.brit...@gmail.com:
First off, why are you using an array? Not that it's bad or anything,
but how does it help you where a simple list would not?
Second, what would be nice is if liststore.insert would take an
interable. Baring that you could simplify a little like this:
keys = ['v1', 'v1', ..., 'vn']
for d in data:
selt.liststore.insert(0, tuple(d[k] for k in keys))
whether it is worth it or not depends on how many keys you have.
Here's another way, assuming that there
are not too many keys to be looked up:
append = self.liststore.append# lookup the method once
from operator import itemgetter
v1 = itemgetter('v1') # special getter for 'v1'
v2 = itemgetter('v2') # ...
v3 = itemgetter('v3')
...
for d in data:
append((v1(d), v2(d), v3(d), ...))
Yet another approach: If you can use OrderedDict (new in Python 2.7
and 3.1), you can do this:
append = self.liststore.append
for d in data:
append(d.values())
2010/7/12 Pietro Battiston m...@pietrobattiston.it:
Il giorno lun, 12/07/2010 alle 17.48 +0200, Cornelius Kölbel ha scritto:
Dear list,
I got an array of dictionaries, that I want to add to a GTKListStore,
that is displayed an a treeview.
This is very slow. I already replaced the liststore.append by
liststore.insert, which is much much faster.
But still, filling the 10.000 values takes about 50 seconds.
Is there any cool mapping function, to push the array to the liststore?
I used a for loop to iterate over the array...
I also tried while an to pop the array elements...
It is something like this:
data: array of dictionaries
data = array( { 'v1' : 'something', 'v2': 'something else' } ,
{ 'v1' : 'another', 'v2': 'something completely diff' }
)
for d in data:
self.liststore.insert( 0, ( d.get(v1), d.get(v2) ))
...is there a better way than doing a for loop?
...or a way to not have to interate over the 10.000 dicts?
...or is there a cool reading on performance tuning pyton?
I really don't know, but... already read
http://faq.pygtk.org/index.py?req=showfile=faq13.043.htp
?
Pietro
___
pygtk mailing list pygtk@daa.com.au
http://www.daa.com.au/mailman/listinfo/pygtk
Read the PyGTK FAQ: http://faq.pygtk.org/
--
Gerald Britton
--
Gerald Britton
___
pygtk mailing list pygtk@daa.com.au
http://www.daa.com.au/mailman/listinfo/pygtk
Read the PyGTK FAQ: http://faq.pygtk.org/
Re: [pygtk] push huge array into liststore
Il giorno lun, 12/07/2010 alle 17.48 +0200, Cornelius Kölbel ha scritto:
Dear list,
I got an array of dictionaries, that I want to add to a GTKListStore,
that is displayed an a treeview.
This is very slow. I already replaced the liststore.append by
liststore.insert, which is much much faster.
But still, filling the 10.000 values takes about 50 seconds.
Is there any cool mapping function, to push the array to the liststore?
I used a for loop to iterate over the array...
I also tried while an to pop the array elements...
It is something like this:
data: array of dictionaries
data = array( { 'v1' : 'something', 'v2': 'something else' } ,
{ 'v1' : 'another', 'v2': 'something completely diff' }
)
for d in data:
self.liststore.insert( 0, ( d.get(v1), d.get(v2) ))
...is there a better way than doing a for loop?
...or a way to not have to interate over the 10.000 dicts?
...or is there a cool reading on performance tuning pyton?
I really don't know, but... already read
http://faq.pygtk.org/index.py?req=showfile=faq13.043.htp
?
Pietro
___
pygtk mailing list pygtk@daa.com.au
http://www.daa.com.au/mailman/listinfo/pygtk
Read the PyGTK FAQ: http://faq.pygtk.org/
Re: [pygtk] push huge array into liststore
First off, why are you using an array? Not that it's bad or anything,
but how does it help you where a simple list would not?
Second, what would be nice is if liststore.insert would take an
interable. Baring that you could simplify a little like this:
keys = ['v1', 'v1', ..., 'vn']
for d in data:
selt.liststore.insert(0, tuple(d[k] for k in keys))
whether it is worth it or not depends on how many keys you have.
2010/7/12 Pietro Battiston m...@pietrobattiston.it:
Il giorno lun, 12/07/2010 alle 17.48 +0200, Cornelius Kölbel ha scritto:
Dear list,
I got an array of dictionaries, that I want to add to a GTKListStore,
that is displayed an a treeview.
This is very slow. I already replaced the liststore.append by
liststore.insert, which is much much faster.
But still, filling the 10.000 values takes about 50 seconds.
Is there any cool mapping function, to push the array to the liststore?
I used a for loop to iterate over the array...
I also tried while an to pop the array elements...
It is something like this:
data: array of dictionaries
data = array( { 'v1' : 'something', 'v2': 'something else' } ,
{ 'v1' : 'another', 'v2': 'something completely diff' }
)
for d in data:
self.liststore.insert( 0, ( d.get(v1), d.get(v2) ))
...is there a better way than doing a for loop?
...or a way to not have to interate over the 10.000 dicts?
...or is there a cool reading on performance tuning pyton?
I really don't know, but... already read
http://faq.pygtk.org/index.py?req=showfile=faq13.043.htp
?
Pietro
___
pygtk mailing list pygtk@daa.com.au
http://www.daa.com.au/mailman/listinfo/pygtk
Read the PyGTK FAQ: http://faq.pygtk.org/
--
Gerald Britton
___
pygtk mailing list pygtk@daa.com.au
http://www.daa.com.au/mailman/listinfo/pygtk
Read the PyGTK FAQ: http://faq.pygtk.org/
Re: [pygtk] push huge array into liststore
On 2010-07-13 3:48, Cornelius Kölbel wrote:
Dear list,
I got an array of dictionaries, that I want to add to a GTKListStore,
that is displayed an a treeview.
This is very slow. I already replaced the liststore.append by
liststore.insert, which is much much faster.
But still, filling the 10.000 values takes about 50 seconds.
Is there any cool mapping function, to push the array to the liststore?
I used a for loop to iterate over the array...
I also tried while an to pop the array elements...
It is something like this:
data: array of dictionaries
data = array( { 'v1' : 'something', 'v2': 'something else' } ,
{ 'v1' : 'another', 'v2': 'something completely diff' }
)
for d in data:
self.liststore.insert( 0, ( d.get(v1), d.get(v2) ))
...is there a better way than doing a for loop?
...or a way to not have to interate over the 10.000 dicts?
...or is there a cool reading on performance tuning pyton?
Some general ideas:
- Make sure the store isn't being viewed when you insert data, that
will slow it down.
- Do attribute lookups outside big loops:
append = self.liststore.append
get = d.get
for d in data:
append(0, (get('v1'), get('v2'), ...))
- Put the treeview into fixed height and width mode. Automatic sizing
is the enemy of treeview performance. This only works if all your
rows are the same height. The function calls you need are:
gtk.TreeViewColumn.set_sizing
with the gtk.TREE_VIEW_COLUMN_FIXED value
gtk.TreeViewColumn.set_fixed_width
gtk.TreeView.set_fixed_height_mode
- If your list is large enough it may be worth subclassing
gtk.TreeModel and overriding the required methods. It's complex, but
it can avoid referencing all your data at tree build time, instead
loading as the user scrolls.
- Write it in C. Always valid, even if as a last resort.
--
Tim Evans
Applied Research Associates NZ
http://www.aranz.com/
___
pygtk mailing list pygtk@daa.com.au
http://www.daa.com.au/mailman/listinfo/pygtk
Read the PyGTK FAQ: http://faq.pygtk.org/
