Re: [pygtk] push huge array into liststore
On 14-07-10 11:19, Cornelius Kölbel wrote: > Hello List, hello Gerald, > > this reduced the time from roughly 20 sec to 19.5 sec. > So real evil thing i was doing was sorting and filtering on the model. > So detaching the model from the treeview while filling it, is far the > best idea ,-) > I improved my fill-treeview-code upon reading this subject, the following gave good results: treeview.freeze_child_notify() treeview.set_model(None) liststore.set_default_sort_func(lambda *args: -1) liststore.set_sort_column_id(-1, gtk.SORT_ASCENDING) for d in data: liststore.insert(0, [d]) treeview.set_model(liststore) treeview.thaw_child_notify() The disconnection of the model and remove the sort function gave great speed performance. But using insert() instead of append() helped too. Cheers, Timo > Thanks a lot and kind regards > Cornelius > > Am 13.07.2010 20:21, schrieb Gerald Britton: > >> 2010/7/12 Gerald Britton: >> >> >>> First off, why are you using an array? Not that it's bad or anything, >>> but how does it help you where a simple list would not? >>> >>> Second, what would be nice is if liststore.insert would take an >>> interable. Baring that you could simplify a little like this: >>> >>> keys = ['v1', 'v1', ..., 'vn'] >>> for d in data: >>> selt.liststore.insert(0, tuple(d[k] for k in keys)) >>> >>> whether it is worth it or not depends on how many keys you have. >>> >>> >> Here's another way, assuming that there >> are not too many keys to be looked up: >> >> append = self.liststore.append# lookup the method once >> from operator import itemgetter >> v1 = itemgetter('v1') # special getter for 'v1' >> v2 = itemgetter('v2') # ... >> v3 = itemgetter('v3') >> ... >> for d in data: >> append((v1(d), v2(d), v3(d), ...)) >> >> Yet another approach: If you can use OrderedDict (new in Python 2.7 >> and 3.1), you can do this: >> >> append = self.liststore.append >> for d in data: >>append(d.values()) >> >> >> >>> 2010/7/12 Pietro Battiston: >>> >>> Il giorno lun, 12/07/2010 alle 17.48 +0200, Cornelius Kölbel ha scritto: > Dear list, > > I got an array of dictionaries, that I want to add to a GTKListStore, > that is displayed an a treeview. > > This is very slow. I already replaced the liststore.append by > liststore.insert, which is much much faster. > But still, filling the 10.000 values takes about 50 seconds. > > Is there any cool mapping function, to push the array to the liststore? > I used a for loop to iterate over the array... > I also tried while an to pop the array elements... > > It is something like this: > > data: array of dictionaries > > data = array( { 'v1' : 'something', 'v2': 'something else' } , >{ 'v1' : 'another', 'v2': 'something completely diff' } >) > > > for d in data: > self.liststore.insert( 0, ( d.get("v1"), d.get("v2") )) > > > ...is there a better way than doing a for loop? > ...or a way to not have to interate over the 10.000 dicts? > > ...or is there a cool reading on performance tuning pyton? > > I really don't know, but... already read http://faq.pygtk.org/index.py?req=show&file=faq13.043.htp ? Pietro ___ pygtk mailing list pygtk@daa.com.au http://www.daa.com.au/mailman/listinfo/pygtk Read the PyGTK FAQ: http://faq.pygtk.org/ >>> >>> -- >>> Gerald Britton >>> >>> >>> >> >> >> > > > > ___ > pygtk mailing list pygtk@daa.com.au > http://www.daa.com.au/mailman/listinfo/pygtk > Read the PyGTK FAQ: http://faq.pygtk.org/ ___ pygtk mailing list pygtk@daa.com.au http://www.daa.com.au/mailman/listinfo/pygtk Read the PyGTK FAQ: http://faq.pygtk.org/
Re: [pygtk] push huge array into liststore
Hello List, hello Gerald, this reduced the time from roughly 20 sec to 19.5 sec. So real evil thing i was doing was sorting and filtering on the model. So detaching the model from the treeview while filling it, is far the best idea ,-) Thanks a lot and kind regards Cornelius Am 13.07.2010 20:21, schrieb Gerald Britton: > 2010/7/12 Gerald Britton : > >> First off, why are you using an array? Not that it's bad or anything, >> but how does it help you where a simple list would not? >> >> Second, what would be nice is if liststore.insert would take an >> interable. Baring that you could simplify a little like this: >> >> keys = ['v1', 'v1', ..., 'vn'] >> for d in data: >>selt.liststore.insert(0, tuple(d[k] for k in keys)) >> >> whether it is worth it or not depends on how many keys you have. >> > Here's another way, assuming that there > are not too many keys to be looked up: > > append = self.liststore.append# lookup the method once > from operator import itemgetter > v1 = itemgetter('v1') # special getter for 'v1' > v2 = itemgetter('v2') # ... > v3 = itemgetter('v3') > ... > for d in data: >append((v1(d), v2(d), v3(d), ...)) > > Yet another approach: If you can use OrderedDict (new in Python 2.7 > and 3.1), you can do this: > > append = self.liststore.append > for d in data: > append(d.values()) > > >> 2010/7/12 Pietro Battiston : >> >>> Il giorno lun, 12/07/2010 alle 17.48 +0200, Cornelius Kölbel ha scritto: >>> Dear list, I got an array of dictionaries, that I want to add to a GTKListStore, that is displayed an a treeview. This is very slow. I already replaced the liststore.append by liststore.insert, which is much much faster. But still, filling the 10.000 values takes about 50 seconds. Is there any cool mapping function, to push the array to the liststore? I used a for loop to iterate over the array... I also tried while an to pop the array elements... It is something like this: data: array of dictionaries data = array( { 'v1' : 'something', 'v2': 'something else' } , { 'v1' : 'another', 'v2': 'something completely diff' } ) for d in data: self.liststore.insert( 0, ( d.get("v1"), d.get("v2") )) ...is there a better way than doing a for loop? ...or a way to not have to interate over the 10.000 dicts? ...or is there a cool reading on performance tuning pyton? >>> >>> >>> I really don't know, but... already read >>> http://faq.pygtk.org/index.py?req=show&file=faq13.043.htp >>> ? >>> >>> Pietro >>> >>> ___ >>> pygtk mailing list pygtk@daa.com.au >>> http://www.daa.com.au/mailman/listinfo/pygtk >>> Read the PyGTK FAQ: http://faq.pygtk.org/ >>> >> >> >> -- >> Gerald Britton >> >> > > > -- Cornelius Kölbel (Senior Security Consultant, Head of Product Management) http://www.lsexperts.de LSE Leading Security Experts GmbH, Postfach 100121, 64201 Darmstadt Tel: +49 6151 9067-252, Fax: -299, Mobil: +49 160 96307089 Unternehmenssitz: Weiterstadt, Amtsgericht Darmstadt: HRB8649 Geschaeftsfuehrer: Oliver Michel, Sven Walther signature.asc Description: OpenPGP digital signature ___ pygtk mailing list pygtk@daa.com.au http://www.daa.com.au/mailman/listinfo/pygtk Read the PyGTK FAQ: http://faq.pygtk.org/
Re: [pygtk] push huge array into liststore
Hi List, hi Zyphos, cl. Detaching the model from the treeview already speeded up things from 50 seconds to unbelievable 20 seconds! I will also take a look at the other hint. Thanks a lot and kind regards Cornelius Am 13.07.2010 15:35, schrieb Zyphos: > Hi, > a solution is written into the FAQ: > http://faq.pygtk.org/index.py?req=show&file=faq13.043.htp > > treeview.freeze_child_notify() > treeview.set_model(None) > > # Add rows to the model > # ... > > treeview.set_model(model) > treeview.thaw_child_notify() > > Regards, > > Zyphos > > > Tim Evans a écrit : > >> On 2010-07-13 3:48, Cornelius Kölbel wrote: >> >> >>> Dear list, >>> >>> I got an array of dictionaries, that I want to add to a GTKListStore, >>> that is displayed an a treeview. >>> >>> This is very slow. I already replaced the liststore.append by >>> liststore.insert, which is much much faster. >>> But still, filling the 10.000 values takes about 50 seconds. >>> >>> Is there any cool mapping function, to push the array to the liststore? >>> I used a for loop to iterate over the array... >>> I also tried while an to pop the array elements... >>> >>> It is something like this: >>> >>> data: array of dictionaries >>> >>> data = array( { 'v1' : 'something', 'v2': 'something else' } , >>>{ 'v1' : 'another', 'v2': 'something completely diff' } >>> ) >>> >>> >>> for d in data: >>> self.liststore.insert( 0, ( d.get("v1"), d.get("v2") )) >>> >>> >>> ...is there a better way than doing a for loop? >>> ...or a way to not have to interate over the 10.000 dicts? >>> >>> ...or is there a cool reading on performance tuning pyton? >>> >>> >> Some general ideas: >> - Make sure the store isn't being viewed when you insert data, that >> will slow it down. >> >> - Do attribute lookups outside big loops: >> >> append = self.liststore.append >> get = d.get >> for d in data: >> append(0, (get('v1'), get('v2'), ...)) >> >> - Put the treeview into fixed height and width mode. Automatic sizing >> is the enemy of treeview performance. This only works if all your >> rows are the same height. The function calls you need are: >> gtk.TreeViewColumn.set_sizing >> with the gtk.TREE_VIEW_COLUMN_FIXED value >> gtk.TreeViewColumn.set_fixed_width >> gtk.TreeView.set_fixed_height_mode >> >> - If your list is large enough it may be worth subclassing >> gtk.TreeModel and overriding the required methods. It's complex, but >> it can avoid referencing all your data at tree build time, instead >> loading as the user scrolls. >> >> - Write it in C. Always valid, even if as a last resort. >> >> >> > ___ > pygtk mailing list pygtk@daa.com.au > http://www.daa.com.au/mailman/listinfo/pygtk > Read the PyGTK FAQ: http://faq.pygtk.org/ > signature.asc Description: OpenPGP digital signature ___ pygtk mailing list pygtk@daa.com.au http://www.daa.com.au/mailman/listinfo/pygtk Read the PyGTK FAQ: http://faq.pygtk.org/
Re: [pygtk] push huge array into liststore
2010/7/12 Gerald Britton : > First off, why are you using an array? Not that it's bad or anything, > but how does it help you where a simple list would not? > > Second, what would be nice is if liststore.insert would take an > interable. Baring that you could simplify a little like this: > > keys = ['v1', 'v1', ..., 'vn'] > for d in data: > selt.liststore.insert(0, tuple(d[k] for k in keys)) > > whether it is worth it or not depends on how many keys you have. Here's another way, assuming that there are not too many keys to be looked up: append = self.liststore.append# lookup the method once from operator import itemgetter v1 = itemgetter('v1') # special getter for 'v1' v2 = itemgetter('v2') # ... v3 = itemgetter('v3') ... for d in data: append((v1(d), v2(d), v3(d), ...)) Yet another approach: If you can use OrderedDict (new in Python 2.7 and 3.1), you can do this: append = self.liststore.append for d in data: append(d.values()) > > 2010/7/12 Pietro Battiston : >> Il giorno lun, 12/07/2010 alle 17.48 +0200, Cornelius Kölbel ha scritto: >>> Dear list, >>> >>> I got an array of dictionaries, that I want to add to a GTKListStore, >>> that is displayed an a treeview. >>> >>> This is very slow. I already replaced the liststore.append by >>> liststore.insert, which is much much faster. >>> But still, filling the 10.000 values takes about 50 seconds. >>> >>> Is there any cool mapping function, to push the array to the liststore? >>> I used a for loop to iterate over the array... >>> I also tried while an to pop the array elements... >>> >>> It is something like this: >>> >>> data: array of dictionaries >>> >>> data = array( { 'v1' : 'something', 'v2': 'something else' } , >>> { 'v1' : 'another', 'v2': 'something completely diff' } >>> ) >>> >>> >>> for d in data: >>> self.liststore.insert( 0, ( d.get("v1"), d.get("v2") )) >>> >>> >>> ...is there a better way than doing a for loop? >>> ...or a way to not have to interate over the 10.000 dicts? >>> >>> ...or is there a cool reading on performance tuning pyton? >> >> >> >> I really don't know, but... already read >> http://faq.pygtk.org/index.py?req=show&file=faq13.043.htp >> ? >> >> Pietro >> >> ___ >> pygtk mailing list pygtk@daa.com.au >> http://www.daa.com.au/mailman/listinfo/pygtk >> Read the PyGTK FAQ: http://faq.pygtk.org/ > > > > -- > Gerald Britton > -- Gerald Britton ___ pygtk mailing list pygtk@daa.com.au http://www.daa.com.au/mailman/listinfo/pygtk Read the PyGTK FAQ: http://faq.pygtk.org/
Re: [pygtk] push huge array into liststore
On Mon, Jul 12, 2010 at 9:09 PM, Tim Evans wrote: > On 2010-07-13 3:48, Cornelius Kölbel wrote: >> Dear list, >> >> I got an array of dictionaries, that I want to add to a GTKListStore, >> that is displayed an a treeview. >> >> This is very slow. I already replaced the liststore.append by >> liststore.insert, which is much much faster. >> But still, filling the 10.000 values takes about 50 seconds. >> >> Is there any cool mapping function, to push the array to the liststore? >> I used a for loop to iterate over the array... >> I also tried while an to pop the array elements... >> >> It is something like this: >> >> data: array of dictionaries >> >> data = array( { 'v1' : 'something', 'v2': 'something else' } , >> { 'v1' : 'another', 'v2': 'something completely diff' } >> ) >> >> >> for d in data: >> self.liststore.insert( 0, ( d.get("v1"), d.get("v2") )) >> >> >> ...is there a better way than doing a for loop? >> ...or a way to not have to interate over the 10.000 dicts? >> >> ...or is there a cool reading on performance tuning pyton? > > Some general ideas: > - Make sure the store isn't being viewed when you insert data, that > will slow it down. > > - Do attribute lookups outside big loops: > > append = self.liststore.append > get = d.get > for d in data: > append(0, (get('v1'), get('v2'), ...)) This sample won't work since you are using "d" before it is bound in the for loop. Also, the "append" method only takes one argument: the row to be added to the store. Here's another way, assuming that there are not too many keys to be looked up: append = self.liststore.append from operator import itemgetter v1 = itemgetter('v1') v2 = itemgetter('v2') v3 = itemgetter('v3') ... for d in data: append((v1(d), v2(d), v3(d), ...)) Yet another approach: If you can use OrderedDict (new in Python 2.7 and 3.1), you can do this: append = self.liststore.append for d in data: append(d.values()) > > - Put the treeview into fixed height and width mode. Automatic sizing > is the enemy of treeview performance. This only works if all your > rows are the same height. The function calls you need are: > gtk.TreeViewColumn.set_sizing > with the gtk.TREE_VIEW_COLUMN_FIXED value > gtk.TreeViewColumn.set_fixed_width > gtk.TreeView.set_fixed_height_mode > > - If your list is large enough it may be worth subclassing > gtk.TreeModel and overriding the required methods. It's complex, but > it can avoid referencing all your data at tree build time, instead > loading as the user scrolls. > > - Write it in C. Always valid, even if as a last resort. > > -- > Tim Evans > Applied Research Associates NZ > http://www.aranz.com/ > ___ > pygtk mailing list pygtk@daa.com.au > http://www.daa.com.au/mailman/listinfo/pygtk > Read the PyGTK FAQ: http://faq.pygtk.org/ > -- Gerald Britton ___ pygtk mailing list pygtk@daa.com.au http://www.daa.com.au/mailman/listinfo/pygtk Read the PyGTK FAQ: http://faq.pygtk.org/
Re: [pygtk] push huge array into liststore
Hi, a solution is written into the FAQ: http://faq.pygtk.org/index.py?req=show&file=faq13.043.htp treeview.freeze_child_notify() treeview.set_model(None) # Add rows to the model # ... treeview.set_model(model) treeview.thaw_child_notify() Regards, Zyphos Tim Evans a écrit : > On 2010-07-13 3:48, Cornelius Kölbel wrote: > >> Dear list, >> >> I got an array of dictionaries, that I want to add to a GTKListStore, >> that is displayed an a treeview. >> >> This is very slow. I already replaced the liststore.append by >> liststore.insert, which is much much faster. >> But still, filling the 10.000 values takes about 50 seconds. >> >> Is there any cool mapping function, to push the array to the liststore? >> I used a for loop to iterate over the array... >> I also tried while an to pop the array elements... >> >> It is something like this: >> >> data: array of dictionaries >> >> data = array( { 'v1' : 'something', 'v2': 'something else' } , >>{ 'v1' : 'another', 'v2': 'something completely diff' } >> ) >> >> >> for d in data: >> self.liststore.insert( 0, ( d.get("v1"), d.get("v2") )) >> >> >> ...is there a better way than doing a for loop? >> ...or a way to not have to interate over the 10.000 dicts? >> >> ...or is there a cool reading on performance tuning pyton? >> > > Some general ideas: > - Make sure the store isn't being viewed when you insert data, that > will slow it down. > > - Do attribute lookups outside big loops: > > append = self.liststore.append > get = d.get > for d in data: > append(0, (get('v1'), get('v2'), ...)) > > - Put the treeview into fixed height and width mode. Automatic sizing > is the enemy of treeview performance. This only works if all your > rows are the same height. The function calls you need are: > gtk.TreeViewColumn.set_sizing > with the gtk.TREE_VIEW_COLUMN_FIXED value > gtk.TreeViewColumn.set_fixed_width > gtk.TreeView.set_fixed_height_mode > > - If your list is large enough it may be worth subclassing > gtk.TreeModel and overriding the required methods. It's complex, but > it can avoid referencing all your data at tree build time, instead > loading as the user scrolls. > > - Write it in C. Always valid, even if as a last resort. > > ___ pygtk mailing list pygtk@daa.com.au http://www.daa.com.au/mailman/listinfo/pygtk Read the PyGTK FAQ: http://faq.pygtk.org/
Re: [pygtk] push huge array into liststore
On 2010-07-13 3:48, Cornelius Kölbel wrote: > Dear list, > > I got an array of dictionaries, that I want to add to a GTKListStore, > that is displayed an a treeview. > > This is very slow. I already replaced the liststore.append by > liststore.insert, which is much much faster. > But still, filling the 10.000 values takes about 50 seconds. > > Is there any cool mapping function, to push the array to the liststore? > I used a for loop to iterate over the array... > I also tried while an to pop the array elements... > > It is something like this: > > data: array of dictionaries > > data = array( { 'v1' : 'something', 'v2': 'something else' } , >{ 'v1' : 'another', 'v2': 'something completely diff' } > ) > > > for d in data: > self.liststore.insert( 0, ( d.get("v1"), d.get("v2") )) > > > ...is there a better way than doing a for loop? > ...or a way to not have to interate over the 10.000 dicts? > > ...or is there a cool reading on performance tuning pyton? Some general ideas: - Make sure the store isn't being viewed when you insert data, that will slow it down. - Do attribute lookups outside big loops: append = self.liststore.append get = d.get for d in data: append(0, (get('v1'), get('v2'), ...)) - Put the treeview into fixed height and width mode. Automatic sizing is the enemy of treeview performance. This only works if all your rows are the same height. The function calls you need are: gtk.TreeViewColumn.set_sizing with the gtk.TREE_VIEW_COLUMN_FIXED value gtk.TreeViewColumn.set_fixed_width gtk.TreeView.set_fixed_height_mode - If your list is large enough it may be worth subclassing gtk.TreeModel and overriding the required methods. It's complex, but it can avoid referencing all your data at tree build time, instead loading as the user scrolls. - Write it in C. Always valid, even if as a last resort. -- Tim Evans Applied Research Associates NZ http://www.aranz.com/ ___ pygtk mailing list pygtk@daa.com.au http://www.daa.com.au/mailman/listinfo/pygtk Read the PyGTK FAQ: http://faq.pygtk.org/
Re: [pygtk] push huge array into liststore
First off, why are you using an array? Not that it's bad or anything, but how does it help you where a simple list would not? Second, what would be nice is if liststore.insert would take an interable. Baring that you could simplify a little like this: keys = ['v1', 'v1', ..., 'vn'] for d in data: selt.liststore.insert(0, tuple(d[k] for k in keys)) whether it is worth it or not depends on how many keys you have. 2010/7/12 Pietro Battiston : > Il giorno lun, 12/07/2010 alle 17.48 +0200, Cornelius Kölbel ha scritto: >> Dear list, >> >> I got an array of dictionaries, that I want to add to a GTKListStore, >> that is displayed an a treeview. >> >> This is very slow. I already replaced the liststore.append by >> liststore.insert, which is much much faster. >> But still, filling the 10.000 values takes about 50 seconds. >> >> Is there any cool mapping function, to push the array to the liststore? >> I used a for loop to iterate over the array... >> I also tried while an to pop the array elements... >> >> It is something like this: >> >> data: array of dictionaries >> >> data = array( { 'v1' : 'something', 'v2': 'something else' } , >> { 'v1' : 'another', 'v2': 'something completely diff' } >> ) >> >> >> for d in data: >> self.liststore.insert( 0, ( d.get("v1"), d.get("v2") )) >> >> >> ...is there a better way than doing a for loop? >> ...or a way to not have to interate over the 10.000 dicts? >> >> ...or is there a cool reading on performance tuning pyton? > > > > I really don't know, but... already read > http://faq.pygtk.org/index.py?req=show&file=faq13.043.htp > ? > > Pietro > > ___ > pygtk mailing list pygtk@daa.com.au > http://www.daa.com.au/mailman/listinfo/pygtk > Read the PyGTK FAQ: http://faq.pygtk.org/ -- Gerald Britton ___ pygtk mailing list pygtk@daa.com.au http://www.daa.com.au/mailman/listinfo/pygtk Read the PyGTK FAQ: http://faq.pygtk.org/
Re: [pygtk] push huge array into liststore
Il giorno lun, 12/07/2010 alle 17.48 +0200, Cornelius Kölbel ha scritto: > Dear list, > > I got an array of dictionaries, that I want to add to a GTKListStore, > that is displayed an a treeview. > > This is very slow. I already replaced the liststore.append by > liststore.insert, which is much much faster. > But still, filling the 10.000 values takes about 50 seconds. > > Is there any cool mapping function, to push the array to the liststore? > I used a for loop to iterate over the array... > I also tried while an to pop the array elements... > > It is something like this: > > data: array of dictionaries > > data = array( { 'v1' : 'something', 'v2': 'something else' } , > { 'v1' : 'another', 'v2': 'something completely diff' } > ) > > > for d in data: >self.liststore.insert( 0, ( d.get("v1"), d.get("v2") )) > > > ...is there a better way than doing a for loop? > ...or a way to not have to interate over the 10.000 dicts? > > ...or is there a cool reading on performance tuning pyton? I really don't know, but... already read http://faq.pygtk.org/index.py?req=show&file=faq13.043.htp ? Pietro ___ pygtk mailing list pygtk@daa.com.au http://www.daa.com.au/mailman/listinfo/pygtk Read the PyGTK FAQ: http://faq.pygtk.org/
[pygtk] push huge array into liststore
Dear list, I got an array of dictionaries, that I want to add to a GTKListStore, that is displayed an a treeview. This is very slow. I already replaced the liststore.append by liststore.insert, which is much much faster. But still, filling the 10.000 values takes about 50 seconds. Is there any cool mapping function, to push the array to the liststore? I used a for loop to iterate over the array... I also tried while an to pop the array elements... It is something like this: data: array of dictionaries data = array( { 'v1' : 'something', 'v2': 'something else' } , { 'v1' : 'another', 'v2': 'something completely diff' } ) for d in data: self.liststore.insert( 0, ( d.get("v1"), d.get("v2") )) ...is there a better way than doing a for loop? ...or a way to not have to interate over the 10.000 dicts? ...or is there a cool reading on performance tuning pyton? Kind regards Cornelius signature.asc Description: OpenPGP digital signature ___ pygtk mailing list pygtk@daa.com.au http://www.daa.com.au/mailman/listinfo/pygtk Read the PyGTK FAQ: http://faq.pygtk.org/