i get more than 30 minutes, due i must parse to a low end machine, not to your 4 cores, 16Gb ram super power machine.. i'm taking about a 1G ram and single core 1,6GHz atom cpu
i need to convert from Result/cursor to other due the problem of the odbc lack of cursor/count .. i thinking about use a sqlite memory structure, how can i force it? documentation said "If Name is null, then a memory database is opened." for sqlite.. so if i used a memory structure can be a good idea? *tested yesterday took about 10 minutes but i dont know if i have a problem in my gambas installation!* Lenz McKAY Gerardo (PICCORO) http://qgqlochekone.blogspot.com 2017-06-30 4:09 GMT-04:00 adamn...@gmail.com <adamn...@gmail.com>: > On Thu, 29 Jun 2017 18:57:29 -0400 > PICCORO McKAY Lenz <mckaygerh...@gmail.com> wrote: > > > can i convert directly or more faster than copy each row, a Result from > > database to a collection or a VArian matrix? > > > > i'm taking about 200.000 rows in a result... the problem its that the > odbc > > db object support only cursor with forward only.. > > > > so with a matrix or a collection i cant emulate the cursor behaviour > > > > Lenz McKAY Gerardo (PICCORO) > > http://qgqlochekone.blogspot.com > > Interesting. > > Well the row by row copy is how we do it here. I added some quick timer > Prints to a program we run each day to verify that the > database updates done overnight were "clean". > The data loaded is a fairly complex join of several tables, the > transactional table is 754,756 rows today and the master table is 733,723 > rows long and the transactional data is compared to the master data to test > a set of possible inconsistencies. ( The actual query returned a set of the > transaction and master records that were actioned overnight - this > generally returns about 5,000 to 10,000 rows - so I jigged it to return the > pairs that were not actioned overnight thereby getting row counts of the > sizes you are talking about.) So the jigged query just returned 556,000 > rows. Here's the timing output. > > 17:05:59:706 Connecting to DB > 17:06:00:202 Loading Data <---- so 406 mSec to establish the db > connection > 17:06:31:417 556502 rows <---- so 31,215 mSec to execute the query > and return the result > 17:06:31:417 Unmarshalling result started > 17:06:44:758 Unmarshalling completed 556502 rows processed <--- so > 13,341 mSec to unmarshall the result into an array of structs > > So, it took roughly 31 seconds to execute the query and return the result > of half a million rows. > To unmarshall that result into the array took just over 13 seconds. The > unmarshalling is fairly well a straight field by field copy. > (Also I must add, I ran this on a local db copy on my old steam driven > laptop, 32 bits and about 1G of memory.) > > That's about 42 mSec unmarshalling time per row. > I don't think that is too bad. From my perspective it is the query that is > eating up my life, not the unmarshalling. > > What sort of times to you get? > > b > > (p.s. the query has been optimised until its' eyes bled. ) > > -- > B Bruen <adamn...@gnail.com (sort of)> > > ------------------------------------------------------------ > ------------------ > Check out the vibrant tech community on one of the world's most > engaging tech sites, Slashdot.org! http://sdm.link/slashdot > _______________________________________________ > Gambas-user mailing list > Gambas-user@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/gambas-user > ------------------------------------------------------------------------------ Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot _______________________________________________ Gambas-user mailing list Gambas-user@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/gambas-user