> Why aren't you measuring the time spent only in the SQLite writing thread? > > That would eliminate the overhead from the read-thread(s) and the queue in > the middle, > measuring only work by SQLite, instead of including the *waiting* for work > in the queue. > > I wrote a similarly "piped" ETL not too long ago, and I output overall > "wall" time of course, > but also time spent reading, time spent writing, but important times when > reader/writer threads > were "stalled", because the pipe is full/empty (respectively). --DD
Hello, Too the question, because it is unnecessary coding, time wasted. If I have an idea already of the goal and with the timing overall can determine where to focus the effort it is more efficient use of my time. The monitoring of the pipe, one coding action, already gives me an idea of when read/writer are stalled. So no need in to have timing for those. Yesterday I put the threads on equal footing and derived the following result which are close to my goal. 50,000 rows queried from a networked MariaDB, fields (integer, real, text, blob). SQLite - 114.838 seconds H2 - 115.868 seconds Derby - 136.984 seconds HSQL - 1291.808 seconds Mind you that the plugin needs to use any query to any of the supported databases, MariaDB, Oracle, PostgreSQL, etc. and derive a comparable schema table from the query to create a file/memory database. Looks like SQLite or H2 are the most likely candidates at this time. All of this is a lot of variables that effect timing. Focusing on only the data transfer timing, above, or writes to the sink db is only part of the timing, though probably the place to derive the most benefit. danap. _______________________________________________ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
