Hi,
On 2/20/13 9:58 AM, Olemis Lang wrote:
Nonetheless the issue is definitely related to this . I've been
tracking down this with PyDev and the source of all evil lays inside
`MultiProductSystem.upgrade_environment` method and is quite bizarre .
Immediately after executing the statement
{{{
#!py
db("CREATE TEMPORARY TABLE %s_temp AS SELECT %s FROM %s" %
(table, cols, table))
}}}
Thanks for pointing that out, that is in fact the actual source of the
problem. According to [1] and [2], sqlite creates temporary database on
the filesystem and keeps it there until the database connection is
closed. That happens despite the fact that the main database that we're
using is in-memory.
r1448062 fixes the issue by not creating temporary tables but rather
tables in the main database when migrating the tables.
... for the first time in `db_installed_version < 3` branch two new
file descriptors are open and not closed even after
{{{
#!py
db("DROP TABLE %s_temp" % table)
}}}
It strikes to me that this statement is inside a loop but new fds
allocated just once inside the loop , but a new pair once again after
a new method call . That's the sustained +2 fd rate I mentioned before
.
The statement is in the loop as the loop is executed for all tables that
need to be migrated ;)
@jure : even after fixing this I recommend to remove MP upgrade inside
EnvironmentStub.__init__
The leaking fd issue is fixed. Multi product upgrade inside
EnvironmentStub.__init__ was added to make sure that the database is
always upgraded to multi-product schema as the SQL translator is
installed in any case. Not upgrading the schema would cause translated
SQLs to hit invalid schema.
Cheers,
Jure
[1] sqlite CREATE TABLE - http://www.sqlite.org/lang_createtable.html
[2] sqlite inmemory database - http://www.sqlite.org/inmemorydb.html
