> o apply the log then append any running transactions to the new log >
This might be the heart of the problem. Dealing with running transactions. I guess, for this reason, there is good merit in making all database access that does not explicity focus on writing to the database, use readonly access. It may be that in almost all cases, we only need readonly access. Right now the entire GUI, all views, trees tables, charts, the map, the spreadsheet view, all require readonly access, and we can divide read-write needs into a few groups: - Neoclipse allows visual editing of the database with explicit 'commits', and so keeps a read-write transaction open at all times (tricky) - Data loaders run for a limited time (but can take a while, from seconds to hours), and do 99% writing (good cases for bulk load). It is sensible to restrict data loading to a single thread (not allow two loaders at the same time), but not sensible to disallow visual components access to the database (readonly access), during a load. - Analyses do a mix of reading and writing (read data, perform calculations and write results, on a stream so writes occur interspersed with reads). These are also time limited, usually seconds to minutes long. - The spreadsheet allows user input to the cells, which is write access to the database. This is tricky, but does not require a continuously open transaction like neoclipse, so is easier than that case. So, perhaps two application level locks are required. One to control neoclipse, switching it into readonly mode for the duration of any load, and one for blocking all analyses and the spreadsheet during a load. Not impossible, but somewhat complicated I think. I think the best way to move forward is to have a look at the writes > you perform now and try tweak it. Hopefully we can achieve good enough > write speed for your use case running in normal transactional > read/write mode. > We have already optimized the bulk load into the spreadsheet, and something that takes 20 seconds in open office takes several minutes in our code. I don't believe we can ever get a generic data structure like a graph database to match the performance of a specialized structure in the mainstream office apps, but they should at least be of the same order of magnitude. Perhap bulk load remains the only option left. Cheers, Craig _______________________________________________ Neo mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
