On 12/11/12 13:04, "Dr. André Lanka" wrote:
Hi Andy,
thanks for your answer. I really missed 0.9.4. You are right, the issues
vanished with the new version. Thanks for the hint.
By the way: We're just patching 0.9.4 to provide a global cache (for
both reading and writing) for both Nodes and Blocks in direct file mode.
This means that all tdb stores share this one big LRU cache.
By this we can easily read and write to thousands of stores in parallel.
Without theses changes we are unable to use more than 200 stores in one
jvm (OOME on a 2GB jvm).
Perhaps it may be an interesting option to integrate our changes to the
official tdb branch.
Yes - that would be interesting. The nest step would be to open an JIRA
for discussion and attaching patches.
It might be better to use that as the default option if the performance
impacts are not too great.
Of course the performance for a single store is slightly worse than your
implementation, but it allows one to have more stores on the same
machine (we have 4,500 on each).
A configuration parameter could control the behaviour of the factories
to use the one or the other approach.
Is each store using named graphs? Or is one store, one graph?
TDB support dynamic query to pick a subsets of the graphs out of a
potentially large pool of graphs [1]. If the choice of graph is exposed
at the API, not just in FROM / FROM NAMED (it may already be there but
with context magic), one dataset can server many graphs with query on
one graph at a time.
Andy
[1] http://jena.apache.org/documentation/tdb/dynamic_datasets.html
Greetings
André
This could be also useful for other users that use plenty of TDB
stores in parallel.
On 06.11.2012 21:53, Andy Seaborne wrote:
Have you tried version 0.9.4?
It has fixes that may apply to your situation.
Andy
On 06/11/12 08:33, "Dr. André Lanka" wrote:
Hello (once more),
the problem could be fixed by doing the following:
BlockMgrJournal: Add "blockMgr.sync();" to sync()
NodeTableTrans: Add "base.sync();" to sync()
Yet, I don't know what side effects this would have. Could some of the
tdb specialists judge these changes?
Best wishes
André
On 06.11.2012 08:52, "Dr. André Lanka" wrote:
Hello TDB developers
thanks for you great work :)
Unfortunately, I encountered some problems while testing TDB 0.9.3. It
turned out that a dataset.commit() doesn't force NodeTableNative to
write the whole data to disk or that NodeTableNative.syncNeeded wasn't
set properly earlier.
By this, tdb stores can get broken even if no write transaction is
active if the jvm crashes. (The other files seem to be written fine,
but
nodes.dat misses some information)
Is there a simple work-around for this problem (Simpler than patching
NodeTableNative)? Closing the store would help but is not feasible for
us.
Thanks in advance
André
