Re: TDB blocked

Thu, 26 Mar 2020 05:22:15 -0700 

Fork - what sort of fork?  Java's Fork/join  or OS process fork?
Doesn't forking (no exec) a JVM bypass all JVM related initialization, e.g. classes. That will break a lot of things, TDB included. I don't know if OS-fork clones everything anyway (e.g. threads and that means ThreadLocal might be broken). [The scala doc is a little parsimonius] 

---

There is no need to have one shared dataset/connection object anyway.
TDBFactory.create* is cheap after the dataset is first used in the JVM process. It's little more than a cache lookup. 

A test can be

   TDBFactory.create*
   <small test>
and be fast enough i.e push the dataset connect/create into the test or test suite.
Bonus - create an in-memory TDB dataset so it leaves no on disk foot print. Or the equivalent of a JUnit Temp folder but the create in-memeory has lower start-up costs. 

    Andy
In-memory TDB dataset is not designed for production use - it is very careful to be true TDB, just without peristent storage. It runs its own copy-in/copy-out block cache so a lot of data copying to get proper isolation of storage and usage. 

Use DatasetFactory.createTxnMem() for a production in-memory database.

On 26/03/2020 08:24, Jean-Marc Vanel wrote:

Le mer. 25 mars 2020 à 17:17, Andy Seaborne <a...@apache.org> a écrit :



On 25/03/2020 14:36, Jean-Marc Vanel wrote:

Le mer. 25 mars 2020 à 13:07, Andy Seaborne <a...@apache.org> a écrit :






...





What's the full stack trace?
(What's the blocking thread?)

I was looking for the other thread that is blocking this one at
being(write).  Same JVM.



See details below. Tell me if you still want it.



Yes, another unended W transaction in the same process might be the

cause.



I thought rather, another unended W transaction in *another* process,

since

the tests execution was parallel.


If I understand correctly,

parallelExecution in Test := false

means parallel in the same JVM.



Yes indeed, that's for taking advantage of multicore CPU's.
The parameter that actually created the freeze problem in the original post
was

fork = true

and it clear that calling TDBFactory.createDataset() and close() , and
other calls, randomly from different JVM's on the same TDB directory is
bound to fail.

On the other hand, the combination

fork := false
parallelExecution in Test := true

is not good for this kind of tests. My understanding is that TDB is made
for multi-thread access in reading and writing (and it works perfectly in
my Play Framework application for years), but not made for creating and
closing.
All tests pass, but one, with this stack (complete!):

[info] TimeSeriesTestJena:
[info] - notifyDataEvent + getTimeSeries
[info] deductions.runtime.semlogs.TimeSeriesTestJena *** ABORTED ***
[info]   java.util.*ConcurrentModificationException*: Reader = 0, Writer = 2
[info]   at
org.apache.jena.tdb.sys.DatasetControlMRSW.policyError(DatasetControlMRSW.java:147)
[info]   at
org.apache.jena.tdb.sys.DatasetControlMRSW.policyError(DatasetControlMRSW.java:143)
[info]   at
org.apache.jena.tdb.sys.DatasetControlMRSW.checkConcurrency(DatasetControlMRSW.java:75)
[info]   at
org.apache.jena.tdb.sys.DatasetControlMRSW.startUpdate(DatasetControlMRSW.java:57)
[info]   at
org.apache.jena.tdb.store.nodetupletable.NodeTupleTableConcrete.startWrite(NodeTupleTableConcrete.java:65)
[info]   at
org.apache.jena.tdb.store.nodetupletable.NodeTupleTableConcrete.sync(NodeTupleTableConcrete.java:251)
[info]   at org.apache.jena.tdb.store.TableBase.sync(TableBase.java:51)
[info]   at
org.apache.jena.tdb.store.DatasetGraphTDB.sync(DatasetGraphTDB.java:253)
[info]   at
org.apache.jena.tdb.transaction.DatasetGraphTransaction.close(DatasetGraphTransaction.java:296)
[info]   at
org.apache.jena.sparql.core.DatasetImpl.close(DatasetImpl.java:231)



      Andy




Test code:


https://github.com/jmvanel/semantic_forms/blob/master/scala/forms/src/test/scala/deductions/runtime/sparql_cache/TestRDFCache.scala

=====
Thread [pool-1-thread-1-ScalaTest-running-TestRDFCache] (Suspended)
Unsafe.park(boolean, long) line: not available [native method]
LockSupport.park(Object) line: 175
Semaphore$FairSync(AbstractQueuedSynchronizer).parkAndCheckInterrupt()
line: 836


Semaphore$FairSync(AbstractQueuedSynchronizer).doAcquireSharedInterruptibly(int)

line: 997


Semaphore$FairSync(AbstractQueuedSynchronizer).acquireSharedInterruptibly(int)

line: 1304
Semaphore.acquire() line: 312
TransactionManager.acquireWriterLock(boolean) line: 616
TransactionManager.beginInternal(TxnType, TxnType, String) line: 358
TransactionManager.begin(TxnType, String) line: 343
StoreConnection.begin(TxnType, String) line: 128
StoreConnection.begin(TxnType) line: 108
DatasetGraphTransaction.begin(TxnType) line: 169
DatasetGraphTransaction.begin(ReadWrite) line: 162
DatasetImpl.begin(ReadWrite) line: 122
JenaDatasetStore.$anonfun$rw$1(Dataset, Function0) line: 23
52027729.apply() line: not available
Try$.apply(Function0<T>) line: 213
JenaDatasetStore.rw(Dataset, Function0<T>) line: 22
JenaDatasetStore.rw(Object, Function0) line: 10
TestRDFCache(RDFCacheAlgo<Rdf,DATASET>).readStoreURI(Object, Object,
DATASET) line: 368
RDFCacheAlgo<Rdf,DATASET>.readStoreURI$(RDFCacheAlgo, Object, Object,
Object) line: 366
TestRDFCache.readStoreURI(Object, Object, Object) line: 10
TestRDFCache(RDFCacheAlgo<Rdf,DATASET>).readStoreURIinOwnGraph(Object)
line: 334
TestRDFCache(RDFCacheAlgo<Rdf,DATASET>).readStoreUriInNamedGraph(Object)
line: 326
RDFCacheAlgo<Rdf,DATASET>.readStoreUriInNamedGraph$(RDFCacheAlgo, Object)
line: 325
TestRDFCache.readStoreUriInNamedGraph(Object) line: 10
TestRDFCache.$anonfun$new$4(TestRDFCache) line: 63
1708361043.apply$mcV$sp() line: not available
1708361043(JFunction0$mcV$sp).apply() line: 23
OutcomeOf$(OutcomeOf).outcomeOf(Function0<Object>) line: 85
OutcomeOf.outcomeOf$(OutcomeOf, Function0) line: 83
OutcomeOf$.outcomeOf(Function0<Object>) line: 104
Transformer<FixtureParam>.apply() line: 22
Transformer<FixtureParam>.apply() line: 20
AnyFunSuiteLike$$anon$1.apply() line: 189
TestRDFCache(TestSuite).withFixture(TestSuite$NoArgTest) line: 196
TestSuite.withFixture$(TestSuite, TestSuite$NoArgTest) line: 195
TestRDFCache(AnyFunSuite).withFixture(TestSuite$NoArgTest) line: 1562
TestRDFCache(AnyFunSuiteLike).invokeWithFixture$1(SuperEngine$TestLeaf,
Args, String) line: 187
AnyFunSuiteLike.$anonfun$runTest$1(AnyFunSuiteLike, Args, String,
SuperEngine$TestLeaf) line: 199
1593480726.apply(Object) line: not available
Engine(SuperEngine<T>).runTestImpl(Suite, String, Args, boolean,
Function1<SuperEngine<TestLea>,Outcome>) line: 306
TestRDFCache(AnyFunSuiteLike).runTest(String, Args) line: 199
AnyFunSuiteLike.runTest$(AnyFunSuiteLike, String, Args) line: 181
TestRDFCache(AnyFunSuite).runTest(String, Args) line: 1562
AnyFunSuiteLike.$anonfun$runTests$1(AnyFunSuiteLike, String, Args) line:
232
438507753.apply(Object, Object) line: not available
SuperEngine<T>.$anonfun$runTestsInBranch$1(SuperEngine, Args, Suite,
SuperEngine$Branch, ListBuffer, Function2, boolean, SuperEngine$Node)

line:

413
855709148.apply(Object) line: not available
$colon$colon<B>(List<A>).foreach(Function1<A,U>) line: 392
Engine(SuperEngine<T>).traverseSubNodes$1(SuperEngine$Branch, Args,

Suite,

ListBuffer, Function2, boolean) line: 401
Engine(SuperEngine<T>).runTestsInBranch(Suite, SuperEngine<Branc>, Args,
boolean, Function2<String,Args,Status>) line: 396
Engine(SuperEngine<T>).runTestsImpl(Suite, Option<String>, Args,

Informer,

boolean, Function2<String,Args,Status>) line: 475
TestRDFCache(AnyFunSuiteLike).runTests(Option<String>, Args) line: 232
AnyFunSuiteLike.runTests$(AnyFunSuiteLike, Option, Args) line: 231
TestRDFCache(AnyFunSuite).runTests(Option<String>, Args) line: 1562
TestRDFCache(Suite).run(Option<String>, Args) line: 1112
Suite.run$(Suite, Option, Args) line: 1094


TestRDFCache(AnyFunSuite).org$scalatest$funsuite$AnyFunSuiteLike$$super$run(Option,

Args) line: 1562
AnyFunSuiteLike.$anonfun$run$1(AnyFunSuiteLike, Option, Args) line: 236
113821125.apply(Object, Object) line: not available
Engine(SuperEngine<T>).runImpl(Suite, Option<String>, Args,
Function2<Option<String>,Args,Status>) line: 535
TestRDFCache(AnyFunSuiteLike).run(Option<String>, Args) line: 236
AnyFunSuiteLike.run$(AnyFunSuiteLike, Option, Args) line: 235
TestRDFCache.org$scalatest$BeforeAndAfterAll$$super$run(Option, Args)

line:

10
TestRDFCache(BeforeAndAfterAll).liftedTree1$1(Args, Option) line: 213
TestRDFCache(BeforeAndAfterAll).run(Option<String>, Args) line: 210
BeforeAndAfterAll.run$(BeforeAndAfterAll, Option, Args) line: 208
TestRDFCache.run(Option<String>, Args) line: 10
Framework.org$scalatest$tools$Framework$$runSuite(TaskDef, String, Suite,
ClassLoader, SuiteSortingReporter, Tracker, EventHandler, Set<String>,
Set<String>, Selector[], boolean, ConfigMap, SummaryCounter,
Option<ScalaTestStatefulStatus>, LinkedBlockingQueue<Status>, Logger[],
boolean, boolean, boolean, boolean, boolean, boolean, boolean, boolean,
boolean, boolean, ExecutorService) line: 318
Framework$ScalaTestTask.execute(EventHandler, Logger[]) line: 513
ForkMain$Run.lambda$runTest$1(Task, ObjectOutputStream, Logger[]) line:

304

1574877131.call() line: not available
FutureTask<V>.run() line: 266
ThreadPoolExecutor.runWorker(ThreadPoolExecutor$Worker) line: 1149
ThreadPoolExecutor$Worker.run() line: 624
Thread.run() line: 748
=====




       Andy


On 25/03/2020 11:08, Jean-Marc Vanel wrote:

I investigated on that.
Now my test suite passes without Jena TDB trouble. I just added this in
Scala SBT :
parallelExecution in Test := false

I didn't know that parallel execution was the default.
So apparently this was an issue with 2 processes accessing the same

TDB .

Normally this is detected and an explicit Jena TDB message appears, but

not

this time.

Jean-Marc Vanel
<



http://semantic-forms.cc:9112/display?displayuri=http://jmvanel.free.fr/jmv.rdf%23me


+33 (0)6 89 16 29 52
Twitter: @jmvanel , @jmvanel_fr ; chat: irc://

irc.freenode.net#eulergui

    Chroniques jardin
<



http://semantic-forms.cc:1952/history?uri=http%3A%2F%2Fdbpedia.org%2Fresource%2FChronicle




Le mar. 24 mars 2020 à 18:56, Jean-Marc Vanel <

jeanmarc.va...@gmail.com>

a

écrit :


Hi Jena users!

After changing my test suite in* semantic_forms
<http://semantic-forms.cc:9112/ldp/semantic_forms>*  , I get
consistently a thread blocked forever. I upgraded to latest Jena

Version =

"3.14.0" , and happily my Web application seems to be working fine on

my

laptop, and the test suite sort of works, that is running tests one by

one.

In my test suite I used to recursively delete the TDB directories

after

each test. What changed is that instead I kept the same TDB

directories

for

each test, calling close() after each test.
This is with TDB 1. Is another unended transaction the issue? Then how

to

discover that ?

Thread [pool-1-thread-1-ScalaTest-running-TestRDFCache] (Suspended)
Unsafe.park(boolean, long) line: not available [native method]
LockSupport.park(Object) line: 175
Semaphore$FairSync(AbstractQueuedSynchronizer).parkAndCheckInterrupt()
line: 836




Semaphore$FairSync(AbstractQueuedSynchronizer).doAcquireSharedInterruptibly(int)

line: 997




Semaphore$FairSync(AbstractQueuedSynchronizer).acquireSharedInterruptibly(int)

line: 1304
Semaphore.acquire() line: 312

* TransactionManager.acquireWriterLock(boolean) line: 616 *
TransactionManager.beginInternal(TxnType, TxnType, String) line: 358
TransactionManager.begin(TxnType, String) line: 343
StoreConnection.begin(TxnType, String) line: 128
StoreConnection.begin(TxnType) line: 108
DatasetGraphTransaction.begin(TxnType) line: 169
DatasetGraphTransaction.begin(ReadWrite) line: 162
DatasetImpl.begin(ReadWrite) line: 122
JenaDatasetStore.$anonfun$rw$1(Dataset, Function0) line: 23


Jean-Marc Vanel
<



http://semantic-forms.cc:9112/display?displayuri=http://jmvanel.free.fr/jmv.rdf%23me



+33 (0)6 89 16 29 52
Twitter: @jmvanel , @jmvanel_fr ; chat: irc://

irc.freenode.net#eulergui

    Chroniques jardin
<



http://semantic-forms.cc:1952/history?uri=http%3A%2F%2Fdbpedia.org%2Fresource%2FChronicle

Reply via email to