Hi Andy,
I ran the test again with the additional code you requested. It took a
long time to fail, over 4.3 million reads but it eventually did. The error
we saw was:
reader3: Models do not compare expected size: 1000 was: 778
java.lang.IllegalArgumentException: ObjectFile.read[nodes]: Bad read: -1
at com.hp.hpl.jena.tdb.base.objectfile.ObjectFileStorage.read(
ObjectFileStorage.java:311)
at com.hp.hpl.jena.tdb.lib.NodeLib.fetchDecode(NodeLib.java:79)
at
com.hp.hpl.jena.tdb.nodetable.NodeTableNative.readNodeFromTable(
NodeTableNative.java:178)
at
com.hp.hpl.jena.tdb.nodetable.NodeTableNative._retrieveNodeByNodeId(
NodeTableNative.java:103)
at com.hp.hpl.jena.tdb.nodetable.NodeTableNative.getNodeForNodeId(
NodeTableNative.java:74)
at
com.hp.hpl.jena.tdb.nodetable.NodeTableCache._retrieveNodeByNodeId(
NodeTableCache.java:103)
at com.hp.hpl.jena.tdb.nodetable.NodeTableCache.getNodeForNodeId(
NodeTableCache.java:74)
at
com.hp.hpl.jena.tdb.nodetable.NodeTableWrapper.getNodeForNodeId(
NodeTableWrapper.java:55)
at com.hp.hpl.jena.tdb.nodetable.NodeTableInline.getNodeForNodeId(
NodeTableInline.java:67)
at
com.hp.hpl.jena.tdb.transaction.NodeTableTrans.getNodeForNodeId(
NodeTableTrans.java:112)
at
com.hp.hpl.jena.tdb.nodetable.NodeTableWrapper.getNodeForNodeId(
NodeTableWrapper.java:55)
at com.hp.hpl.jena.tdb.nodetable.NodeTableInline.getNodeForNodeId(
NodeTableInline.java:67)
at
com.hp.hpl.jena.tdb.nodetable.NodeTableWrapper.getNodeForNodeId(
NodeTableWrapper.java:55)
at com.hp.hpl.jena.tdb.lib.TupleLib.quad(TupleLib.java:161)
at com.hp.hpl.jena.tdb.lib.TupleLib.quad(TupleLib.java:153)
at com.hp.hpl.jena.tdb.lib.TupleLib.access$100(TupleLib.java:45)
at com.hp.hpl.jena.tdb.lib.TupleLib$4.convert(TupleLib.java:87)
at com.hp.hpl.jena.tdb.lib.TupleLib$4.convert(TupleLib.java:83)
at org.apache.jena.atlas.iterator.Iter$4.next(Iter.java:322)
at org.apache.jena.atlas.iterator.Iter$4.next(Iter.java:322)
at org.apache.jena.atlas.iterator.Iter.next(Iter.java:920)
at com.hp.hpl.jena.util.iterator.WrappedIterator.next(
WrappedIterator.java:94)
at com.hp.hpl.jena.util.iterator.Map1Iterator.next(
Map1Iterator.java:45)
at com.hp.hpl.jena.util.iterator.WrappedIterator.next(
WrappedIterator.java:94)
at com.hp.hpl.jena.rdf.model.impl.StmtIteratorImpl.next(
StmtIteratorImpl.java:42)
at com.hp.hpl.jena.rdf.model.impl.StmtIteratorImpl.next(
StmtIteratorImpl.java:30)
at com.ibm.jenatest.ReaderThread.test(ReaderThread.java:68)
at com.ibm.jenatest.GenericTestThread.run(
GenericTestThread.java:39)
Hope this allows you to move closer to determining the issue.
Thanks
Steve Groeger
From: Andy Seaborne <a...@apache.org>
To: users@jena.apache.org,
Date: 27/02/2014 15:09
Subject: Re: BlockException: No such block
On 27/02/14 13:50, Steve Groeger wrote:
> Hi Andy,
>
> All the latest tests I was running were setting SystemTDB.NullOut =
true;
>
> I will add the code you specified to our tests and re-run them and will
> send you our results shortly.
Thanks - if send the code as well, I'll try to run it at my end. Even
though I wasn't managing to catch this nearly as frequently as you, this
new situation might be better.
Andy
>
> Thanks
> Steve Groeger
> IBM WebSphere Service Registry Development
> MP 211, Hursley
> Tel: (44) 1962 816911 Mobex: 279990 Mobile: 07718 517 129
> Fax (44) 1962 816800
> Lotus Notes: Steve Groeger/UK/IBM
> Internet: groe...@uk.ibm.com
>
>
>
> From: Andy Seaborne <a...@apache.org>
> To: users@jena.apache.org,
> Date: 27/02/2014 13:36
> Subject: Re: BlockException: No such block
>
>
>
> Hi Steve,
>
> Thanks for the update.
>
> On 25/02/14 15:15, Steve Groeger wrote:
>> Hi Andy,
>>
>> I picked up a snapshot of 2.11.2 and tried running our test using these
>> libraries.
>>
>> Adding the changes you suggested seems to have stopped the exception "
>> BlockException: No such block", the but we are now seeing other
> symptoms,
>> which we saw previously but not as frequently as we are now.
>> The change in behaviour seems to be result of the SystemTDB.NullOut =
> true
>> ; as I also tried various version of the listStatements and this gave
> the
>> same results, What does the SystemTDB.NullOut=true do?
>
> NullOut makes the B+Tree code overwrite areas of blocks that are now
> unused because of a delete with some value (FillByte = 0xFF). In normal
> operation, it just leaves it as junk, but of NullOut is set it goes and
> sets the bytes to a specific value, and one which does not make sense.
>
> I would not expect it to change the frequency of errors, but it would
> change the symptom.
>
>> What we are now seeing is data not being returned correctly.
>> We initially populate the database with a Resource with 1000
Properties,
>> then in the ReaderThread we query the database, using the following:
>>
>> StmtIterator found = model.listStatements((Resource)null,
>> (Property)null, (RDFNode)null);
>>
>> we are expecting to get back 1000 statements when we perform the
>>
>> count = found.toList().size();
>>
>> however we occasionally get a batch of results that show 0 or non 1000
>> statements being returned, then after several more reads this goes back
> to
>> getting 1000 statements and everything continues as normal.
>
> I'm not clear whether that is with or without NullOut set true.
>
>>
>> We need to determine why we occasionally dont get the 1000 statements
>> returned that we are expecting. Any advice you caould provide would be
>> great.
>
> It looks as if under some unusual conditions, either the B+Tree branch
> node or leaf node is not being updated correctly for deletions.
> ("unusual" as this code is not new and for anyone doing deletes, heavily
> used).
>
> That gives us something to go on.
>
> Could you please add some code that not only counts the StmtIterator but
> also touches every part of the statement. Execute with
>
> SystemTDB.NullOut=true
>
> There is some delayed evaluation under the covers and this forces that
> evaluation.
>
> StmtIterator found = ....
> long count = 0 ;
> while(found.hasNext()) {
> count++ ;
> Statement s = found.next();
> try {
> Node subj = s.getSubject().asNode() ;
> Node pred = s.getPredicate().asNode() ;
> Node obj = s.getObject().asNode() ;
> } catch (Exception ex) {
> ex.printStackTrace(System.err) ;
> System.exit(99) ;
> }
> }
>
> Warning - this may slow the execution down, or not, depending on cache
> efficiencies.
>
> Andy
>
>>
>> Thanks
>> Steve Groeger
>> IBM WebSphere Service Registry Development
>> MP 211, Hursley
>> Tel: (44) 1962 816911 Mobex: 279990 Mobile: 07718 517 129
>> Fax (44) 1962 816800
>> Lotus Notes: Steve Groeger/UK/IBM
>> Internet: groe...@uk.ibm.com
>
>
>
> Unless stated otherwise above:
> IBM United Kingdom Limited - Registered in England and Wales with number
> 741598.
> Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6
3AU
>
Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number
741598.
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
