Re: BlockException: No such block

[Andy Seaborne]

On 27/02/14 13:50, Steve Groeger wrote:
Hi Andy,
All the latest tests I was running were setting SystemTDB.NullOut = true;

I will add the code you specified to our tests and re-run them and will
send you our results shortly.

Thanks - if send the code as well, I'll try to run it at my end.  Even 
though I wasn't managing to catch this nearly as frequently as you, this 
new situation might be better.

        Andy


Thanks
Steve Groeger
IBM WebSphere Service Registry Development
MP 211,  Hursley
Tel: (44) 1962 816911  Mobex: 279990  Mobile: 07718 517 129
Fax (44) 1962 816800
Lotus Notes: Steve Groeger/UK/IBM
Internet: groe...@uk.ibm.com



From:   Andy Seaborne <a...@apache.org>
To:     users@jena.apache.org,
Date:   27/02/2014 13:36
Subject:        Re: BlockException: No such block



Hi Steve,

Thanks for the update.

On 25/02/14 15:15, Steve Groeger wrote:
Hi Andy,

I picked up a snapshot of 2.11.2 and tried running our test using these
libraries.

Adding the changes you suggested seems to have stopped the exception "
BlockException: No such block", the but we are now seeing other
symptoms,
which we saw previously but not as frequently as we are now.
The change in behaviour seems to be result of the SystemTDB.NullOut =
true
; as I also tried various version of the listStatements and this gave
the
same results, What does the SystemTDB.NullOut=true do?

NullOut makes the B+Tree code overwrite areas of blocks that are now
unused because of a delete with some value (FillByte = 0xFF).  In normal
operation, it just leaves it as junk, but of NullOut is set it goes and
sets the bytes to a specific value, and one which does not make sense.

I would not expect it to change the frequency of errors, but it would
change the symptom.

What we are now seeing is data not being returned correctly.
We initially populate the database with a Resource with 1000 Properties,
then in the ReaderThread we query the database, using the following:

              StmtIterator found = model.listStatements((Resource)null,
(Property)null, (RDFNode)null);

we are expecting to get back 1000 statements when we perform the

              count = found.toList().size();

however we occasionally get a batch of results that show 0 or non 1000
statements being returned, then after several more reads this goes back
to
getting 1000 statements and everything continues as normal.

I'm not clear whether that is with or without NullOut set true.


We need to determine why we occasionally dont get the 1000 statements
returned that we are expecting. Any advice you caould provide would be
great.

It looks as if under some unusual conditions, either the B+Tree branch
node or leaf node is not being updated correctly for deletions.
("unusual" as this code is not new and for anyone doing deletes, heavily
used).

That gives us something to go on.

Could you please add some code that not only counts the StmtIterator but
also touches every part of the statement.  Execute with

SystemTDB.NullOut=true

There is some delayed evaluation under the covers and this forces that
evaluation.

          StmtIterator found = ....
          long count = 0 ;
          while(found.hasNext()) {
              count++ ;
              Statement s = found.next();
              try {
                  Node subj = s.getSubject().asNode() ;
                  Node pred = s.getPredicate().asNode() ;
                  Node obj = s.getObject().asNode() ;
              } catch (Exception ex) {
                  ex.printStackTrace(System.err) ;
                  System.exit(99) ;
              }
          }

Warning - this may slow the execution down, or not, depending on cache
efficiencies.

                  Andy


Thanks
Steve Groeger
IBM WebSphere Service Registry Development
MP 211,  Hursley
Tel: (44) 1962 816911  Mobex: 279990  Mobile: 07718 517 129
Fax (44) 1962 816800
Lotus Notes: Steve Groeger/UK/IBM
Internet: groe...@uk.ibm.com



Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number
741598.
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU