Checkpoint might not get cleaned up in case of abrupt shutdown and thus stop GC for 3 years!

2014-09-09 Thread Chetan Mehrotra
Currently AsyncIndexUpdate creates a checkpoint with a lifetime of 3 years. Post index complete the checkpoint is moved to point to revision upto which indexing is done and previous checkpoint is released. Now if a system is killed when a index update is in progress particularly in run method wher

Re: segment size differs

2014-09-09 Thread Marcel Reutegger
Hi, with maven we set a custom -Dupdate.limit=100 The default for the SegmentNodeBuilder is 1. Maybe this is causing different segment sizes? Regards Marcel On 09/09/14 16:27, "Martin Böttcher" wrote: >Hi, > >I'm not sure what the "execution mode" is. In IntelliJ Idea it makes no >differ

buildbot success in ASF Buildbot on oak-trunk

2014-09-09 Thread buildbot
The Buildbot has detected a restored build on builder oak-trunk while building ASF Buildbot. Full details are available at: http://ci.apache.org/builders/oak-trunk/builds/506 Buildbot URL: http://ci.apache.org/ Buildslave for this Build: bb-vm_ubuntu Build Reason: scheduler Build Source Stamp:

Re: segment size differs

2014-09-09 Thread Martin Böttcher
Hi, I'm not sure what the "execution mode" is. In IntelliJ Idea it makes no difference whether I RUN or DEBUG the test case. The size if always 2576. Speaking of maven the size (2720) does not depend on the target ("test" or "clean install"). If I misunderstood the question please specify what

Re: segment size differs

2014-09-09 Thread Chetan Mehrotra
> gives me a segment with the size of 2576 bytes if I run it using IntelliJ > Idea and 2720 bytes if I use "mvn test". Were you running the test in debug mode or normal execution mode? Chetan Mehrotra On Tue, Sep 9, 2014 at 7:16 PM, Martin Böttcher wrote: > Hi, > > I noticed that running the s

segment size differs

2014-09-09 Thread Martin Böttcher
Hi, I noticed that running the same code (inside a test case) leads to different results depending on how the test is started. The code: --- NodeStore store = new SegmentNodeStore(storeS); NodeBuilder rootbuilder = store.getRoot().builder(); NodeBuilder b = rootbuilder.child("store"); NodeBuilder

Re: TarMK Failover does not recover from some error situations

2014-09-09 Thread Alex Parvulescu
Hi, I agree with the FileStoreCorruptException idea. The change helps clarify the problem (missing segment) and will not affect any existing code. best, alex On Tue, Sep 9, 2014 at 2:25 PM, Martin Böttcher < martin.boettc...@greenbytes.de> wrote: > Hi, > > I'm working on the TarMK failover pa

TarMK Failover does not recover from some error situations

2014-09-09 Thread Martin Böttcher
Hi,I'm working on the TarMK failover package and trying to make it more robust and reliable. I noticed that there are still some situations in which the current implementation will run into persistent problems. It's not obvious what the "right" strategy is so let me paint the picture quite quickly:

Re: Using BlobStore by default with SegmentNodeStore

2014-09-09 Thread Chetan Mehrotra
I have updated OAK-2082 with test run results. Looking at the result I think FDS does provide a benefit in terms of lesser storage space. Putting Lucene index on file system provides best storage efficiency but then it would not work once we have TarMK failover implemented. Chetan Mehrotra On Tu

Re: Using BlobStore by default with SegmentNodeStore

2014-09-09 Thread Thomas Mueller
Hi, In addition or instead of using the BlobStore, we could store the Lucene index to the filesystem (persistence = file, path = ...). But I would probably only do that on a case-by-case basis. I think it would reduce, but not solve the compaction problem. Some numbers from a test repository I ha

Re: Using BlobStore by default with SegmentNodeStore

2014-09-09 Thread Chetan Mehrotra
On Fri, Sep 5, 2014 at 3:26 PM, Michael Dürig wrote: > I'd leave the default as it is for Oak as this has the beauty of simplicity. > We could just change it for applications where we know that the inline > storing of binaries is troublesome. Ack. Yes default setup should not be modified > OTOH