Re: Using DBCursor in MongoDocumentStore

2014-03-20 Thread Chetan Mehrotra
> What do you mean by "wrap the current impl"? I thought to wrap the current impl which returns list to return an iterator based on that. However looking at Marcel's note on OAK-1567 and requirement to close the iterator I need to rethink this change a bit. Also current query method might not be

Re: Using DBCursor in MongoDocumentStore

2014-03-20 Thread Julian Reschke
On 2014-03-20 10:18, Chetan Mehrotra wrote: So I would refactor the logic to use Iterables and the required changes in MongoDocumentStore. For the RDB one I would just wrap current impl. Should be able to push the change once Load 19 is cut. ... What do you mean by "wrap the current impl"? Bes

Re: Using DBCursor in MongoDocumentStore

2014-03-20 Thread Chetan Mehrotra
So I would refactor the logic to use Iterables and the required changes in MongoDocumentStore. For the RDB one I would just wrap current impl. Should be able to push the change once Load 19 is cut. Would that work for you Julian? Chetan Mehrotra On Thu, Mar 20, 2014 at 2:39 PM, Julian Reschke w

Re: Using DBCursor in MongoDocumentStore

2014-03-20 Thread Julian Reschke
On 2014-03-20 10:02, Thomas Mueller wrote: Hi, For the database case, what we could do is return an iterator that internally chunks, that is: 1) run the query "select * from datastore where id > ? limit 1 order by id" 2) if the results set is empty, then iterator.hasNext is false 3) else re

RE: Using DBCursor in MongoDocumentStore

2014-03-20 Thread Marcel Reutegger
Hi, > > I think the best you can do with implementations that require an > > explicit release of resource is to perform some kind of batch loading > > ever N items with increasing offset. this is actually what the MongoDB > > Java driver does under the hood. hmm, I had another look at how mongo d

Re: Using DBCursor in MongoDocumentStore

2014-03-20 Thread Thomas Mueller
Hi, For the database case, what we could do is return an iterator that internally chunks, that is: 1) run the query "select * from datastore where id > ? limit 1 order by id" 2) if the results set is empty, then iterator.hasNext is false 3) else read this in memory and close the connection 4)

Re: Using DBCursor in MongoDocumentStore

2014-03-20 Thread Julian Reschke
On 2014-03-20 08:44, Marcel Reutegger wrote: Hi, I noticed that the BlobStore API recently acquired a similar interface through GarbageCollectableBlobStore. The current impl in RDBBlobStore just returns an iterator that wraps a result set, which works right now as the RDBBlobStore keeps holdin

RE: Using DBCursor in MongoDocumentStore

2014-03-20 Thread Marcel Reutegger
Hi, > I noticed that the BlobStore API recently acquired a similar interface > through GarbageCollectableBlobStore. > > The current impl in RDBBlobStore just returns an iterator that wraps a > result set, which works right now as the RDBBlobStore keeps holding the > Connection. > > I was plannin

Re: Using DBCursor in MongoDocumentStore

2014-03-18 Thread Julian Reschke
On 2014-03-18 18:52, Chetan Mehrotra wrote: Hi, I was looking into the support for Version GC (OAK-1341). For that I had a look at BlobReferenceIterator#loadBatchQuery which currently does a pagination over all Documents which have binaries. The DBCursor returned by Mongo [1] implements the Iter

Using DBCursor in MongoDocumentStore

2014-03-18 Thread Chetan Mehrotra
Hi, I was looking into the support for Version GC (OAK-1341). For that I had a look at BlobReferenceIterator#loadBatchQuery which currently does a pagination over all Documents which have binaries. The DBCursor returned by Mongo [1] implements the Iterator contract and would lazily fetch the next