Re: Large flat commit problems

2013-02-26 Thread Thomas Mueller
Hi, Large transactions: I think we didn't define this as a strict requirement. I'm not aware we got into big troubles with Jackrabbit 2.x where this is not supported. For me, this is still a nice to have. But of course it's something we should test and try to achieve (and resolve problems if we fi

Re: Large flat commit problems

2013-02-26 Thread Jukka Zitting
Hi, On Tue, Feb 26, 2013 at 12:04 PM, Thomas Mueller wrote: > Large transactions: I think we didn't define this as a strict requirement. It's probably not the most important thing for Oak to achieve, but we did list it as a goal in http://wiki.apache.org/jackrabbit/Goals%20and%20non%20goals%20fo

Re: Large flat commit problems

2013-02-26 Thread Tommaso Teofili
On 26/feb/2013, at 11:12, Jukka Zitting wrote: > Hi, > > On Tue, Feb 26, 2013 at 12:04 PM, Thomas Mueller wrote: >> Large transactions: I think we didn't define this as a strict requirement. > > It's probably not the most important thing for Oak to achieve, but we > did list it as a goal in >

Re: Large flat commit problems

2013-02-26 Thread Thomas Mueller
Hi, I created OAK-656 "Large number of child nodes not working well with orderable node types". Until this is fixed, I guess we could use nt:folder (which is unordered). Regards, Thomas On 2/26/13 11:22 AM, "Tommaso Teofili" wrote: > >On 26/feb/2013, at 11:12, Jukka Zitting wrote: > >> Hi, >

RE: Large flat commit problems

2013-02-26 Thread Marcel Reutegger
> > I didn't analyze the results, but could the problem be orderable child > > nodes? > > That may well be, in the benchmark code I don't explicitly specify a > non-orderable node type so it defaults to the orderable > nt:unstructured. I think we should make a compromise here. either you get orde

RE: Large flat commit problems

2013-02-26 Thread Marcel Reutegger
> I didn't analyze the results, but could the problem be orderable child > nodes? Currently, oak-core stores a property ":childOrder". no, the problem is how oak-core detects changes between two node state revisions. for a node with many child nodes in two revisions, oak-core currently loads all c

Re: Large flat commit problems

2013-02-26 Thread Chetan Mehrotra
I modified the importer logic to use a custom nodeType similar to SlingFolder (no orderable nodes) and following are the results Segment MK -- 05:30:31 {benchmark} ~/git/apache/jackrabbit-oak$ java -DOAK-652=true -jar oak-run/target/oak-run-0.7-SNAPSHOT.jar benchmark --wikipedia=

Re: Large flat commit problems

2013-02-27 Thread Jukka Zitting
Hi, On Tue, Feb 26, 2013 at 2:19 PM, Chetan Mehrotra wrote: > I modified the importer logic to use a custom nodeType similar to > SlingFolder (no orderable nodes) and following are the results Thanks! It indeed looks like the orderability is an issue here. With the oak:unstructured type I added

Re: Large flat commit problems

2013-04-26 Thread Jukka Zitting
Hi, On Wed, Feb 27, 2013 at 12:24 PM, Jukka Zitting wrote: > Added 167000 pages in 467 seconds (2.80ms/page) > Imported 167404 pages in 1799 seconds (10.75ms/page) Here's an update on the latest status with the Wikipedia import benchmark: $ java -Xmx1500m -jar oak-run/target/oak-run

Re: Large flat commit problems

2013-04-26 Thread Jukka Zitting
Hi, On Fri, Apr 26, 2013 at 3:15 PM, Jukka Zitting wrote: > Imported 171382 pages in 355 seconds (2.07ms/page) For comparison, here's the result if I change the benchmark code to call save() after every 1k pages: Imported 171382 pages in 1154 seconds (6.74ms/page) BR, Jukka Zitting

Re: Large flat commit problems

2013-04-29 Thread Lukas Eder
Hi, On 4/26/13 2:15 PM, "Jukka Zitting" wrote: >Hi, > >On Wed, Feb 27, 2013 at 12:24 PM, Jukka Zitting >wrote: >> Added 167000 pages in 467 seconds (2.80ms/page) >> Imported 167404 pages in 1799 seconds (10.75ms/page) > >Here's an update on the latest status with the Wikipedia import be

Re: Large flat commit problems

2013-04-29 Thread Jukka Zitting
Hi, On Mon, Apr 29, 2013 at 10:17 AM, Lukas Eder wrote: > Do comparisons with Jackrabbit exist? Not for this particular benchmark, since Jackrabbit 2.x is unable to deal with such a large transaction (~400MB of non-binary content) or the flat hierarchy (170k child nodes). Some Jackrabbit compar

Re: Large flat commit problems

2013-05-30 Thread Jukka Zitting
Hi, On Fri, Apr 26, 2013 at 3:15 PM, Jukka Zitting wrote: > On Wed, Feb 27, 2013 at 12:24 PM, Jukka Zitting > wrote: >> Added 167000 pages in 467 seconds (2.80ms/page) >> Imported 167404 pages in 1799 seconds (10.75ms/page) > > Added 171000 pages in 166 seconds (0.97ms/page) > I

RE: Large flat commit problems

2013-05-30 Thread Marcel Reutegger
> And with the TarMK: > > Added 171000 pages in 36 seconds (0.21ms/page) > Imported 171382 pages in 54 seconds (0.32ms/page) > [...] > Traversed 171382 pages in 2 seconds (0.01ms/page) > > I particularly like that last line. :-) very nice! regards marcel

Re: Large flat commit problems

2013-05-30 Thread Thomas Mueller
Hi, Great! If we continue like this, the TarMK will next month have negative values :-) Regards, Thomas On 5/30/13 2:45 PM, "Jukka Zitting" wrote: >Hi, > >On Fri, Apr 26, 2013 at 3:15 PM, Jukka Zitting >wrote: >> On Wed, Feb 27, 2013 at 12:24 PM, Jukka Zitting >> wrote: >>> Added 167000