Some more benchmarks

2013-03-27 Thread Jukka Zitting
Hi, Here's a few more simple benchmark results to show where we are: Apache Jackrabbit Oak 0.7-SNAPSHOT # ReadPropertyTest min 10% 50% 90% max N Jackrabbit31 31 33 92 1211470 Oak-Default

Re: Some more benchmarks

2013-03-27 Thread Thomas Mueller
Hi, Thanks! The SmallFileWriteTest is quite slow for Oak-Mongo: 6 times slower than Jackrabbit on average, as far as I see. It seems to be, at least partially, a problem of the AbstractBlobStore. I will have a look. Regards, Thomas On 3/27/13 10:41 AM, "Jukka Zitting" wrote: >Hi, > >Here's

Re: Some more benchmarks

2013-03-27 Thread Jukka Zitting
Hi, On Wed, Mar 27, 2013 at 11:41 AM, Jukka Zitting wrote: > # ReadPropertyTest min 10% 50% 90% max > N > Jackrabbit31 31 33 92 121 > 1470 > Oak-Default 56 58 61 120

Re: Some more benchmarks

2013-03-27 Thread Michael Dürig
On 27.3.13 11:54, Jukka Zitting wrote: Hi, Do we need to explicitly validate all paths that get passed to us? Especially in cases like getProperty(), where in the vast majority of the cases the given path matches an existing property (whose path by definition is valid), it would make more s

Re: Some more benchmarks

2013-03-27 Thread Jukka Zitting
Hi, On Wed, Mar 27, 2013 at 2:12 PM, Michael Dürig wrote: > IIUC you propose to not validate paths in the read case but rely on the > downstream code to fail. Might be worth a try. However we'd need different > path parsing then for the read an the write case since circumventing path > validation

Re: Some more benchmarks

2013-03-27 Thread Michael Dürig
On 27.3.13 12:21, Jukka Zitting wrote: Hi, On Wed, Mar 27, 2013 at 2:12 PM, Michael Dürig wrote: IIUC you propose to not validate paths in the read case but rely on the downstream code to fail. Might be worth a try. However we'd need different path parsing then for the read an the write case

Re: Some more benchmarks

2013-03-27 Thread Jukka Zitting
Hi, On Wed, Mar 27, 2013 at 2:32 PM, Michael Dürig wrote: > That's right. The easiest thing is to try it out, remove pre-emptive path > validation and see what breaks. I have the vague memory that there were some > overly picky TCK tests which required us to put this upfront validation in. > Howe

Re: Some more benchmarks

2013-03-27 Thread Jukka Zitting
Hi, On Wed, Mar 27, 2013 at 1:54 PM, Jukka Zitting wrote: > Quick benchmarking of the Oak-Default run shows > NamePathMapperImpl.getOakPath() calling JcrPathParser.validate() > taking about 20% of the time in this test. Updated numbers after the latest OAK-108 change: # ReadPropertyTest

Re: Some more benchmarks

2013-03-27 Thread Jukka Zitting
Hi, On Wed, Mar 27, 2013 at 11:41 AM, Jukka Zitting wrote: > Here's a few more simple benchmark results to show where we are: Some notes to help read and produce benchmark results like these. The oak-run jar that you can find under oak-run/target has a "benchmark" mode that produces these resul

Re: Some more benchmarks

2013-03-28 Thread Michael Dürig
On 27.3.13 14:41, Jukka Zitting wrote: Profiling the getProperty calls shows the following distribution of time spent: NodeImpl.getProperty() 61% NodeDelegate.getProperty() via perform() 31% ItemImpl.isStale() via checkStatus() 8% other stuff The status check would

Re: Some more benchmarks

2013-03-28 Thread Michael Dürig
On 27.3.13 14:41, Jukka Zitting wrote: Drilling down to the NodeDelegate.getProperty() method, we have the following distribution of time: NodeDelegate.getProperty() 95% NodeDelegate.getChildLocation() 5% TreeImpl.internalGetProperty() via NodeLocation.getProperty() See wh

Re: Some more benchmarks

2013-04-02 Thread Michael Dürig
On 28.3.13 15:19, Angela Schreiber wrote: hi michael With the resolution of OAK-690, I made tree instances stable across save and refresh operations. does that mean that the AuthorizableImpl could hold a Tree instance instead of re-accessing it over and over again using a lookup by id? if t

Re: Some more benchmarks

2013-04-02 Thread Angela Schreiber
oh... already fixed it before lunch and committed modifications before re-checking mails (using OAK-690 and an appropriate comment). thanks anyway... test passed and i assume it's fine. the internal method AuthorizableImpl#getTree will now return the Tree associated with the authorizable as lon

Re: Some more benchmarks

2013-04-26 Thread Jukka Zitting
Hi, On Wed, Mar 27, 2013 at 11:41 AM, Jukka Zitting wrote: > Here's a few more simple benchmark results to show where we are: Updated numbers with latest Oak: # ReadPropertyTest min 10% 50% 90% max N Jackrabbit34 35 3

Re: Some more benchmarks

2013-04-29 Thread Lukas Eder
Hello, I'm interested in estimating performance (and load) impacts of ACL checking on read access. I'm specifically interested in a comparison where paths like /a, /a/b, /a/b/c, /a/.../y/z are accessed, and ACL has to be evaluated "upwards" on the path. Since such a test is more high-level and may

Re: Some more benchmarks

2013-04-29 Thread Jukka Zitting
Hi, On Mon, Apr 29, 2013 at 10:17 AM, Lukas Eder wrote: > Are there any test results available with respect to ACL, comparing > Jackrabbit with Oak? Not yet. See the o.a.j.oak.benchmark package in oak-run for some of the existing benchmarks I've been using so far. It should be fairly straightfor

Re: Some more benchmarks

2013-05-31 Thread Jukka Zitting
Hi, On Fri, Apr 26, 2013 at 2:12 PM, Jukka Zitting wrote: > On Wed, Mar 27, 2013 at 11:41 AM, Jukka Zitting > wrote: >> Here's a few more simple benchmark results to show where we are: > > Updated numbers with latest Oak: And another one: Apache Jackrabbit Oak 0.9-SNAPSHOT # ReadPrope

RE: Some more benchmarks

2013-05-31 Thread Michael C. Moore
ginal Message- From: Jukka Zitting [mailto:jukka.zitt...@gmail.com] Sent: Friday, May 31, 2013 8:14 AM To: Oak devs Subject: Re: Some more benchmarks Hi, On Fri, Apr 26, 2013 at 2:12 PM, Jukka Zitting wrote: > On Wed, Mar 27, 2013 at 11:41 AM, Jukka Zitting > wrote: >> Here&#x

Re: Some more benchmarks

2013-05-31 Thread Jukka Zitting
Hi, On Fri, May 31, 2013 at 3:52 PM, Michael C. Moore wrote: > Can you briefly explain the test results or point me to a wiki or link that > has the explanation? I just committed the description to a README file, see https://github.com/apache/jackrabbit-oak/blob/trunk/oak-run/README.md BR, Ju

RE: Some more benchmarks

2013-05-31 Thread Michael C. Moore
Great, thanks! -Original Message- From: Jukka Zitting [mailto:jukka.zitt...@gmail.com] Sent: Friday, May 31, 2013 9:28 AM To: Oak devs Subject: Re: Some more benchmarks Hi, On Fri, May 31, 2013 at 3:52 PM, Michael C. Moore wrote: > Can you briefly explain the test results or point

Re: Some more benchmarks

2013-06-03 Thread Thomas Mueller
Hi, A bit weird is, when I run the tests separately I get different numbers: java -mx1g -jar target/oak-run-0.9-SNAPSHOT.jar benchmark SmallFileReadTest Oak-Tar # SmallFileReadTest min 10% 50% 90% max N Oak-Tar 53 54 55 57

Re: Some more benchmarks

2013-06-03 Thread Jukka Zitting
Hi, On Mon, Jun 3, 2013 at 11:09 AM, Thomas Mueller wrote: > A bit weird is, when I run the tests separately I get different numbers: The results depend on the hardware you're using, so in general numbers from two different environments are not directly comparable. > In your case, the N was 304

Re: Some more benchmarks

2013-06-03 Thread Thomas Mueller
Hi, I was not talking about differences in hardware. I know using different hardware will result in different numbers. I was worried about results would be different if you run one test alone versus if you run all tests. That would indicate a problem in the benchmark (framework) itself. But luck

Re: Some more benchmarks

2013-06-03 Thread Jukka Zitting
Hi, On Mon, Jun 3, 2013 at 12:51 PM, Thomas Mueller wrote: > I was not talking about differences in hardware. I know using different > hardware will result in different numbers. > > I was worried about results would be different if you run one test alone > versus if you run all tests. That would

Re: Some more benchmarks

2013-06-03 Thread Thomas Mueller
Hi, >At least the TarMK should have no problems with the 100 child nodes >(see the Wikipedia import test results :-). Yes, I also thought 100 child nodes shouldn't be a problem. The profiling data I have so far doesn't show a clear bottleneck. I really wonder what the problem is in this case. Re

Re: Some more benchmarks

2013-06-06 Thread Jukka Zitting
Hi, On Fri, May 31, 2013 at 3:14 PM, Jukka Zitting wrote: > It looks like we have a performance regression in ReadPropertyTest. > Quick profiling shows a lot of the time seems to be going to > MemoryNodeBuilder$ConnectedHead.update(), which is weird since we're > only reading and thus the related

Re: Some more benchmarks

2013-07-02 Thread Jukka Zitting
Hi, On Fri, May 31, 2013 at 3:14 PM, Jukka Zitting wrote: > On Fri, Apr 26, 2013 at 2:12 PM, Jukka Zitting > wrote: >> On Wed, Mar 27, 2013 at 11:41 AM, Jukka Zitting >> wrote: >>> Here's a few more simple benchmark results to show where we are: >> >> Updated numbers with latest Oak: > > And

Re: Some more benchmarks

2013-07-03 Thread Thomas Mueller
Hi, Thanks a lot! > I've only included the 90th percentile I usually look at "N" first :-) There is one strange result: SmallFileWriteTest; Oak-Segment: 90%=257, max=14763 - Maybe the warmup phase is too short, or the test isn't that great? As for SmallFileReadTest and SmallFileWriteTest with

Re: Some more benchmarks

2013-07-03 Thread Jukka Zitting
Hi, On Wed, Jul 3, 2013 at 11:22 AM, Thomas Mueller wrote: >> I've only included the 90th percentile > > I usually look at "N" first :-) It's also a good measure. I like the 90th percentile better as it filters out outliers that may otherwise weight pretty heavily on the total or average executi

Re: Some more benchmarks

2013-07-03 Thread Jukka Zitting
Hi, On Wed, Jul 3, 2013 at 11:54 AM, Jukka Zitting wrote: > On Wed, Jul 3, 2013 at 11:22 AM, Thomas Mueller wrote: >> I usually look at "N" first :-) > > It's also a good measure. Actually not that good, as only the lower limit on the amount of time over which those N iterations happen is defin

Re: Some more benchmarks

2013-07-03 Thread Thomas Mueller
Hi, >I don't know what's dragging the performance in the SmallFileReadTest, I think it's because all binaries are loaded from the backend (no caching). I bumped the blob cache size from 8 MB to 16 MB, let's see if this helps. Regards, Thomas

Re: Some more benchmarks

2013-07-04 Thread Jukka Zitting
Hi, On Wed, Jul 3, 2013 at 5:34 PM, Thomas Mueller wrote: > I think it's because all binaries are loaded from the backend (no > caching). I bumped the blob cache size from 8 MB to 16 MB, let's see if > this helps. Yes, that could be it (I'll run the tests again soon to confirm). The working set

Re: Some more benchmarks

2013-07-04 Thread Jukka Zitting
Hi, On Thu, Jul 4, 2013 at 1:12 PM, Jukka Zitting wrote: > On Wed, Jul 3, 2013 at 5:34 PM, Thomas Mueller wrote: >> I think it's because all binaries are loaded from the backend (no >> caching). I bumped the blob cache size from 8 MB to 16 MB, let's see if >> this helps. > > Yes, that could be i

Re: Some more benchmarks

2013-09-24 Thread Jukka Zitting
Hi, Updating this thread with the latest numbers. No major changes on these benchmarks: # ReadPropertyTest 90% Jackrabbit48 Oak-Default 39 Oak-Mongo 41 Oak-Segment 41

Re: Some more benchmarks

2013-09-24 Thread Jukka Zitting
Hi, On Tue, Sep 24, 2013 at 8:08 PM, Jukka Zitting wrote: > I'll add a few more benchmarks to my test script. Here are results for three more benchmarks: # SetPropertyTest 90% Jackrabbit 740 Oak-Default 916 Oak-Mongo

Re: Some more benchmarks

2013-09-24 Thread Jukka Zitting
Hi, On Tue, Sep 24, 2013 at 10:47 PM, Jukka Zitting wrote: > The concurrent read and read/write test cases look like more attention > is needed on the test code, as it's currently hard to interpret the > results. I'll see what I can do there. It turns out most of the reported time was going to l

Re: Some more benchmarks

2013-09-28 Thread Jukka Zitting
Hi, On Tue, Sep 24, 2013 at 11:19 PM, Jukka Zitting wrote: > On Tue, Sep 24, 2013 at 10:47 PM, Jukka Zitting > wrote: >> The concurrent read and read/write test cases look like more attention >> is needed on the test code, as it's currently hard to interpret the >> results. I'll see what I can

Re: Some more benchmarks

2014-07-01 Thread Jukka Zitting
Hi, I'm resurrecting this thread with some new findings. I re-ran many of the benchmarks we've been following, pitting Jackrabbit 2.8.0 against Oak 1.0.1 with TarMK. The results look pretty nice: Summary (90%, lower is better) Benchmark Jackrabbit Oak-Tar --

Re: Some more benchmarks

2014-07-01 Thread Jukka Zitting
Hi, On Tue, Jul 1, 2014 at 9:38 AM, Jukka Zitting wrote: > I also tried including MongoMK results, but the benchmark got stuck in > ConcurrentReadTest. I'll re-try today and will file a bug if I can > reproduce the problem. I guess it was a transient problem. Here are the results with Oak-Mongo

Re: Some more benchmarks

2014-07-02 Thread Stefan Guggisberg
On Tue, Jul 1, 2014 at 8:37 PM, Jukka Zitting wrote: > Hi, > > On Tue, Jul 1, 2014 at 9:38 AM, Jukka Zitting wrote: >> I also tried including MongoMK results, but the benchmark got stuck in >> ConcurrentReadTest. I'll re-try today and will file a bug if I can >> reproduce the problem. > > I guess