Hi, >From the results published at [1]. Reversing the order shows up that the create time is quite variable and probably not a function of the number of nodes within a collection, but probably a function the activity and buffering within the Cassandra layer.
Reverse the order and the first 200 items written will have the lowest latency. Delete shows no difference in latency between 1K and 100M child nodes. That result is very encouraging, it means that performance is related the IO speed of the disks underneath Cassandra and not a function of the volume of data within Cassandra or the number of items within a collection. I think we have enough information, next is to implement access control. I'll start a new thread for that. Create Add 1 child node to a collection of 100M child nodes takes 26ms Add 1 child node to a collection of 10M child nodes takes 135ms Add 1 child node to a collection of 1M child nodes takes 109ms Add 1 child node to a collection of 100K child nodes takes 259ms Add 1 child node to a collection of 10K child nodes takes 225ms Add 1 child node to a collection of 1K child nodes takes 101ms Update Update 1 child node to a collection of 100M child nodes takes 44ms Update 1 child node to a collection of 10M child nodes takes 84ms Update 1 child node to a collection of 1M child nodes takes 258ms Update 1 child node to a collection of 100K child nodes takes 116ms Update 1 child node to a collection of 10K child nodes takes 231ms Update 1 child node to a collection of 1K child nodes takes 113ms Delete Delete 1 child node to a collection of 100M child nodes takes 17ms Delete 1 child node to a collection of 10M child nodes takes 11ms Delete 1 child node to a collection of 1M child nodes takes 7ms Delete 1 child node to a collection of 100K child nodes takes 7ms Delete 1 child node to a collection of 10K child nodes takes 6ms Delete 1 child node to a collection of 1K child nodes takes 7ms On 12 September 2013 05:06, Dishara Wijewardana <ddwijeward...@gmail.com> wrote: > Hi Ian, > Once you look at the results, let me know whether you want new tests run > results. Also please advice how to proceed further. > > > On Wed, Sep 11, 2013 at 7:59 PM, Dishara Wijewardana < > ddwijeward...@gmail.com> wrote: > >> Hi Ian, >> I attached the results.zip file to the JIRA which has the latest results >> as your expectation. I added new set of nodes in reverse order "F", "E" >> .."A". >> But here I came across in a issue (which I didn't came up last time, may >> be due to my heap size), that hector fails to execute batches > around >> 300-500.. So in my case batch size is 600. I googled for the issue >> (me.prettyprint.hector.api.exceptions.HectorException: All host pools >> marked down. Retry burden pushed out to client) and the solution was reduce >> the batch size. Hence I ran the test for each node separately. Hence In new >> results zip file each node has its own report file. I categorized them in >> two 3 main folders so that you can easily locate the results. i.,e folders >> "_create", "_update" , "_delete" .. >> >> >> >> >> >> On Wed, Sep 11, 2013 at 1:58 PM, Ian Boston <i...@tfd.co.uk> wrote: >> >>> Hi, >>> Could you re-run the tests for create in the reverse order, largest >>> collection first. >>> >>> The reason for this is in the detail of the tests adding resources to >>> collection E (10M child nodes) is flat until half way through and then >>> starts to rise which makes me thing there might be something else >>> going on on the machine of the create mechanism. I would like to rule >>> out that Cassandra is caching writes and the speed of update bears no >>> correlation to collection size, but is a function of sustained load. I >>> would also like to rule out any possibility that another OS level >>> process started to hog resources in the latter part of your tests. >>> >>> The results do look comparatively flat all the way up 100M items in a >>> collection. >>> >>> Once we have done this we can move on to providing access control, on >>> items. >>> >>> Best Regards >>> Ian >>> >>> >>> >>> On 11 September 2013 03:37, Dishara Wijewardana <ddwijeward...@gmail.com> >>> wrote: >>> > Hi Ian, >>> > Sorry the delay, I found my self some battery problems with my laptop. I >>> > believe this is what you asked for. Here I added new 100 collections i.e >>> > 1a,3a,5a etc in each old collections and did CUD. Please advice what I >>> > should do next. >>> > >>> > CREATE >>> > [RESULT] Average Latency Under Node A(1K) = 11 (ms) >>> > [RESULT] Average Latency Under Node B(10K) = 9 (ms) >>> > [RESULT] Average Latency Under Node C(100K) = 13 (ms) >>> > [RESULT] Average Latency Under Node D(1M) = 23 (ms) >>> > [RESULT] Average Latency Under Node E(10M) = 110 (ms) >>> > [RESULT] Average Latency Under Node F(100M) = 306 (ms) >>> > [FIRST RUN] #TOTAL CALLS = 600 Total Average Latency = 79 (ms) >>> > >>> > UPDATE >>> > [RESULT] Average Latency Under Node A(1K) = 9 (ms) >>> > [RESULT] Average Latency Under Node B(10K) = 8 (ms) >>> > [RESULT] Average Latency Under Node C(100K) = 9 (ms) >>> > [RESULT] Average Latency Under Node D(1M) = 17 (ms) >>> > [RESULT] Average Latency Under Node E(10M) = 20 (ms) >>> > [RESULT] Average Latency Under Node F(100M) = 26 (ms) >>> > [FIRST RUN] #TOTAL CALLS = 600 Total Average Latency = 15 (ms) >>> > >>> > DELETE >>> > [RESULT] Average Latency Under Node A(1K) = 12 (ms) >>> > [RESULT] Average Latency Under Node B(10K) = 12 (ms) >>> > [RESULT] Average Latency Under Node C(100K) = 11 (ms) >>> > [RESULT] Average Latency Under Node C(100K) = 14 (ms) >>> > [RESULT] Average Latency Under Node C(100K) = 19 (ms) >>> > [RESULT] Average Latency Under Node C(100K) = 16 (ms) >>> > [FIRST RUN] #TOTAL CALLS = 600 Total Average Latency = 14 (ms) >>> > >>> > >>> > >>> > On Tue, Sep 10, 2013 at 5:25 PM, Ian Boston <i...@tfd.co.uk> wrote: >>> > >>> >> Are they pre-populated with different numbers of items, or are they >>> >> empty when the test starts ? >>> >> >>> >> If they are empty and created by the test, can you run the test on the >>> >> collections used in the read tests so we get an idea how long it takes >>> >> to add 100 items to each of those collections. >>> >> >>> >> Thanks >>> >> Ian >>> >> >>> >> On 10 September 2013 12:25, Dishara Wijewardana < >>> ddwijeward...@gmail.com> >>> >> wrote: >>> >> > Hi Ian, >>> >> > Those are newly added collections. >>> >> > >>> >> > >>> >> > On Tue, Sep 10, 2013 at 12:00 PM, Ian Boston <i...@tfd.co.uk> wrote: >>> >> > >>> >> >> Hi Dishara, >>> >> >> Great, thanks. >>> >> >> What do the LA, MA, and SA stand for ? >>> >> >> Best regards >>> >> >> Ian >>> >> >> >>> >> >> On Tuesday, September 10, 2013, Dishara Wijewardana wrote: >>> >> >> >>> >> >> > Hi Ian >>> >> >> > Also commited the 3 new test classes for CUD to google code repo. >>> >> >> > >>> >> >> > >>> >> >> > On Tue, Sep 10, 2013 at 10:30 AM, Dishara Wijewardana < >>> >> >> > ddwijeward...@gmail.com <javascript:;>> wrote: >>> >> >> > >>> >> >> > > Hi Ian, >>> >> >> > > Finally was able to fix ;-). Had to debug along with the path >>> and >>> >> fix >>> >> >> > some >>> >> >> > > other issues also in Cassandra Provider impl. I have updated the >>> >> JIRA >>> >> >> > with >>> >> >> > > the new reports of CUD. >>> >> >> > > (I ran them one after the other respectively) >>> >> >> > > >>> >> >> > > In brief as follows. >>> >> >> > > CREATE >>> >> >> > > Average Latency Under Node LA = 165 (ms) >>> >> >> > > Average Latency Under Node MA = 203 (ms) >>> >> >> > > Average Latency Under Node SA = 256 (ms) >>> >> >> > > #TOTAL CALLS = 300 Total Average Latency = 208 (ms) >>> >> >> > > >>> >> >> > > UPDATE >>> >> >> > > Average Latency Under Node LA = 36 (ms) >>> >> >> > > Average Latency Under Node MA = 29 (ms) >>> >> >> > > Average Latency Under Node SA = 25 (ms) >>> >> >> > > #TOTAL CALLS = 300 Total Average Latency = 30 (ms) >>> >> >> > > >>> >> >> > > DELETE >>> >> >> > > Average Latency Under Node LA = 20 (ms) >>> >> >> > > Average Latency Under Node MA = 19 (ms) >>> >> >> > > Average Latency Under Node SA = 19 (ms) >>> >> >> > > #TOTAL CALLS = 300 Total Average Latency = 19 (ms) >>> >> >> > > >>> >> >> > > >>> >> >> > > >>> >> >> > > >>> >> >> > > >>> >> >> > > >>> >> >> > > >>> >> >> > > On Mon, Sep 9, 2013 at 5:47 PM, Ian Boston <i...@tfd.co.uk> >>> wrote: >>> >> >> > > >>> >> >> > >> Hi >>> >> >> > >> The CassandraImpl needs to return something suitable to >>> represent >>> >> >> > >> >>> >> >> > >> /content/cassandra/p >>> >> >> > >> >>> >> >> > >> For the moment, if the column family p does exist, return a new >>> >> type >>> >> >> > >> of read only resource, eg CassandraColumnFamilyResource. >>> >> >> > >> >>> >> >> > >> Ian >>> >> >> > >> >>> >> >> > >> >>> >> >> > >> On 9 September 2013 13:02, Dishara Wijewardana < >>> >> >> ddwijeward...@gmail.com >>> >> >> > > >>> >> >> > >> wrote: >>> >> >> > >> > On Mon, Sep 9, 2013 at 2:09 PM, Ian Boston <i...@tfd.co.uk> >>> >> wrote: >>> >> >> > >> > >>> >> >> > >> >> Hi, >>> >> >> > >> >> The call to processCreate a few lines earlier should have >>> >> created >>> >> >> the >>> >> >> > >> >> resource so that the subsequent call to getResource >>> returned the >>> >> >> new >>> >> >> > >> >> resource into newResource. >>> >> >> > >> >> >>> >> >> > >> >> If you look through the calls the create operation is >>> called in >>> >> >> > >> >> >>> >> >> > >> >> >>> >> >> > >> >> >>> >> >> > >> >>> >> >> > >>> >> >> >>> >> >>> org.apache.sling.servlets.post.impl.operations.AbstractCreateOperation.deepGetOrCreateNode(ResourceResolver, >>> >> >> > >> >> String, Map<String, RequestProperty>, List<Modification>, >>> >> >> > >> >> VersioningConfiguration) >>> >> >> > >> >> >>> >> >> > >> >> Line 533 >>> >> >> > >> >> >>> >> >> > >> >> resource = resolver.create(resource, >>> name, >>> >> >> > props); >>> >> >> > >> >> >>> >> >> > >> >> Stick a breakpoint at that line and see, if its called, and >>> if >>> >> the >>> >> >> > >> >> resource resolver calls your code. >>> >> >> > >> >> >>> >> >> > >> >> >>> >> >> > >> >> >>> >> >> > >> > Hi Ian, >>> >> >> > >> > Yes it calls the create method. As I see the issue is the >>> commit >>> >> not >>> >> >> > >> > getting called and hence resolver cannot get the created >>> >> resource. >>> >> >> > >> > So I thought of commit on the fly @create method to verify >>> that. >>> >> But >>> >> >> > we >>> >> >> > >> > have a problem here. >>> >> >> > >> > >>> >> >> > >> > When I try to create /content/cassandra/p/c node, as you >>> exactly >>> >> >> > >> mentioned >>> >> >> > >> > before, it tries to create /content/cassandra/p node. In >>> >> Cassandra >>> >> >> > >> Impl, >>> >> >> > >> > there cannot be a node like /content/cassandra/p and hence >>> >> returns >>> >> >> > >> NULL as >>> >> >> > >> > earlier and still create fails. Because "p" is the column >>> family >>> >> >> (the >>> >> >> > >> model >>> >> >> > >> > which we agreed and implemented). So what is the best >>> approach to >>> >> >> > >> overcome >>> >> >> > >> > this. >>> >> >> > >> > >>> >> >> > >> > >>> >> >> > >> > >>> >> >> > >> >> BTW: Its Ok that there is a newResource.adaptTo(Node.class) >>> as >>> >> the >>> >> >> > >> >> result is checked for Null a moment later. You do not need >>> to >>> >> >> support >>> >> >> > >> >> adaptTo(Node.class), which is a special case for Jcr. >>> >> >> > >> >> >>> >> >> > >> >> Best Regards >>> >> >> > >> >> Ian >>> >> >> > >> >> >>> >> >> > >> >> On 9 September 2013 06:05, Dishara Wijewardana < >>> >> >> > >> ddwijeward...@gmail.com> >>> >> >> > >> >> wrote: >>> >> >> > >> >> > Hi Ian, >>> >> >> > >> >> > I debug the servlet from line to line and locate the NPE >>> that >>> >> >> > causes >>> >> >> > >> this >>> >> >> > >-- >>> >> >> > Thanks >>> >> >> > /Dishara >>> >> >> > >>> >> >> >>> >> > >>> >> > >>> >> > >>> >> > -- >>> >> > Thanks >>> >> > /Dishara >>> >> >>> > >>> > >>> > >>> > -- >>> > Thanks >>> > /Dishara >>> >> >> >> >> -- >> Thanks >> /Dishara >> > > > > -- > Thanks > /Dishara
