I've fixed this now in svn.apache.org/viewvc?view=revision&revision=1679367
Pierre, your loadtest now runs to completion - thanks for reporting this issue! I can see that the results for the parallel tests are a little bit different than before, but I'm not sure how to read them so I'll leave the interpretation of that to you :) Cheers, David On 14 May 2015 at 14:38, David Bosschaert <david.bosscha...@gmail.com> wrote: > I think I know what this is. I had some additional changes exactly in > this area that I simply forgot to apply this morning. I should have it > fixed sometime today. > > Cheers, > > David > > On 14 May 2015 at 14:03, David Bosschaert <david.bosscha...@gmail.com> wrote: >> Hi Pierre, >> >> I'll take a look today. >> >> Cheers, >> >> David >> >> On 14 May 2015 at 14:00, Pierre De Rop <pierre.de...@gmail.com> wrote: >>> I just committed the benchmark tool in >>> http://svn.apache.org/viewvc/felix/sandbox/pderop/loadtest/, if you can >>> take a look. >>> >>> To run the scenario: >>> >>> - install jdk8: >>> >>> [nxuser@nx0012 pderop]$ java -version >>> java version "1.8.0_40" >>> Java(TM) SE Runtime Environment (build 1.8.0_40-b26) >>> Java HotSpot(TM) 64-Bit Server VM (build 25.40-b25, mixed mode) >>> >>> - checkout the loadtest from >>> http://svn.apache.org/viewvc/felix/sandbox/pderop/loadtest/ >>> >>> - go the the "loadtest" directory and start the test, just like this: >>> >>> $ java -server -jar bin/felix.jar >>> Welcome to Apache Felix Gogo >>> >>> g! Starting benchmarks (each tested bundle will add/remove 630 components >>> during bundle activation). >>> >>> [Starting benchmarks with no processing done in components start >>> methods] >>> >>> Benchmarking bundle: >>> org.apache.felix.dependencymanager.benchmark.dependencymanager >>> .................................................. >>> -> results in nanos: [139,129,744 | 143,957,687 | 152,157,581 | 319,631,722 >>> | 919,838,078] >>> >>> Benchmarking bundle: >>> org.apache.felix.dependencymanager.benchmark.dependencymanager.parallel . >>> >>> >>> Here, the first >>> "org.apache.felix.dependencymanager.benchmark.dependencymanager" test >>> (single-threaded) passes OK. But the next one hangs >>> (org.apache.felix.dependencymanager.benchmark.dependencymanager.parallel). >>> it uses a fork join pool with size=4. >>> >>> and when typing "log warn", we see: >>> >>> "log warn" >>> >>> 2015.05.14 13:56:10 ERROR - Bundle: >>> org.apache.felix.dependencymanager.benchmark.dependencymanager.parallel - >>> [ForkJoinPool-1-worker-3] Error processing tasks - >>> java.util.ConcurrentModificationException >>> at java.util.HashMap$HashIterator.nextNode(HashMap.java:1429) >>> at java.util.HashMap$KeyIterator.next(HashMap.java:1453) >>> at java.util.AbstractCollection.addAll(AbstractCollection.java:343) >>> at >>> org.apache.felix.framework.capabilityset.CapabilitySet.match(CapabilitySet.java:245) >>> at >>> org.apache.felix.framework.capabilityset.CapabilitySet.match(CapabilitySet.java:212) >>> at >>> org.apache.felix.framework.capabilityset.CapabilitySet.match(CapabilitySet.java:189) >>> at >>> org.apache.felix.framework.ServiceRegistry.getServiceReferences(ServiceRegistry.java:269) >>> at >>> org.apache.felix.framework.Felix.getServiceReferences(Felix.java:3577) >>> at >>> org.apache.felix.framework.Felix.getAllowedServiceReferences(Felix.java:3655) >>> at >>> org.apache.felix.framework.BundleContextImpl.getServiceReferences(BundleContextImpl.java:434) >>> at >>> org.apache.felix.dm.tracker.ServiceTracker.getInitialReferences(ServiceTracker.java:422) >>> at >>> org.apache.felix.dm.tracker.ServiceTracker.open(ServiceTracker.java:375) >>> at >>> org.apache.felix.dm.tracker.ServiceTracker.open(ServiceTracker.java:319) >>> at >>> org.apache.felix.dm.tracker.ServiceTracker.open(ServiceTracker.java:295) >>> at >>> org.apache.felix.dm.impl.ServiceDependencyImpl.start(ServiceDependencyImpl.java:226) >>> at >>> org.apache.felix.dm.impl.ComponentImpl.startDependencies(ComponentImpl.java:657) >>> at >>> org.apache.felix.dm.impl.ComponentImpl.performTransition(ComponentImpl.java:535) >>> at >>> org.apache.felix.dm.impl.ComponentImpl.handleChange(ComponentImpl.java:492) >>> at >>> org.apache.felix.dm.impl.ComponentImpl.access$5(ComponentImpl.java:482) >>> at >>> org.apache.felix.dm.impl.ComponentImpl$3.run(ComponentImpl.java:227) >>> at >>> org.apache.felix.dm.impl.DispatchExecutor.runTask(DispatchExecutor.java:182) >>> at >>> org.apache.felix.dm.impl.DispatchExecutor.run(DispatchExecutor.java:165) >>> at >>> java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1402) >>> at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) >>> at >>> java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) >>> at >>> java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1689) >>> at >>> java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157) >>> >>> >>> (I will investigate also in my code to check if the problem does not come >>> from me ?) >>> >>> cheers; >>> /Pierre >>> >>> >>> On Thu, May 14, 2015 at 1:47 PM, Pierre De Rop <pierre.de...@gmail.com> >>> wrote: >>> >>>> Hi David, >>>> >>>> I don't know if it's me (a bug in my benchmark tool) or if if there is a >>>> regression somewhere in the framework, by my parallel test does not pass >>>> anymore. >>>> >>>> The test first starts with a single-threaded scenario, which passes OK >>>> (org.apache.felix.dependencymanager.benchmark.dependencymanager), then when >>>> the parallel test starts >>>> (org.apache.felix.dependencymanager.benchmark.dependencymanager.parallel) >>>> it suddenly hangs, and when I type "log warn" under the gogo shell, I see >>>> the following exception: >>>> >>>> (I'm using java8): >>>> >>>> $ java -server -Xmx4g -Xms4g -jar bin/felix.jar >>>> ____________________________ >>>> Welcome to Apache Felix Gogo >>>> >>>> Benchmarking bundle: >>>> org.apache.felix.dependencymanager.benchmark.dependencymanager.parallel . >>>> >>>> (here, the dependencymanager.parallel test hangs and when I type "log >>>> warn", I see this:) >>>> >>>> g! log warn >>>> 2015.05.14 13:31:03 ERROR - Bundle: >>>> org.apache.felix.dependencymanager.benchmark.dependencymanager.parallel - >>>> [ForkJoinPool-1-worker-3] Error processing tasks - >>>> java.util.ConcurrentModificationException >>>> at java.util.HashMap$HashIterator.nextNode(HashMap.java:1429) >>>> at java.util.HashMap$KeyIterator.next(HashMap.java:1453) >>>> at java.util.AbstractCollection.addAll(AbstractCollection.java:343) >>>> at >>>> org.apache.felix.framework.capabilityset.CapabilitySet.match(CapabilitySet.java:245) >>>> at >>>> org.apache.felix.framework.capabilityset.CapabilitySet.match(CapabilitySet.java:212) >>>> at >>>> org.apache.felix.framework.capabilityset.CapabilitySet.match(CapabilitySet.java:189) >>>> at >>>> org.apache.felix.framework.ServiceRegistry.getServiceReferences(ServiceRegistry.java:269) >>>> at >>>> org.apache.felix.framework.Felix.getServiceReferences(Felix.java:3577) >>>> at >>>> org.apache.felix.framework.Felix.getAllowedServiceReferences(Felix.java:3655) >>>> at >>>> org.apache.felix.framework.BundleContextImpl.getServiceReferences(BundleContextImpl.java:434) >>>> at >>>> org.apache.felix.dm.tracker.ServiceTracker.getInitialReferences(ServiceTracker.java:422) >>>> at >>>> org.apache.felix.dm.tracker.ServiceTracker.open(ServiceTracker.java:375) >>>> at >>>> org.apache.felix.dm.tracker.ServiceTracker.open(ServiceTracker.java:319) >>>> at >>>> org.apache.felix.dm.tracker.ServiceTracker.open(ServiceTracker.java:295) >>>> at >>>> org.apache.felix.dm.impl.ServiceDependencyImpl.start(ServiceDependencyImpl.java:226) >>>> at >>>> org.apache.felix.dm.impl.ComponentImpl.startDependencies(ComponentImpl.java:657) >>>> at >>>> org.apache.felix.dm.impl.ComponentImpl.performTransition(ComponentImpl.java:535) >>>> at >>>> org.apache.felix.dm.impl.ComponentImpl.handleChange(ComponentImpl.java:492) >>>> at >>>> org.apache.felix.dm.impl.ComponentImpl.access$5(ComponentImpl.java:482) >>>> at >>>> org.apache.felix.dm.impl.ComponentImpl$3.run(ComponentImpl.java:227) >>>> at >>>> org.apache.felix.dm.impl.DispatchExecutor.runTask(DispatchExecutor.java:182) >>>> at >>>> org.apache.felix.dm.impl.DispatchExecutor.run(DispatchExecutor.java:165) >>>> at >>>> java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1402) >>>> at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) >>>> at >>>> java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) >>>> at >>>> java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1689) >>>> at >>>> java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157) >>>> >>>> (If I configure my threadpool to 1, I have no problems, but with >>>> threadpool=4, then I have the problem) >>>> >>>> I will investigate, but Ideally, may be it would be helpful if you could >>>> also run the test by yourself; so I will commit soon something to reproduce >>>> the problem in my sandbox. >>>> >>>> cheers; >>>> /Pierre >>>> >>>> On Thu, May 14, 2015 at 11:11 AM, David Bosschaert < >>>> david.bosscha...@gmail.com> wrote: >>>> >>>>> I've committed this now in >>>>> http://svn.apache.org/viewvc?view=revision&revision=1679327 >>>>> >>>>> Curious to see what others are measuring. My tests were focused on >>>>> multiple bundles/threads obtaining the same service, as that's were I >>>>> saw a bit of contention. >>>>> >>>>> Cheers, >>>>> >>>>> David >>>>> >>>>> On 13 May 2015 at 15:10, Pierre De Rop <pierre.de...@gmail.com> wrote: >>>>> > Hi David, >>>>> > >>>>> > I'm looking forward to test your improvements using the >>>>> dependencymanager >>>>> > benchmark tool ([1]). >>>>> > >>>>> > >>>>> > [1] >>>>> > >>>>> http://svn.apache.org/viewvc/felix/trunk/dependencymanager/org.apache.felix.dependencymanager.benchmark/ >>>>> > >>>>> > /Pierre >>>>> > >>>>> > On Wed, May 13, 2015 at 3:02 PM, David Bosschaert < >>>>> > david.bosscha...@gmail.com> wrote: >>>>> > >>>>> >> I have implemented the performance improvements that I was thinking of >>>>> >> using Java 5 concurrency tools, they can be viewed at [1]. >>>>> >> >>>>> >> I wrote a little performance test suite [2] that tests multithreaded >>>>> >> service registry performance (10 threads) from single / multiple >>>>> >> bundles with either singleton services and Prototype Service Factory >>>>> >> services and the results are quite impressive. I'm getting performance >>>>> >> improvements compared to the current trunk from 8 times better than >>>>> >> the original (800%) to more than 30 times better (3000%). >>>>> >> >>>>> >> Carsten has already reviewed the code (thanks Carsten!) and I'm >>>>> >> planning to commit it to Felix tomorrow if nobody objects. >>>>> >> >>>>> >> Cheers, >>>>> >> >>>>> >> David >>>>> >> >>>>> >> [1] >>>>> >> >>>>> https://github.com/bosschaert/felix/commit/e6a1b06c6e66d9c98e6d81b91ef7003c8e725450 >>>>> >> [2] >>>>> >> >>>>> https://github.com/bosschaert/coderthoughts/tree/master/service-registry-perftest/srperf >>>>> >> >>>>> >> On 23 March 2015 at 15:39, Richard S. Hall <he...@ungoverned.org> >>>>> wrote: >>>>> >> > On 3/23/15 10:17 , David Bosschaert wrote: >>>>> >> >> >>>>> >> >> On 23 March 2015 at 13:39, Richard S. Hall <he...@ungoverned.org> >>>>> >> wrote: >>>>> >> >>> >>>>> >> >>> On 3/23/15 03:55 , Guillaume Nodet wrote: >>>>> >> >>>> >>>>> >> >>>> There's a call to interrupt() in Felix#acquireBundleLock(), not >>>>> sure >>>>> >> if >>>>> >> >>>> it >>>>> >> >>>> can be the culprit though. >>>>> >> >>>> Interrupts could also be caused by a bundle being shutdown while >>>>> one >>>>> >> of >>>>> >> >>>> its >>>>> >> >>>> thread is waiting for a service, which should is a valid use case >>>>> >> imho. >>>>> >> >>>> Anyway, I think sanely reacting to a thread being interrupted >>>>> would be >>>>> >> >>>> good. >>>>> >> >>> >>>>> >> >>> >>>>> >> >>> Yes, threads can be interrupted if they are holding a bundle lock >>>>> and >>>>> >> the >>>>> >> >>> global lock holder needs the bundle lock. >>>>> >> >>> >>>>> >> >>> I admit that I do not recall why we ignore the interrupt here, but >>>>> >> didn't >>>>> >> >>> we >>>>> >> >>> implement service lookup so that a bundle lock wasn't necessary? I >>>>> >> >>> thought >>>>> >> >>> we just checked for the validity of the bundle context before >>>>> returning >>>>> >> >>> or >>>>> >> >>> something. Perhaps we felt there was no reason to be interrupted in >>>>> >> that >>>>> >> >>> case. I really don't know. >>>>> >> >> >>>>> >> >> I think that the Service Registry could be rewritten to be >>>>> completely >>>>> >> >> free of synchronized blocks using the Java 5 concurrency libraries, >>>>> >> > >>>>> >> > >>>>> >> > Well, that just moves the sync blocks to the library, but yeah sure. >>>>> >> > >>>>> >> >> which I think would really be a better approach. There is too much >>>>> >> >> locking going on in the current SR implementation IMHO. >>>>> >> > >>>>> >> > >>>>> >> > I don't really think there is too much, but it is complicated. >>>>> >> > Unfortunately, it is complicated to make sure that locks aren't held >>>>> >> while >>>>> >> > do service lookups and this is complicated because you can run into >>>>> >> cycles, >>>>> >> > etc. >>>>> >> > >>>>> >> > But feel free to try to simplify it. >>>>> >> > >>>>> >> >> >>>>> >> >> This brings the question: can we move to Java 5 (or Java 6) for the >>>>> >> >> Framework codebase? AFAIK we're currently still JDK 1.4 compatible >>>>> but >>>>> >> >> I would be surprised if there is anyone who still needs a JDK that >>>>> >> >> went end-of-life 7 years ago. >>>>> >> > >>>>> >> > >>>>> >> > At this point, it doesn't really matter to me. >>>>> >> > >>>>> >> > -> richard >>>>> >> > >>>>> >> >> >>>>> >> >> Best regards, >>>>> >> >> >>>>> >> >> David >>>>> >> > >>>>> >> > >>>>> >> >>>>> >>>> >>>>
