[infinispan-dev] Time measurement and expiry
Hi all, I'm looking at some performance details, and noticed the current code is using System.currentTimeMillis() to estimate performance for short lived operations. For the purpose of CacheMgmtInterceptor we should use System.nanoTime() instead: http://blogs.oracle.com/dholmes/entry/inside_the_hotspot_vm_clocks Searching for "currentTimeMillis()" in the code base reveals it's being used for expiry of entries as well. If someone messes with system clocks data will expire; is it expected that entries expire at calendar time and not at a set lifetime? I understand we need to use the absolute time to be able to transmit the value to other nodes, where the nanoTime of different nodes might not be comparable, but we could store one and use both: transmit the absolute value only to other nodes and refer to the other for accurate expiries. Other nodes receiving the absolute value will check for the remaining lifespane and store the corresponding nanoTime. ExpiryHelper itself will invoke the currentTimeMillis() operation internally, that means that it's going to be invoked at least once for each entry being accessed and might result in a lot of invocations when traversing several entries; I'm wondering if it shouldn't take a millisecond parameter to consider as current, so that this relatively expensive method can be invoked only once at the beginning of a batch of operations. Also reading this time information is a high overhead in some configurations, I'm wondering if we should make it possible to configure Infinispan to not track performance on each cache operation? Someone might prefer to estimate an average from multiple calls; I'm going to remove CacheMgmtInterceptor for my tests. Sanne ___ infinispan-dev mailing list infinispan-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev
[infinispan-dev] Time measurement and expiry
Rather than a heap, I recall Netty had a hash wheel timer[1]. Which should be better performing than a heap. You could get clever by having multiple heaps, segmented by orders of magnitude. For example wheel 1 could have < 1 minute, 2 < 1 hour etc. Netty uses a lot of short lived timers and performs well. [1] http://docs.jboss.org/netty/3.1/api/org/jboss/netty/util/HashedWheelTimer.html Obviously this uses some memory, but you could probably optimize it somewhat to only work with Node instances. ___ infinispan-dev mailing list infinispan-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev
Re: [infinispan-dev] Time measurement and expiry
On Sat, Oct 15, 2011 at 7:25 PM, Sanne Grinovero wrote: > Hi all, > I'm looking at some performance details, and noticed the current code > is using System.currentTimeMillis() to estimate performance for short > lived operations. > > For the purpose of CacheMgmtInterceptor we should use System.nanoTime() > instead: > http://blogs.oracle.com/dholmes/entry/inside_the_hotspot_vm_clocks > Very interesting, I knew that in Windows currentTimeMillis() basically just reads a volatile because I got bit by the 15 millisecond accuracy issue before, so I thought it would always be very fast. I had no idea on Linux it would have the same performance as nanoTime(). > Searching for "currentTimeMillis()" in the code base reveals it's > being used for expiry of entries as well. If someone messes with > system clocks data will expire; is it expected that entries expire at > calendar time and not at a set lifetime? I would say if someone's messing with system clocks they already have a much bigger problem than their cached data expiring... Besides, someone may intentionally change the system time to test the ISPN expiration algorithm :) > I understand we need to use the absolute time to be able to transmit > the value to other nodes, where the nanoTime of different nodes might > not be comparable, but we could store one and use both: transmit the > absolute value only to other nodes and refer to the other for accurate > expiries. > Other nodes receiving the absolute value will check for the remaining > lifespane and store the corresponding nanoTime. > > ExpiryHelper itself will invoke the currentTimeMillis() operation > internally, that means that it's going to be invoked at least once for > each entry being accessed and might result in a lot of invocations > when traversing several entries; I'm wondering if it shouldn't take a > millisecond parameter to consider as current, so that this relatively > expensive method can be invoked only once at the beginning of a batch > of operations. > + 1 for adding a currentTime parameter to the isExpiredX() methods in ExpiryHelper and also to isExpired() CacheEntry and its implementations. > Also reading this time information is a high overhead in some > configurations, I'm wondering if we should make it possible to > configure Infinispan to not track performance on each cache operation? > Someone might prefer to estimate an average from multiple calls; I'm > going to remove CacheMgmtInterceptor for my tests. > CacheMgmtInterceptor doesn't seem to be enabled when JMX statistics are disabled, so I don't think you need to explicitly remove CacheMgmtInterceptor for your performance tests. Cheers Dan ___ infinispan-dev mailing list infinispan-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev
Re: [infinispan-dev] Time measurement and expiry
On 17 October 2011 09:02, Dan Berindei wrote: > On Sat, Oct 15, 2011 at 7:25 PM, Sanne Grinovero wrote: >> Hi all, >> I'm looking at some performance details, and noticed the current code >> is using System.currentTimeMillis() to estimate performance for short >> lived operations. >> >> For the purpose of CacheMgmtInterceptor we should use System.nanoTime() >> instead: >> http://blogs.oracle.com/dholmes/entry/inside_the_hotspot_vm_clocks >> > > Very interesting, I knew that in Windows currentTimeMillis() basically > just reads a volatile because I got bit by the 15 millisecond accuracy > issue before, so I thought it would always be very fast. I had no idea > on Linux it would have the same performance as nanoTime(). Even in Windows it seems to be a bit more expensive than just reading a volatile; >> Searching for "currentTimeMillis()" in the code base reveals it's >> being used for expiry of entries as well. If someone messes with >> system clocks data will expire; is it expected that entries expire at >> calendar time and not at a set lifetime? > > I would say if someone's messing with system clocks they already have > a much bigger problem than their cached data expiring... Absolutely, but this is just an example. I'd expect that if I'm using it as a cache, the data will stay in it for a certain amount of relative time. It doesn't have to be a user changing the clock, for example there might be automated changes like daylight adjustments (summer/winter time) or relatively small updates via NTP to keep the clocks in sync.. of course they might not affect the functionality significantly, but I'm just pointing out that we might prefer to use the nanoTime source as our API suggests a period of entry validity in milliseconds, not an expiry in terms of calendar Date. > Besides, someone may intentionally change the system time to test the > ISPN expiration algorithm :) > >> I understand we need to use the absolute time to be able to transmit >> the value to other nodes, where the nanoTime of different nodes might >> not be comparable, but we could store one and use both: transmit the >> absolute value only to other nodes and refer to the other for accurate >> expiries. >> Other nodes receiving the absolute value will check for the remaining >> lifespane and store the corresponding nanoTime. >> >> ExpiryHelper itself will invoke the currentTimeMillis() operation >> internally, that means that it's going to be invoked at least once for >> each entry being accessed and might result in a lot of invocations >> when traversing several entries; I'm wondering if it shouldn't take a >> millisecond parameter to consider as current, so that this relatively >> expensive method can be invoked only once at the beginning of a batch >> of operations. >> > > + 1 for adding a currentTime parameter to the isExpiredX() methods in > ExpiryHelper and also to isExpired() CacheEntry and its > implementations. Agreed: ISPN-1459 >> Also reading this time information is a high overhead in some >> configurations, I'm wondering if we should make it possible to >> configure Infinispan to not track performance on each cache operation? >> Someone might prefer to estimate an average from multiple calls; I'm >> going to remove CacheMgmtInterceptor for my tests. >> > > CacheMgmtInterceptor doesn't seem to be enabled when JMX statistics > are disabled, so I don't think you need to explicitly remove > CacheMgmtInterceptor for your performance tests. Yes that's how I do it ;) What concerns me is that there is going to be a relatively strong overhead just by enabling statistics. > > Cheers > Dan ___ infinispan-dev mailing list infinispan-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev
Re: [infinispan-dev] Time measurement and expiry
On Mon, Oct 17, 2011 at 4:13 PM, Sanne Grinovero wrote: > On 17 October 2011 09:02, Dan Berindei wrote: >> On Sat, Oct 15, 2011 at 7:25 PM, Sanne Grinovero >> wrote: >>> Hi all, >>> I'm looking at some performance details, and noticed the current code >>> is using System.currentTimeMillis() to estimate performance for short >>> lived operations. >>> >>> For the purpose of CacheMgmtInterceptor we should use System.nanoTime() >>> instead: >>> http://blogs.oracle.com/dholmes/entry/inside_the_hotspot_vm_clocks >>> >> >> Very interesting, I knew that in Windows currentTimeMillis() basically >> just reads a volatile because I got bit by the 15 millisecond accuracy >> issue before, so I thought it would always be very fast. I had no idea >> on Linux it would have the same performance as nanoTime(). > > Even in Windows it seems to be a bit more expensive than just reading > a volatile; > The article you linked to said it's about 6 cycles, I don't think it can do much more than reading a variable and returning it in that time. >>> Searching for "currentTimeMillis()" in the code base reveals it's >>> being used for expiry of entries as well. If someone messes with >>> system clocks data will expire; is it expected that entries expire at >>> calendar time and not at a set lifetime? >> >> I would say if someone's messing with system clocks they already have >> a much bigger problem than their cached data expiring... > > Absolutely, but this is just an example. I'd expect that if I'm using > it as a cache, the data will stay in it for a certain amount of > relative time. > It doesn't have to be a user changing the clock, for example there > might be automated changes like daylight adjustments (summer/winter > time) or relatively small updates via NTP to keep the clocks in sync.. > of course they might not affect the functionality significantly, but > I'm just pointing out that we might prefer to use the nanoTime source > as our API suggests a period of entry validity in milliseconds, not an > expiry in terms of calendar Date. > Daylight changes don't modify the UTC time, so System.currentTimeMillis() should not be affected. It's true that we only offer a validity period in milliseconds, but I'm sure if we had a Date-based alternative many people would use that instead. In the end I don't think the extra complexity is worth it, at least until somebody actually asks for it. >> Besides, someone may intentionally change the system time to test the >> ISPN expiration algorithm :) >> >>> I understand we need to use the absolute time to be able to transmit >>> the value to other nodes, where the nanoTime of different nodes might >>> not be comparable, but we could store one and use both: transmit the >>> absolute value only to other nodes and refer to the other for accurate >>> expiries. >>> Other nodes receiving the absolute value will check for the remaining >>> lifespane and store the corresponding nanoTime. >>> >>> ExpiryHelper itself will invoke the currentTimeMillis() operation >>> internally, that means that it's going to be invoked at least once for >>> each entry being accessed and might result in a lot of invocations >>> when traversing several entries; I'm wondering if it shouldn't take a >>> millisecond parameter to consider as current, so that this relatively >>> expensive method can be invoked only once at the beginning of a batch >>> of operations. >>> >> >> + 1 for adding a currentTime parameter to the isExpiredX() methods in >> ExpiryHelper and also to isExpired() CacheEntry and its >> implementations. > > Agreed: ISPN-1459 > >>> Also reading this time information is a high overhead in some >>> configurations, I'm wondering if we should make it possible to >>> configure Infinispan to not track performance on each cache operation? >>> Someone might prefer to estimate an average from multiple calls; I'm >>> going to remove CacheMgmtInterceptor for my tests. >>> >> >> CacheMgmtInterceptor doesn't seem to be enabled when JMX statistics >> are disabled, so I don't think you need to explicitly remove >> CacheMgmtInterceptor for your performance tests. > > Yes that's how I do it ;) > What concerns me is that there is going to be a relatively strong > overhead just by enabling statistics. > Makes sense, many people will probably have their own tooling to measure their app's performance. We could remove this from the JMX statistics (at least by default) but make sure we have the hooks in place for the users to integrate with a different monitoring system. ___ infinispan-dev mailing list infinispan-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev
Re: [infinispan-dev] Time measurement and expiry
On 17 Oct 2011, at 14:13, Sanne Grinovero wrote: >> Very interesting, I knew that in Windows currentTimeMillis() basically >> just reads a volatile because I got bit by the 15 millisecond accuracy >> issue before, so I thought it would always be very fast. I had no idea >> on Linux it would have the same performance as nanoTime(). Indeed very nice! I miss the part where the article says nanoTime has the same performance on modern Linux as currentTimeMillis, are you sure? > > Even in Windows it seems to be a bit more expensive than just reading > a volatile; > >>> Searching for "currentTimeMillis()" in the code base reveals it's >>> being used for expiry of entries as well. If someone messes with >>> system clocks data will expire; is it expected that entries expire at >>> calendar time and not at a set lifetime? >> >> I would say if someone's messing with system clocks they already have >> a much bigger problem than their cached data expiring... > > Absolutely, but this is just an example. I'd expect that if I'm using > it as a cache, the data will stay in it for a certain amount of > relative time. > It doesn't have to be a user changing the clock, for example there > might be automated changes like daylight adjustments (summer/winter > time) or relatively small updates via NTP to keep the clocks in sync.. I'd expect System.currentTimeMillis() to not be affected in any way by the daylight savings, as it returns: "the difference, measured in milliseconds, between the current time and midnight, January 1, 1970 UTC." > of course they might not affect the functionality significantly, +- 10/15 millis > but > I'm just pointing out that we might prefer to use the nanoTime source > as our API suggests a period of entry validity in milliseconds, not an > expiry in terms of calendar Date. the calendar bit (i.e. timesavings) isn't counted here. > >> Besides, someone may intentionally change the system time to test the >> ISPN expiration algorithm :) >> >>> I understand we need to use the absolute time to be able to transmit >>> the value to other nodes, where the nanoTime of different nodes might >>> not be comparable, but we could store one and use both: transmit the >>> absolute value only to other nodes and refer to the other for accurate >>> expiries. >>> Other nodes receiving the absolute value will check for the remaining >>> lifespane and store the corresponding nanoTime. >>> >>> ExpiryHelper itself will invoke the currentTimeMillis() operation >>> internally, that means that it's going to be invoked at least once for >>> each entry being accessed and might result in a lot of invocations >>> when traversing several entries; I'm wondering if it shouldn't take a >>> millisecond parameter to consider as current, so that this relatively >>> expensive method can be invoked only once at the beginning of a batch >>> of operations. >>> >> >> + 1 for adding a currentTime parameter to the isExpiredX() methods in >> ExpiryHelper and also to isExpired() CacheEntry and its >> implementations. > > Agreed: ISPN-1459 > >>> Also reading this time information is a high overhead in some >>> configurations, I'm wondering if we should make it possible to >>> configure Infinispan to not track performance on each cache operation? >>> Someone might prefer to estimate an average from multiple calls; I'm >>> going to remove CacheMgmtInterceptor for my tests. I think it is relevant for local caches and not that relevant for clustered caches. >>> >> >> CacheMgmtInterceptor doesn't seem to be enabled when JMX statistics >> are disabled, so I don't think you need to explicitly remove >> CacheMgmtInterceptor for your performance tests. > > Yes that's how I do it ;) > What concerns me is that there is going to be a relatively strong > overhead just by enabling statistics. > >> >> Cheers >> Dan > ___ > infinispan-dev mailing list > infinispan-dev@lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev ___ infinispan-dev mailing list infinispan-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev
Re: [infinispan-dev] Time measurement and expiry
On Tue, Oct 18, 2011 at 1:32 AM, Mircea Markus wrote: > > On 17 Oct 2011, at 14:13, Sanne Grinovero wrote: > >>> Very interesting, I knew that in Windows currentTimeMillis() basically >>> just reads a volatile because I got bit by the 15 millisecond accuracy >>> issue before, so I thought it would always be very fast. I had no idea >>> on Linux it would have the same performance as nanoTime(). > Indeed very nice! > I miss the part where the article says nanoTime has the same performance on > modern Linux as currentTimeMillis, are you sure? Yeah, the article didn't talk about Linux but I found this article: http://blogs.oracle.com/ksrini/entry/we_take_java_performance_very There's also this JDK bug complaining about both being slow: http://bugs.sun.com/view_bug.do?bug_id=6876279 (the test output is in a weird format, ignore the 1st number, the 2nd one is nanos/call). Except when I actually ran the code in the bug report I my timings were much better than those reported in the bug description: java version "1.6.0_22" OpenJDK Runtime Environment (IcedTea6 1.10.3) (fedora-59.1.10.3.fc15-x86_64) OpenJDK 64-Bit Server VM (build 20.0-b11, mixed mode) Intel(R) Core(TM) i5 CPU M 540 @ 2.53GHz currentTimeMillis: 36ns nanoTime: 28ns -server or -XX:AggressiveOpts don't seem to make a difference. I also ran the test on cluster10 and the results are slightly worse, but not significantly: java version "1.6.0_17" OpenJDK Runtime Environment (IcedTea6 1.7.10) (rhel-1.39.b17.el6_0-x86_64) OpenJDK 64-Bit Server VM (build 14.0-b16, mixed mode) Intel(R) Xeon(R) CPU E5640 @ 2.67GHz currentTimeMillis: 40ns nanoTime: 35ns It would be interesting if we could run the test on all our machines and see how the timings vary by machine and OS. It seems we're not the only ones with this problem either, Oracle (the database) apparently calls gettimeofday() a lot so RHEL has some optimizations to remove the system call overhead and make it even faster (more like Windows I presume, but I don't have a Windows machine on hand to confirm): http://docs.redhat.com/docs/en-US/Red_Hat_Enterprise_MRG/1.3/html/Realtime_Tuning_Guide/sect-Realtime_Tuning_Guide-General_System_Tuning-gettimeofday_speedup.html http://docs.redhat.com/docs/en-US/Red_Hat_Enterprise_MRG/1.3/html/Realtime_Tuning_Guide/sect-Realtime_Tuning_Guide-Realtime_Specific_Tuning-RT_Specific_gettimeofday_speedup.html ___ infinispan-dev mailing list infinispan-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev
Re: [infinispan-dev] Time measurement and expiry
The replies so far are very interesting, but at this stage I'd be more interested in discussing the fundamental question: Why are we preferring to provide more cache misses && slower performance, to fake a slightly better precision in eviction? Since we can't guarantee precision, I'd want to actually remove these checks altogether: is there any use case requiring high precision in the eviction process? I hope not, I wouldn't suggest to rely on Infinispan as a reliable clock. In the couple of cases we should really keep this logic - likely the CacheLoader - we can discuss how to optimize this. For example I like Dan's idea of introducing a Clock component; we could explore such solutions for the remaining bits which will still need a time source but I'd first want to remove the main bottleneck. --Sanne On 18 October 2011 07:57, Dan Berindei wrote: > On Tue, Oct 18, 2011 at 1:32 AM, Mircea Markus > wrote: >> >> On 17 Oct 2011, at 14:13, Sanne Grinovero wrote: >> Very interesting, I knew that in Windows currentTimeMillis() basically just reads a volatile because I got bit by the 15 millisecond accuracy issue before, so I thought it would always be very fast. I had no idea on Linux it would have the same performance as nanoTime(). >> Indeed very nice! >> I miss the part where the article says nanoTime has the same performance on >> modern Linux as currentTimeMillis, are you sure? > > Yeah, the article didn't talk about Linux but I found this article: > http://blogs.oracle.com/ksrini/entry/we_take_java_performance_very > There's also this JDK bug complaining about both being slow: > http://bugs.sun.com/view_bug.do?bug_id=6876279 (the test output is in > a weird format, ignore the 1st number, the 2nd one is nanos/call). > Except when I actually ran the code in the bug report I my timings > were much better than those reported in the bug description: > > java version "1.6.0_22" > OpenJDK Runtime Environment (IcedTea6 1.10.3) (fedora-59.1.10.3.fc15-x86_64) > OpenJDK 64-Bit Server VM (build 20.0-b11, mixed mode) > Intel(R) Core(TM) i5 CPU M 540 @ 2.53GHz > > currentTimeMillis: 36ns > nanoTime: 28ns > > -server or -XX:AggressiveOpts don't seem to make a difference. > > > I also ran the test on cluster10 and the results are slightly worse, > but not significantly: > > java version "1.6.0_17" > OpenJDK Runtime Environment (IcedTea6 1.7.10) (rhel-1.39.b17.el6_0-x86_64) > OpenJDK 64-Bit Server VM (build 14.0-b16, mixed mode) > Intel(R) Xeon(R) CPU E5640 @ 2.67GHz > > currentTimeMillis: 40ns > nanoTime: 35ns > > > It would be interesting if we could run the test on all our machines > and see how the timings vary by machine and OS. > > > It seems we're not the only ones with this problem either, Oracle (the > database) apparently calls gettimeofday() a lot so RHEL has some > optimizations to remove the system call overhead and make it even > faster (more like Windows I presume, but I don't have a Windows > machine on hand to confirm): > http://docs.redhat.com/docs/en-US/Red_Hat_Enterprise_MRG/1.3/html/Realtime_Tuning_Guide/sect-Realtime_Tuning_Guide-General_System_Tuning-gettimeofday_speedup.html > http://docs.redhat.com/docs/en-US/Red_Hat_Enterprise_MRG/1.3/html/Realtime_Tuning_Guide/sect-Realtime_Tuning_Guide-Realtime_Specific_Tuning-RT_Specific_gettimeofday_speedup.html > ___ > infinispan-dev mailing list > infinispan-dev@lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev > ___ infinispan-dev mailing list infinispan-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev
Re: [infinispan-dev] Time measurement and expiry
On Tue, Oct 18, 2011 at 10:57 AM, Sanne Grinovero wrote: > The replies so far are very interesting, but at this stage I'd be more > interested in discussing the fundamental question: > > Why are we preferring to provide more cache misses && slower > performance, to fake a slightly better precision in eviction? > I guess this is the main question, how much worse would the eviction precision be if we only relied on the periodic eviction thread? I would agree with you if we could guarantee a constant precision in eviction, but with the current algorithm the eviction interval has to grow with the cache size so we can't offer any guarantees. And that's even without considering CacheLoaders. Elias' suggestion of using a heap for eviction is interesting, but adding entries to the heap will slow puts. We could try a combined approach, every minute scan the data container and add the entries that will expire in the next minute to a heap. Only puts that expire before the deadline will add the entry to the eviction heap. Then another eviction thread can evict entries from the heap at a higher frequency. I'm sure there are better ideas, but I we'd have to prototype them and show that we can keep the staleness under an upper bound (say 1 second) for reasonably large caches before changing the algorithm. > Since we can't guarantee precision, I'd want to actually remove these > checks altogether: is there any use case requiring high precision in > the eviction process? > I hope not, I wouldn't suggest to rely on Infinispan as a reliable clock. > Relying on (relatively) precise expiration is the kind of implicit assumption that people make and don't even realize they depend on it until an upgrade breaks their application. I'm curious if other cache impls offer any guarantees in this area. > In the couple of cases we should really keep this logic - likely the > CacheLoader - we can discuss how to optimize this. For example I like > Dan's idea of introducing a Clock component; we could explore such > solutions for the remaining bits which will still need a time source > but I'd first want to remove the main bottleneck. > > --Sanne > > > On 18 October 2011 07:57, Dan Berindei wrote: >> On Tue, Oct 18, 2011 at 1:32 AM, Mircea Markus >> wrote: >>> >>> On 17 Oct 2011, at 14:13, Sanne Grinovero wrote: >>> > Very interesting, I knew that in Windows currentTimeMillis() basically > just reads a volatile because I got bit by the 15 millisecond accuracy > issue before, so I thought it would always be very fast. I had no idea > on Linux it would have the same performance as nanoTime(). >>> Indeed very nice! >>> I miss the part where the article says nanoTime has the same performance on >>> modern Linux as currentTimeMillis, are you sure? >> >> Yeah, the article didn't talk about Linux but I found this article: >> http://blogs.oracle.com/ksrini/entry/we_take_java_performance_very >> There's also this JDK bug complaining about both being slow: >> http://bugs.sun.com/view_bug.do?bug_id=6876279 (the test output is in >> a weird format, ignore the 1st number, the 2nd one is nanos/call). >> Except when I actually ran the code in the bug report I my timings >> were much better than those reported in the bug description: >> >> java version "1.6.0_22" >> OpenJDK Runtime Environment (IcedTea6 1.10.3) (fedora-59.1.10.3.fc15-x86_64) >> OpenJDK 64-Bit Server VM (build 20.0-b11, mixed mode) >> Intel(R) Core(TM) i5 CPU M 540 @ 2.53GHz >> >> currentTimeMillis: 36ns >> nanoTime: 28ns >> >> -server or -XX:AggressiveOpts don't seem to make a difference. >> >> >> I also ran the test on cluster10 and the results are slightly worse, >> but not significantly: >> >> java version "1.6.0_17" >> OpenJDK Runtime Environment (IcedTea6 1.7.10) (rhel-1.39.b17.el6_0-x86_64) >> OpenJDK 64-Bit Server VM (build 14.0-b16, mixed mode) >> Intel(R) Xeon(R) CPU E5640 @ 2.67GHz >> >> currentTimeMillis: 40ns >> nanoTime: 35ns >> >> >> It would be interesting if we could run the test on all our machines >> and see how the timings vary by machine and OS. >> >> >> It seems we're not the only ones with this problem either, Oracle (the >> database) apparently calls gettimeofday() a lot so RHEL has some >> optimizations to remove the system call overhead and make it even >> faster (more like Windows I presume, but I don't have a Windows >> machine on hand to confirm): >> http://docs.redhat.com/docs/en-US/Red_Hat_Enterprise_MRG/1.3/html/Realtime_Tuning_Guide/sect-Realtime_Tuning_Guide-General_System_Tuning-gettimeofday_speedup.html >> http://docs.redhat.com/docs/en-US/Red_Hat_Enterprise_MRG/1.3/html/Realtime_Tuning_Guide/sect-Realtime_Tuning_Guide-Realtime_Specific_Tuning-RT_Specific_gettimeofday_speedup.html >> ___ >> infinispan-dev mailing list >> infinispan-dev@lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/infinispan-dev >> > > ___
Re: [infinispan-dev] Time measurement and expiry
On Oct 19, 2011, at 2:13 PM, Dan Berindei wrote: > On Tue, Oct 18, 2011 at 10:57 AM, Sanne Grinovero > wrote: >> The replies so far are very interesting, but at this stage I'd be more >> interested in discussing the fundamental question: >> >> Why are we preferring to provide more cache misses && slower >> performance, to fake a slightly better precision in eviction? >> > > I guess this is the main question, how much worse would the eviction > precision be if we only relied on the periodic eviction thread? We did use to have a periodic eviction thread in the JBoss Cache days, but that used to cause us problems since we queued events to the cache in order to apply eviction algorithms. I don't see the precision as being paramount in the eviction area, so these avenue could be revisited if a non-blocking, low-cost solution could be found. > > I would agree with you if we could guarantee a constant precision in > eviction, but with the current algorithm the eviction interval has to > grow with the cache size so we can't offer any guarantees. And that's > even without considering CacheLoaders. > > Elias' suggestion of using a heap for eviction is interesting, but > adding entries to the heap will slow puts. We could try a combined > approach, every minute scan the data container and add the entries > that will expire in the next minute to a heap. Only puts that expire > before the deadline will add the entry to the eviction heap. Then > another eviction thread can evict entries from the heap at a higher > frequency. > > I'm sure there are better ideas, but I we'd have to prototype them and > show that we can keep the staleness under an upper bound (say 1 > second) for reasonably large caches before changing the algorithm. > >> Since we can't guarantee precision, I'd want to actually remove these >> checks altogether: is there any use case requiring high precision in >> the eviction process? >> I hope not, I wouldn't suggest to rely on Infinispan as a reliable clock. >> > > Relying on (relatively) precise expiration is the kind of implicit > assumption that people make and don't even realize they depend on it > until an upgrade breaks their application. > > I'm curious if other cache impls offer any guarantees in this area. > >> In the couple of cases we should really keep this logic - likely the >> CacheLoader - we can discuss how to optimize this. For example I like >> Dan's idea of introducing a Clock component; we could explore such >> solutions for the remaining bits which will still need a time source >> but I'd first want to remove the main bottleneck. >> >> --Sanne >> >> >> On 18 October 2011 07:57, Dan Berindei wrote: >>> On Tue, Oct 18, 2011 at 1:32 AM, Mircea Markus >>> wrote: On 17 Oct 2011, at 14:13, Sanne Grinovero wrote: >> Very interesting, I knew that in Windows currentTimeMillis() basically >> just reads a volatile because I got bit by the 15 millisecond accuracy >> issue before, so I thought it would always be very fast. I had no idea >> on Linux it would have the same performance as nanoTime(). Indeed very nice! I miss the part where the article says nanoTime has the same performance on modern Linux as currentTimeMillis, are you sure? >>> >>> Yeah, the article didn't talk about Linux but I found this article: >>> http://blogs.oracle.com/ksrini/entry/we_take_java_performance_very >>> There's also this JDK bug complaining about both being slow: >>> http://bugs.sun.com/view_bug.do?bug_id=6876279 (the test output is in >>> a weird format, ignore the 1st number, the 2nd one is nanos/call). >>> Except when I actually ran the code in the bug report I my timings >>> were much better than those reported in the bug description: >>> >>> java version "1.6.0_22" >>> OpenJDK Runtime Environment (IcedTea6 1.10.3) (fedora-59.1.10.3.fc15-x86_64) >>> OpenJDK 64-Bit Server VM (build 20.0-b11, mixed mode) >>> Intel(R) Core(TM) i5 CPU M 540 @ 2.53GHz >>> >>> currentTimeMillis: 36ns >>> nanoTime: 28ns >>> >>> -server or -XX:AggressiveOpts don't seem to make a difference. >>> >>> >>> I also ran the test on cluster10 and the results are slightly worse, >>> but not significantly: >>> >>> java version "1.6.0_17" >>> OpenJDK Runtime Environment (IcedTea6 1.7.10) (rhel-1.39.b17.el6_0-x86_64) >>> OpenJDK 64-Bit Server VM (build 14.0-b16, mixed mode) >>> Intel(R) Xeon(R) CPU E5640 @ 2.67GHz >>> >>> currentTimeMillis: 40ns >>> nanoTime: 35ns >>> >>> >>> It would be interesting if we could run the test on all our machines >>> and see how the timings vary by machine and OS. >>> >>> >>> It seems we're not the only ones with this problem either, Oracle (the >>> database) apparently calls gettimeofday() a lot so RHEL has some >>> optimizations to remove the system call overhead and make it even >>> faster (more like Windows I presume, but I don't have a Windows >>> machine on hand to confirm): >>> http://docs.redhat.com/d
Re: [infinispan-dev] Time measurement and expiry
Thanks for the clarification Dan! On 18 Oct 2011, at 07:57, Dan Berindei wrote: > On Tue, Oct 18, 2011 at 1:32 AM, Mircea Markus > wrote: >> >> On 17 Oct 2011, at 14:13, Sanne Grinovero wrote: >> Very interesting, I knew that in Windows currentTimeMillis() basically just reads a volatile because I got bit by the 15 millisecond accuracy issue before, so I thought it would always be very fast. I had no idea on Linux it would have the same performance as nanoTime(). >> Indeed very nice! >> I miss the part where the article says nanoTime has the same performance on >> modern Linux as currentTimeMillis, are you sure? > > Yeah, the article didn't talk about Linux but I found this article: > http://blogs.oracle.com/ksrini/entry/we_take_java_performance_very > There's also this JDK bug complaining about both being slow: > http://bugs.sun.com/view_bug.do?bug_id=6876279 (the test output is in > a weird format, ignore the 1st number, the 2nd one is nanos/call). > Except when I actually ran the code in the bug report I my timings > were much better than those reported in the bug description: > > java version "1.6.0_22" > OpenJDK Runtime Environment (IcedTea6 1.10.3) (fedora-59.1.10.3.fc15-x86_64) > OpenJDK 64-Bit Server VM (build 20.0-b11, mixed mode) > Intel(R) Core(TM) i5 CPU M 540 @ 2.53GHz > > currentTimeMillis: 36ns > nanoTime: 28ns > > -server or -XX:AggressiveOpts don't seem to make a difference. > > > I also ran the test on cluster10 and the results are slightly worse, > but not significantly: > > java version "1.6.0_17" > OpenJDK Runtime Environment (IcedTea6 1.7.10) (rhel-1.39.b17.el6_0-x86_64) > OpenJDK 64-Bit Server VM (build 14.0-b16, mixed mode) > Intel(R) Xeon(R) CPU E5640 @ 2.67GHz > > currentTimeMillis: 40ns > nanoTime: 35ns > > > It would be interesting if we could run the test on all our machines > and see how the timings vary by machine and OS. > > > It seems we're not the only ones with this problem either, Oracle (the > database) apparently calls gettimeofday() a lot so RHEL has some > optimizations to remove the system call overhead and make it even > faster (more like Windows I presume, but I don't have a Windows > machine on hand to confirm): > http://docs.redhat.com/docs/en-US/Red_Hat_Enterprise_MRG/1.3/html/Realtime_Tuning_Guide/sect-Realtime_Tuning_Guide-General_System_Tuning-gettimeofday_speedup.html > http://docs.redhat.com/docs/en-US/Red_Hat_Enterprise_MRG/1.3/html/Realtime_Tuning_Guide/sect-Realtime_Tuning_Guide-Realtime_Specific_Tuning-RT_Specific_gettimeofday_speedup.html > ___ > infinispan-dev mailing list > infinispan-dev@lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev ___ infinispan-dev mailing list infinispan-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev
Re: [infinispan-dev] Time measurement and expiry
On 20 Oct 2011, at 08:47, Galder Zamarreño wrote: >>> >>> Why are we preferring to provide more cache misses && slower >>> performance, to fake a slightly better precision in eviction? >>> >> >> I guess this is the main question, how much worse would the eviction >> precision be if we only relied on the periodic eviction thread? > > We did use to have a periodic eviction thread in the JBoss Cache days, but > that used to cause us problems since we queued events to the cache in order > to apply eviction algorithms. That was a different case. In JBC we picked nodes for eviction and put them on an eviction queue and they were analysed periodically. That queue became a bottleneck. The suggestion here is to essentially cache time lookups so that you don't call System.cTM() all the time. See Elias Ross' parallel post to this list about Netty's HashedWheelTimer. Sanne, I recall your creating a JIRA for this? -- Manik Surtani ma...@jboss.org twitter.com/maniksurtani Lead, Infinispan http://www.infinispan.org ___ infinispan-dev mailing list infinispan-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev
Re: [infinispan-dev] Time measurement and expiry
On 27 October 2011 16:58, Manik Surtani wrote: > > On 20 Oct 2011, at 08:47, Galder Zamarreño wrote: > > Why are we preferring to provide more cache misses && slower > > performance, to fake a slightly better precision in eviction? > > > I guess this is the main question, how much worse would the eviction > > precision be if we only relied on the periodic eviction thread? > > We did use to have a periodic eviction thread in the JBoss Cache days, but > that used to cause us problems since we queued events to the cache in order > to apply eviction algorithms. > > That was a different case. In JBC we picked nodes for eviction and put them > on an eviction queue and they were analysed periodically. That queue became > a bottleneck. > The suggestion here is to essentially cache time lookups so that you don't > call System.cTM() all the time. See Elias Ross' parallel post to this list > about Netty's HashedWheelTimer. > Sanne, I recall your creating a JIRA for this? Yes I created ISPN-1459 as a starting point, only to not look at the clock for each entry during an eviction iteration. I have an experimental branch implementing a POC and a test with fills the cache with 10.000.000 small entries, the eviction thread is twice as fast than before, and using less CPU too. I hoped better, but still worth to polish it. So that makes me think that it would benefit all non-immortal entries also during read operations to remove this same check, but I can't move the time read out of the entry code without introducing a new overhead to the read of immortal entries - which are dearer to me :) So to get the best benefit for both cases I'll have to do some more experiments, after JUDcon and current OGM priorities. Sanne ___ infinispan-dev mailing list infinispan-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev
Re: [infinispan-dev] Time measurement and expiry
Cool link Elias, I read the paper linked in the javadoc and it's very interesting. Netty seems to use scheme 6 (a single array of lists with hashing and without any overflow). I believe in general our timeouts will be much longer so I suspect scheme 5 or scheme 7 should perform better for us. Scheme 5 should be much better than 6 if most of the timeouts use the same value, and I believe we fit right in that use case. On the other hand I think we could save some memory if we implement scheme 7 by not actually storing the higher level wheels and building them on the fly by iterating the data container. The downside is that we'll need periodic sweeps of the data container, but those can be on a separate thread. One scenario that we should keep in mind is the L1 cache, whose entries I would guess are invalidated much more often than they expire after their 1 hour lifetime. If the user stores only immortal entries + L1 entries in the cache we might end up with a ticker task that never rarely does anything useful so we should minimize its impact. It might also be worth it to implement our own list type to make adding and removing timeouts faster. I see Netty uses IdentityHashMap-backed sets for the bucket lists, but as Sanne found out they aren't the fastest thing around. Using a doubly-linked list and keeping a reference to the Node in the cache entry should make it much faster. Cheers Dan On Tue, Oct 25, 2011 at 8:02 PM, Elias Ross wrote: > Rather than a heap, I recall Netty had a hash wheel timer[1]. Which should > be better performing than a heap. You could get clever by having > multiple heaps, segmented by orders of magnitude. For example wheel 1 > could have < 1 minute, 2 < 1 hour etc. Netty uses a lot of short lived > timers and performs well. > > [1] > http://docs.jboss.org/netty/3.1/api/org/jboss/netty/util/HashedWheelTimer.html > > Obviously this uses some memory, but you could probably optimize it > somewhat to only work with Node instances. > ___ > infinispan-dev mailing list > infinispan-dev@lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev > ___ infinispan-dev mailing list infinispan-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev
Re: [infinispan-dev] Time measurement and expiry
On 30 Oct 2011, at 23:58, Dan Berindei wrote: > Cool link Elias, I read the paper linked in the javadoc and it's very > interesting. > > Netty seems to use scheme 6 (a single array of lists with hashing and > without any overflow). I believe in general our timeouts will be much > longer so I suspect scheme 5 or scheme 7 should perform better for us. > > Scheme 5 should be much better than 6 if most of the timeouts use the > same value, and I believe we fit right in that use case. > On the other hand I think we could save some memory if we implement > scheme 7 by not actually storing the higher level wheels and building > them on the fly by iterating the data container. The downside is that > we'll need periodic sweeps of the data container, but those can be on > a separate thread. -1. I'd prefer an approach that does not involve any additional management threads. > One scenario that we should keep in mind is the L1 cache, whose > entries I would guess are invalidated much more often than they expire > after their 1 hour lifetime. If the user stores only immortal entries > + L1 entries in the cache we might end up with a ticker task that > never rarely does anything useful so we should minimize its impact. > > It might also be worth it to implement our own list type to make > adding and removing timeouts faster. I see Netty uses > IdentityHashMap-backed sets for the bucket lists, but as Sanne found > out they aren't the fastest thing around. Using a doubly-linked list > and keeping a reference to the Node in the cache entry should make it > much faster. Yes, I think implementing our own is the way forward as well. No sense in adding dependencies for just a single class, etc. Cheers Manik -- Manik Surtani ma...@jboss.org twitter.com/maniksurtani Lead, Infinispan http://www.infinispan.org ___ infinispan-dev mailing list infinispan-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev
Re: [infinispan-dev] Time measurement and expiry
I've experimented with some implementations; since the simplest approach just removes and simplifies code while providing a significant speedup I'm going to send a pull request for that one: I just have to add some additional methods to InternalCacheEntry, and get a good improvement in the eviction "scanning". Since with 10 million entries it's able to scan/check them all in a timewindow close to a single second (~ 1,2 s on my laptop), I'm then proposing as well the second change to remove the "precision check" at all, which means we don't need any clock and I just need to remove some ~40 lines of useless and slow code. It seems we're all agreeing that higher precision is pointless, correct? Especially since the eviction thread can finish a full scan quicker. Cheers, Sanne On 2 November 2011 14:45, Manik Surtani wrote: > > On 30 Oct 2011, at 23:58, Dan Berindei wrote: > >> Cool link Elias, I read the paper linked in the javadoc and it's very >> interesting. >> >> Netty seems to use scheme 6 (a single array of lists with hashing and >> without any overflow). I believe in general our timeouts will be much >> longer so I suspect scheme 5 or scheme 7 should perform better for us. >> >> Scheme 5 should be much better than 6 if most of the timeouts use the >> same value, and I believe we fit right in that use case. >> On the other hand I think we could save some memory if we implement >> scheme 7 by not actually storing the higher level wheels and building >> them on the fly by iterating the data container. The downside is that >> we'll need periodic sweeps of the data container, but those can be on >> a separate thread. > > -1. I'd prefer an approach that does not involve any additional management > threads. > >> One scenario that we should keep in mind is the L1 cache, whose >> entries I would guess are invalidated much more often than they expire >> after their 1 hour lifetime. If the user stores only immortal entries >> + L1 entries in the cache we might end up with a ticker task that >> never rarely does anything useful so we should minimize its impact. >> >> It might also be worth it to implement our own list type to make >> adding and removing timeouts faster. I see Netty uses >> IdentityHashMap-backed sets for the bucket lists, but as Sanne found >> out they aren't the fastest thing around. Using a doubly-linked list >> and keeping a reference to the Node in the cache entry should make it >> much faster. > > Yes, I think implementing our own is the way forward as well. No sense in > adding dependencies for just a single class, etc. > > Cheers > Manik > -- > Manik Surtani > ma...@jboss.org > twitter.com/maniksurtani > > Lead, Infinispan > http://www.infinispan.org > > > > > ___ > infinispan-dev mailing list > infinispan-dev@lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev > ___ infinispan-dev mailing list infinispan-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev
Re: [infinispan-dev] Time measurement and expiry
+1. Looking forward to your patch. On 2 Nov 2011, at 15:39, Sanne Grinovero wrote: > I've experimented with some implementations; since the simplest > approach just removes and simplifies code while providing a > significant speedup I'm going to send a pull request for that one: I > just have to add some additional methods to InternalCacheEntry, and > get a good improvement in the eviction "scanning". > > Since with 10 million entries it's able to scan/check them all in a > timewindow close to a single second (~ 1,2 s on my laptop), I'm then > proposing as well the second change to remove the "precision check" at > all, which means we don't need any clock and I just need to remove > some ~40 lines of useless and slow code. It seems we're all agreeing > that higher precision is pointless, correct? Especially since the > eviction thread can finish a full scan quicker. > > Cheers, > Sanne > > On 2 November 2011 14:45, Manik Surtani wrote: >> >> On 30 Oct 2011, at 23:58, Dan Berindei wrote: >> >>> Cool link Elias, I read the paper linked in the javadoc and it's very >>> interesting. >>> >>> Netty seems to use scheme 6 (a single array of lists with hashing and >>> without any overflow). I believe in general our timeouts will be much >>> longer so I suspect scheme 5 or scheme 7 should perform better for us. >>> >>> Scheme 5 should be much better than 6 if most of the timeouts use the >>> same value, and I believe we fit right in that use case. >>> On the other hand I think we could save some memory if we implement >>> scheme 7 by not actually storing the higher level wheels and building >>> them on the fly by iterating the data container. The downside is that >>> we'll need periodic sweeps of the data container, but those can be on >>> a separate thread. >> >> -1. I'd prefer an approach that does not involve any additional management >> threads. >> >>> One scenario that we should keep in mind is the L1 cache, whose >>> entries I would guess are invalidated much more often than they expire >>> after their 1 hour lifetime. If the user stores only immortal entries >>> + L1 entries in the cache we might end up with a ticker task that >>> never rarely does anything useful so we should minimize its impact. >>> >>> It might also be worth it to implement our own list type to make >>> adding and removing timeouts faster. I see Netty uses >>> IdentityHashMap-backed sets for the bucket lists, but as Sanne found >>> out they aren't the fastest thing around. Using a doubly-linked list >>> and keeping a reference to the Node in the cache entry should make it >>> much faster. >> >> Yes, I think implementing our own is the way forward as well. No sense in >> adding dependencies for just a single class, etc. >> >> Cheers >> Manik >> -- >> Manik Surtani >> ma...@jboss.org >> twitter.com/maniksurtani >> >> Lead, Infinispan >> http://www.infinispan.org >> >> >> >> >> ___ >> infinispan-dev mailing list >> infinispan-dev@lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/infinispan-dev >> > > ___ > infinispan-dev mailing list > infinispan-dev@lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev -- Manik Surtani ma...@jboss.org twitter.com/maniksurtani Lead, Infinispan http://www.infinispan.org ___ infinispan-dev mailing list infinispan-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev
Re: [infinispan-dev] Time measurement and expiry
On 11-11-02 11:39 AM, Sanne Grinovero wrote: > I've experimented with some implementations; since the simplest > approach just removes and simplifies code while providing a > significant speedup I'm going to send a pull request for that one: I > just have to add some additional methods to InternalCacheEntry, and > get a good improvement in the eviction "scanning". > > Since with 10 million entries it's able to scan/check them all in a > timewindow close to a single second (~ 1,2 s on my laptop), I'm then > proposing as well the second change to remove the "precision check" at > all, which means we don't need any clock and I just need to remove > some ~40 lines of useless and slow code. It seems we're all agreeing > that higher precision is pointless, correct? Especially since the > eviction thread can finish a full scan quicker. > > Sanne have you integrated this? I would love to see what you have done! ___ infinispan-dev mailing list infinispan-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev
Re: [infinispan-dev] Time measurement and expiry
On 3 November 2011 15:47, Vladimir Blagojevic wrote:
> Sanne have you integrated this? I would love to see what you have done!
Yes it's in master as ISPN-1459
Again it's a super-simple patch; it's all about
the following change:
for each entry {
long now = System.currentTimeMillis();
if (entry.isExpired(now)){
evict(entry);
}
}
into
long now = System.currentTimeMillis();
for each entry {
if (entry.isExpired(now)){
evict(entry);
}
}
(which involves some API chages, but that's the point).
Cheers,
Sanne
___
infinispan-dev mailing list
infinispan-dev@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
