Re: [infinispan-dev] Faster LRU
On Thu, Jul 7, 2011 at 1:21 PM, Manik Surtani wrote: > I think we leave the old LRU as LRU_OLD and mark it as deprecated. I for one am against keeping the old policy around, as the new LRU policy implements exactly the same algorithm only in O(1) instead of O(n). It would have made sense to keep it if there was a difference in the algorithm, but Vladimir even kept the batching from the old LRU policy. We should test it as thoroughly as we can, and to that end I've been working on some additions to MapStressTest that try to measure how good the eviction algorithm is. I think it already helped me find a problem with the new LRU. I've updated pull #414 (https://github.com/infinispan/infinispan/pull/414) to work on top of Vladimir's pull request, in case you want to have a look. You might want to adjust the number of keys and/or disable some of the options in the data providers before running it though, it takes a lot of time to run (and it also needs -Xmx2000m). I've left it running overnight on the test cluster (cluster01 and cluster10), I'll send an update with the results in the morning. Cheers Dan ___ infinispan-dev mailing list infinispan-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev
Re: [infinispan-dev] Atomic operations and transactions
On 5 Jul 2011, at 11:45, Dan Berindei wrote: > On Tue, Jul 5, 2011 at 12:23 PM, Galder Zamarreño wrote: >> >> >> On Jul 4, 2011, at 11:25 AM, Sanne Grinovero wrote: >> >>> I agree they don't make sense, but only in the sense of exposed API >>> during a transaction: some time ago I admit I was expecting them to >>> just work: the API is there, nice public methods in the public >>> interface with javadocs explaining that that was exactly what I was >>> looking for, no warnings, no failures. Even worse, all works fine when >>> running a local test because how the locks currently work they are >>> acquired locally first, so unless you're running such a test in DIST >>> mode, and happen to be *not* the owner of the being tested key, people >>> won't even notice that this is not supported. >>> >>> Still being able to use them is very important, also in combination >>> with transactions: I might be running blocks of transactional code >>> (like a CRUD operation via OGM) and still require to advance a >>> sequence for primary key generation. This needs to be an atomic >>> operation, and I should really not forget to suspend the transaction. >> >> Fair point. At first glance, the best way to deal with this is suspending >> the tx cos that guarantees the API contract while not forcing locks to be >> acquired for too long. >> >> I'd advice though that whoever works on this though needs to go over >> existing use cases and see if the end result could differ somehow if this >> change gets applied. If any divergences are found and are to be expected, >> these need to be thoroughly documented. >> >> I've gone through some cases and end results would not differ at first >> glance if the atomic ops suspend the txs. The only thing that would change >> would be the expectations of lock acquisition timeouts by atomic ops within >> txs. >> >> For example: >> >> Cache contains: k1=galder >> >> 1. Tx1 does a cache.replace(k1, "galder", "sanne") -> suspends tx and >> applies change -> k1=sanne now >> 2. Tx2 does a cache.replace(k1, "galder", "manik") -> suspends tx and is not >> able to apply change >> 3. Tx2 commits >> 4. Tx1 commits >> End result: k1=sanne >> >> 1. Tx1 does a cache.replace(k1, "galder", "sanne") -> acquires lock >> 2. Tx2 does a cache.replace(k1, "galder", "manik") -> waits for lock >> 3. Tx2 rollback -> times out acquiring lock >> 4. Tx1 commits -> applies change >> End result: k1=sanne >> > > Galder, the big difference would be that with optimistic transactions > you don't acquire the lock on the spot, so the tx will be rolled back > if someone else modifies the key between your get and the prepare. > We could make a compromise, instead of forcing the atomic operations > to happen outside the transaction we could force them to always use > pessimistic locking. Actually Mircea and I discussed this Friday > evening too, but I forgot about it until now. > > After all Sanne has two use cases for atomic operations: sequences and > reference counts. Sequences can and should happen outside > transactions, but as we discussed on the train we could make each > node/thread acquire a range of say 100 seq numbers at a time and > remove the need for any locking to get a new sequence number. > Reference counts on the other hand should remain inside the > transaction, because you would have to to the refcount rollback by > hand Not sure that's even possible: what if you rollback the tx and node crashes before rolling back refcount.. > (plus I don't think you should let other transactions see the > modified refcount before they see the new data). > > Dan > >>> >>> Sanne >>> >>> 2011/7/4 Galder Zamarreño : Do these atomic operations really make sense within an (optimitic) transaction? For example, putIfAbsent(): it stores a k,v pair if the key is not present. But the key about it's usability is that the return of putIfAbsent can tell you whether the put succeeded or not. Once you go into transactions, the result is only valid once the transaction has been prepared unless the pessimistic locking as per definition in http://community.jboss.org/docs/DOC-16973 is in use, and that's already pretty confusing IMO. I get the feeling that those atomic operations are particularly useful when transactions are not used cos they allow you to reduce to cache operations to one, hence avoiding the need to use a lock or synchronized block, or in our case, a transaction. On Jun 30, 2011, at 3:11 PM, Sanne Grinovero wrote: > Hello all, > some team members had a meeting yesterday, one of the discussed > subjects was about using atomic operations (putIfAbsent, etc..). > Mircea just summarised it in the following proposal: > > The atomic operations, as defined by the ConcurrentHashMap, don't fit > well within the scope of optimistic transaction: this is because there > is
Re: [infinispan-dev] Contributing to Infinispan Guide updated
Sure. Let me ping you back in few minutes. Need to go to a customer call. ;-) - Original Message - | Do you want to write that Prabhat? If you can log in to confluence, I | can give you write access to the docs. | | On 7 Jul 2011, at 17:15, Prabhat Jha wrote: | | > This is awesome. I have often seen occasional questions in other | > forums about configuring IDE so that license headers automatically | > get added in a new file. Adding that will be helpful too. | > | > - Original Message - | > | Hi All | > | | > | Please take a look at | > | https://docs.jboss.org/author/display/ISPN/Contributing+to+Infinispan | > | - I've given it a thorough review and reorganisation. Please let | > | me | > | know if I've removed material that you think is still needed. | > | | > | pete | > | ___ | > | infinispan-dev mailing list | > | infinispan-dev@lists.jboss.org | > | https://lists.jboss.org/mailman/listinfo/infinispan-dev | > ___ | > infinispan-dev mailing list | > infinispan-dev@lists.jboss.org | > https://lists.jboss.org/mailman/listinfo/infinispan-dev | | | ___ | infinispan-dev mailing list | infinispan-dev@lists.jboss.org | https://lists.jboss.org/mailman/listinfo/infinispan-dev ___ infinispan-dev mailing list infinispan-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev
Re: [infinispan-dev] Contributing to Infinispan Guide updated
Do you want to write that Prabhat? If you can log in to confluence, I can give you write access to the docs. On 7 Jul 2011, at 17:15, Prabhat Jha wrote: > This is awesome. I have often seen occasional questions in other forums about > configuring IDE so that license headers automatically get added in a new > file. Adding that will be helpful too. > > - Original Message - > | Hi All > | > | Please take a look at > | https://docs.jboss.org/author/display/ISPN/Contributing+to+Infinispan > | - I've given it a thorough review and reorganisation. Please let me > | know if I've removed material that you think is still needed. > | > | pete > | ___ > | infinispan-dev mailing list > | infinispan-dev@lists.jboss.org > | https://lists.jboss.org/mailman/listinfo/infinispan-dev > ___ > infinispan-dev mailing list > infinispan-dev@lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev ___ infinispan-dev mailing list infinispan-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev
Re: [infinispan-dev] Contributing to Infinispan Guide updated
This is awesome. I have often seen occasional questions in other forums about configuring IDE so that license headers automatically get added in a new file. Adding that will be helpful too. - Original Message - | Hi All | | Please take a look at | https://docs.jboss.org/author/display/ISPN/Contributing+to+Infinispan | - I've given it a thorough review and reorganisation. Please let me | know if I've removed material that you think is still needed. | | pete | ___ | infinispan-dev mailing list | infinispan-dev@lists.jboss.org | https://lists.jboss.org/mailman/listinfo/infinispan-dev ___ infinispan-dev mailing list infinispan-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev
[infinispan-dev] New documentation type - Glossary
I've added a new docs category - Glossary. This is for defining terminology we use that new users (and experienced users) might have trouble with. "Data Grid" would be a great example. For each item in the glossary, create a page underneath Glossary, and then we can link it in using an include. An item in the Glossary should use an h2. heading and include the heading in the text body as well as have a page title - this just makes the item reusable. ___ infinispan-dev mailing list infinispan-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev
[infinispan-dev] Contributing to Infinispan Guide updated
Hi All Please take a look at https://docs.jboss.org/author/display/ISPN/Contributing+to+Infinispan - I've given it a thorough review and reorganisation. Please let me know if I've removed material that you think is still needed. pete ___ infinispan-dev mailing list infinispan-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev
Re: [infinispan-dev] Faster LRU
On 11-07-07 6:21 AM, Manik Surtani wrote: > I think we leave the old LRU as LRU_OLD and mark it as deprecated. > > Vladimir, does this apply to LIRS as well? No, not LIRS, this was LRU optimization overlooked from the beginning :-) ___ infinispan-dev mailing list infinispan-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev
Re: [infinispan-dev] Tidy up of forum + wiki
On 7 Jul 2011, at 09:32, Galder Zamarreño wrote: >> * The wiki "home page" is now the SBS document view, as all documents in the >> Infinispan space are wiki content (design docs, notes etc.). This means we >> no longer need to manually maintain a contents page for the wiki :-) > > Some categories have been created but these are empty so far? Only "Design > Documents" has a document in it. Yes. I was just experimenting to see how it works :-) It doesn't work quite right atm as you've discovered. I'll chat to Libor and Mark to see how quickly we can get categories to be useful. > And how do you get back to the overall article view once you've clicked on a > category? It keeps coming back to the category selection you have. The only > way to get out of it is to manually edit the URL and remove the extra bits on > it that make it select that category (WTF!) Yeah, this view is no use right now. I'll work with the .org team to get this fixed up. ___ infinispan-dev mailing list infinispan-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev
Re: [infinispan-dev] Faster LRU
I think we leave the old LRU as LRU_OLD and mark it as deprecated.
Vladimir, does this apply to LIRS as well?
On 6 Jul 2011, at 21:08, Vladimir Blagojevic wrote:
> No :-) We can leave it, it all depends on how many more CR cycles we have
> before Final release to field test new LRU!
>
> On 11-07-06 2:46 PM, Sanne Grinovero wrote:
>>
>> Awesome!
>> Are you really sure about eradicating the old implementation?
>>
>> Cheers,
>> Sanne
>>
>> On 6 Jul 2011 19:39, "Vladimir Blagojevic" wrote:
>> > Hey,
>> >
>> > Ok I've issued a pull request. Please review it
>> > https://github.com/infinispan/infinispan/pull/418
>> > I am very happy with performance increase observed in new LRU. It
>> > significantly beats synchronized Map and performance is very close to
>> > ConcurrentHashMap itself.
>> > I ran tests overnight for 12 hours (each test run is 20 minutes).
>> >
>> > Dan, please have a look at new MapStressTest and add enhancements you
>> > discussed with Galder and Sanne. Once you integrate these changes lets
>> > do another round of testing of both LRU impls and if all goes well we
>> > can drop old LRU entirely.
>> >
>> > Cheers,
>> > Vladimir
>> >
>> >
>> >
>> > [ec2-user@ip-10-38-110-25 infinispan]$ ps -e | grep java
>> > [ec2-user@ip-10-38-110-25 infinispan]$ cat perf_new_lru.log
>> > [INFO] Scanning for projects...
>> > [INFO] Reactor build order:
>> > [INFO] Infinispan Common Parent
>> > [INFO] Infinispan Core
>> > [INFO] Infinispan Tools
>> > [INFO] Infinispan Query API
>> > [INFO] Infinispan Tree API
>> > [INFO] Parent pom for cachestore modules
>> > [INFO] Infinispan JDBC CacheStore
>> > [INFO] Infinispan Lucene Directory Implementation
>> > [INFO] Infinispan JDBM CacheStore
>> > [INFO] Infinispan BDBJE CacheStore
>> > [INFO] Infinispan CloudCacheStore
>> > [INFO] Parent pom for server modules
>> > [INFO] Infinispan Server Core Module
>> > [INFO] Infinispan Server Hotrod Module
>> > [INFO] Infinispan Client Hotrod Module
>> > [INFO] Infinispan remote CacheStore
>> > [INFO] Infinispan CassandraCacheStore
>> > [INFO] Infinispan Server Memcached Module
>> > [INFO] Infinispan WebSocket Server
>> > [INFO] Infinispan REST Server
>> > [INFO] Infinispan RHQ Plugin
>> > [INFO] Infinispan Spring Integration
>> > [INFO] Infinispan GUI Demo
>> > [INFO] Infinispan EC2 Demo
>> > [INFO] Infinispan Distributed Executors and Map/Reduce Demo
>> > [INFO] Infinispan EC2 Demo UI
>> > [INFO] Infinispan Directory Demo
>> > [INFO] Infinispan Lucene Directory Demo
>> > [INFO] Infinispan GridFileSystem WebDAV interface
>> > [INFO] Infinispan Distribution
>> > [INFO]
>> >
>> > [INFO] Building Infinispan Common Parent
>> > [INFO] task-segment: [test]
>> > [INFO]
>> >
>> > [INFO] snapshot org.jboss.ws:jbossws-api:1.0.0-SNAPSHOT: checking for
>> > updates from jboss-public-repository
>> > [INFO] snapshot org.jboss.ws:jbossws-api:1.0.0-SNAPSHOT: checking for
>> > updates from jboss-public-repository-group
>> > [INFO] snapshot org.jboss.ws:jbossws-parent:1.0.10-SNAPSHOT: checking
>> > for updates from jboss-public-repository
>> > [INFO] snapshot org.jboss.ws:jbossws-parent:1.0.10-SNAPSHOT: checking
>> > for updates from jboss-public-repository-group
>> > [INFO] [enforcer:enforce {execution: enforce-java}]
>> > [INFO]
>> >
>> > [INFO] Building Infinispan Core
>> > [INFO] task-segment: [test]
>> > [INFO]
>> >
>> > [INFO] [enforcer:enforce {execution: enforce-java}]
>> > [INFO] [resources:resources {execution: default-resources}]
>> > [INFO] Using 'UTF-8' encoding to copy filtered resources.
>> > [INFO] Copying 17 resources
>> > [INFO] [compiler:compile {execution: default-compile}]
>> > [INFO] Compiling 52 source files to
>> > /home/ec2-user/infinispan/core/target/classes
>> > [INFO] Preparing exec:java
>> > [WARNING] Removing: java from forked lifecycle, to prevent recursive
>> > invocation.
>> > [INFO] [enforcer:enforce {execution: enforce-java}]
>> > [INFO] [exec:java {execution: default}]
>> > Generating schema file in
>> > /home/ec2-user/infinispan/core/src/main/resources/schema
>> > Using file name infinispan-config-5.0.xsd for schema
>> > Generated schema file successfully
>> > [INFO] [resources:testResources {execution: default-testResources}]
>> > [INFO] Using 'UTF-8' encoding to copy filtered resources.
>> > [INFO] Copying 24 resources
>> > [INFO] [compiler:testCompile {execution: default-testCompile}]
>> > [INFO] Nothing to compile - all classes are up to date
>> > [INFO] [surefire:test {execution: default-test}]
>> > [INFO] Surefire report directory:
>> > /home/ec2-user/infinispan/core/target/surefire-reports
>> >
>> > ---
>> > T E S T S
>> >
Re: [infinispan-dev] Tidy up of forum + wiki
On Jul 5, 2011, at 8:06 PM, Pete Muir wrote: > All, > > Today I have spent some time tidying up the wiki and forum. > > Wiki > -- > > * All documentation articles have been moved to the "Infinispan Archive". > This space is not generally writable by the community. Contact me if an > article has been moved there by mistake > * The wiki "home page" is now the SBS document view, as all documents in the > Infinispan space are wiki content (design docs, notes etc.). This means we no > longer need to manually maintain a contents page for the wiki :-) Some categories have been created but these are empty so far? Only "Design Documents" has a document in it. And how do you get back to the overall article view once you've clicked on a category? It keeps coming back to the category selection you have. The only way to get out of it is to manually edit the URL and remove the extra bits on it that make it select that category (WTF!) > > Forums > -- > > * All open questions older than 2 months have been marked as answered > regardless of their status. This was necessary to give us a clean slate to > work from. > * I have added http://community.jboss.org/wiki/HowToAskAForumQuestion to > encourage people to ask questions with all necessary info up front and also > to mark questions as answered. This will appear as an "Announcement" when > there are no other announcements showing as well. > * I have added guidance on how the community can contribute by answering > questions > https://docs.jboss.org/author/display/ISPN/Contributing+to+Infinispan#ContributingtoInfinispan-HelpingOthersOut > - this section also addresses how we manage the forum. I will add sections > on stackoverflow etc. here as I get to them ;-) Thx for writing that up, Pete! > * I have asked the jboss.org team if we can have all discussions be > questions, this will allow much easier tracking of "open user questions" > > Core team, please can you go through the open questions we currently have > (http://community.jboss.org/en/infinispan?view=discussions#/?filter=open) and > try to answer any that are open. Some you may feel you have already answered > and not seen any activity from the user for a couple of weeks. Feel free to > mark these as "Assumed Answered". > > Pete > ___ > infinispan-dev mailing list > infinispan-dev@lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev -- Galder Zamarreño Sr. Software Engineer Infinispan, JBoss Cache ___ infinispan-dev mailing list infinispan-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev
