Dynamic classloading in Geode
Vito and I spent some time hacking up a prototype for dynamic and distributed classloading of Geode functions. Currently a user has to compile a function into a jar and deploy it using gfsh before it can be executed. If we could enable automatic deployment of functions across a running cluster it would speed up the development cycle for Geode applications and pave the way for other interesting features (like Java8 lambdas). Here’s how it works: A function wrapper (DynamicFunction) serializes the original function object and captures dependent classes as byte arrays. We generate an MD5 hash over the bytecode and use that as the key for storing the bytecode in a replicated region (“hackday”) within the cache. When the function is invoked, we call putIfAbsent() to distribute the byte code prior to executing the function across the cluster. During execution, we extend the TCCL with a new class loader that loads classes from our region while the original function is being deserialized. The original function is then executed in parallel on the cluster members. This allows an application developer to iteratively modify and test function code without any manual steps to upload class files. Obviously, there is a lot more thinking and design work to do around these ideas. Here’s our super-hacky code if you’re interested: https://gist.github.com/metatype/9b1f39a24e52f5c6f3e1 https://gist.github.com/metatype/9b1f39a24e52f5c6f3e1 Caveats: 1) Currently we only capture static class dependencies. Any class dependencies present during method invocations are ignored. This could be addressed by doing byte code inspection (using ASM, javaassist, etc). 2) The region we use to cache class byte code should be automatically recreated as a metadata region, similar to how we store pdx types. We also need to configure eviction and expiration attributes to control resource usage and remove garbage. 3) We only injected the byte code caching hack into the code path for FunctionService.onServers(pool). Also, the putIfAbsent() call adds another network roundtrip. Anthony Vito
Review Request 37450: GEODE-216: Handle any exception that might be thrown
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/37450/ --- Review request for geode and Dan Smith. Bugs: GEODE-216 https://issues.apache.org/jira/browse/GEODE-216 Repository: geode Description --- The old code was trying to prevent a deadlock under some certain conditions. But if it can not get a PR because of an exception then a deadlock is not possible. So the new code just catches and ignores exceptions and falls through to the code that will process the message the old way. This code will encounter the same exception and has the proper logic to handle it. Diffs - gemfire-core/src/main/java/com/gemstone/gemfire/internal/cache/partitioned/GetMessage.java c8d990d7e2dba1f61a2aa7f9df703c0be6306a9d Diff: https://reviews.apache.org/r/37450/diff/ Testing --- RedisDistDUnitTest was failing for me without this change. With it testConcCreateDestroy has been passing every time. Thanks, Darrel Schneider
Review Request 37453: GEODE-77 adding Messenger statistics
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/37453/ --- Review request for geode, anilkumar gingade, Jason Huynh, Jianxia Chen, and Qihong Chen. Repository: geode Description --- This adds the recording of statistics to DistributionStats for the Messenger component of the membership manager. Old statistics that were being recorded by the modified JGroups stack have been removed (stats with JG in their names, primarily). A new unit test ensures that the stat recorder is in place and is properly invoking DistributionStats methods. Diffs - gemfire-core/src/main/java/com/gemstone/gemfire/distributed/internal/membership/gms/messenger/StatRecorder.java PRE-CREATION gemfire-core/src/test/java/com/gemstone/gemfire/distributed/internal/membership/gms/membership/StatRecorderJUnitTest.java PRE-CREATION Diff: https://reviews.apache.org/r/37453/diff/ Testing --- New unit test Thanks, Bruce Schuchardt
Re: 16 days without a green build
For spurious failures, the problematic tests are predominantly dunit tests which are actually end-to-end tests involving 6 jvms. End-to-end tests are notorious for having reliability issues. This is magnified by reusing jvms from one test case to the next. Eventually, one of the 6 jvms hits a problem: 1) a race condition (writting a good distributed test is tricky) or 2) a long stop-the-world GC pause which causes a test to fail with a timeout or pause long enough to hit a more rare race condition. Since the jvms are reused, a test failure is also very likely to pollute the jvm which can in turn cause subsequent test failure(s). The best overall way to improve the reliability of these dunit tests is to fork brand-new jvms for every dunit test case which would force each test case to run in isolation. I think changing build.gradle distributedTest task from forkEvery 30 to forkEvery 1 would help greatly. I'll file a JIRA ticket for forking the dunit jvms every test and propose it on dev. Updating dunit to use junit 4 syntax would help a little by allowing us to use some newer features and rules that are suitable to integration or end-to-end tests. This is already under way with GEODE-217. End-to-end tests are important but have some issues: 1) spurious failures and reliability problems are common 2) excessive redundant code coverage -- they tend to execute the same 10% of the code base 90% of the time 3) long execution time I would be remiss if I didn't use this opportunity to encourage the Geode dev community to refocus on writing true unit tests with Mockito or JMock while moderating the creation of new dunit (end-to-end) tests. We have our automated developer tests categorized as UnitTest, IntegrationTest and DistributedTest. True unit tests written using Mockito or JMock do result in meaningful, worthwhile tests both as tests and as design tools even for a code base such as Geode. These unit tests no less important than integration tests or end-to-end tests (both are still important and all three provide different types of complementary test coverage). Lastly a list of my favorite books (so far) on unit testing: Working Effectively with Legacy Code by Michael Feathers Practical Unit Testing with JUnit and Mockito by Tomek Kaczanowski Effective Unit Testing by Lasse Koskela Growing Object-Oriented Software, Guided by Tests by Steve Freeman and Nat Pryce Working Effectively with Legacy Code in particular covers best practices for breaking dependencies so you can isolate a class for testing it. -Kirk On Thu, Aug 13, 2015 at 12:55 PM, Roman Shaposhnik ro...@shaposhnik.org wrote: Hi! it seems that we're running into various spurious test failures. What's the plan to get it back to green again? Thanks, Roman.
Re: YCSB driver for Geode
I guess I'm not following: why would you have a server side configuration for YCSB? Thanks, Roman. On Thu, Aug 13, 2015 at 2:33 PM, William Markito wmark...@pivotal.io wrote: The current cache.xml I'm looking at is - https://github.com/brianfrankcooper/YCSB/blob/master/gemfire/src/main/conf/cache.xml - So something like this should work for the server: *cache* xmlns=http://schema.pivotal.io/gemfire/cache; xmlns:xsi=http://www.w3.org/2001/XMLSchema-instance; xsi:schemaLocation=http://schema.pivotal.io/gemfire/cache http://schema.pivotal.io/gemfire/cache/cache-8.1.xsd; version=8.1 cache region name=usertable refid=PARTITION/ /cache */cache* Then for the client: client-cache xmlns=http://schema.pivotal.io/gemfire/cache; xmlns:xsi=http://www.w3.org/2001/XMLSchema-instance; xsi:schemaLocation=http://schema.pivotal.io/gemfire/cache http://schema.pivotal.io/gemfire/cache/cache-8.1.xsd; version=8.1” pool name=client subscription-enabled=true locator host=10.0.2.15 port=10334/ /pool region name=usertable region-attributes pool-name=client refid=CACHING_PROXY/ /region /client-cache On Thu, Aug 13, 2015 at 2:23 PM, Roman Shaposhnik ro...@shaposhnik.org wrote: On Thu, Aug 13, 2015 at 1:54 PM, William Markito wmark...@pivotal.io wrote: Hi Roman, There is the cache and client-cache xml. http://gemfire.docs.pivotal.io/latest/userguide/reference/topics/elements_ref.html Sure. What I'm asking is: what is the most reasonable content of cache.xml that gets shipped as part of YCSB. The current content clearly doesn't work. Thanks, Roman. -- William Markito Oliveira Enterprise Architect -- For questions about Apache Geode, please write to *dev@geode.incubator.apache.org dev@geode.incubator.apache.org*
Re: Review Request 37450: GEODE-216: Handle any exception that might be thrown
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/37450/#review95347 --- Ship it! Ship It! - Dan Smith On Aug. 13, 2015, 11:22 p.m., Darrel Schneider wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/37450/ --- (Updated Aug. 13, 2015, 11:22 p.m.) Review request for geode and Dan Smith. Bugs: GEODE-216 https://issues.apache.org/jira/browse/GEODE-216 Repository: geode Description --- The old code was trying to prevent a deadlock under some certain conditions. But if it can not get a PR because of an exception then a deadlock is not possible. So the new code just catches and ignores exceptions and falls through to the code that will process the message the old way. This code will encounter the same exception and has the proper logic to handle it. Diffs - gemfire-core/src/main/java/com/gemstone/gemfire/internal/cache/partitioned/GetMessage.java c8d990d7e2dba1f61a2aa7f9df703c0be6306a9d Diff: https://reviews.apache.org/r/37450/diff/ Testing --- RedisDistDUnitTest was failing for me without this change. With it testConcCreateDestroy has been passing every time. Thanks, Darrel Schneider
Re: 16 days without a green build
I've created GEODE-218 Change distributedTest task to fork every 1 test case for review and group discussion. -Kirk On Thu, Aug 13, 2015 at 5:29 PM, Kirk Lund kirk.l...@gmail.com wrote: For spurious failures, the problematic tests are predominantly dunit tests which are actually end-to-end tests involving 6 jvms. End-to-end tests are notorious for having reliability issues. This is magnified by reusing jvms from one test case to the next. Eventually, one of the 6 jvms hits a problem: 1) a race condition (writting a good distributed test is tricky) or 2) a long stop-the-world GC pause which causes a test to fail with a timeout or pause long enough to hit a more rare race condition. Since the jvms are reused, a test failure is also very likely to pollute the jvm which can in turn cause subsequent test failure(s). The best overall way to improve the reliability of these dunit tests is to fork brand-new jvms for every dunit test case which would force each test case to run in isolation. I think changing build.gradle distributedTest task from forkEvery 30 to forkEvery 1 would help greatly. I'll file a JIRA ticket for forking the dunit jvms every test and propose it on dev. Updating dunit to use junit 4 syntax would help a little by allowing us to use some newer features and rules that are suitable to integration or end-to-end tests. This is already under way with GEODE-217. End-to-end tests are important but have some issues: 1) spurious failures and reliability problems are common 2) excessive redundant code coverage -- they tend to execute the same 10% of the code base 90% of the time 3) long execution time I would be remiss if I didn't use this opportunity to encourage the Geode dev community to refocus on writing true unit tests with Mockito or JMock while moderating the creation of new dunit (end-to-end) tests. We have our automated developer tests categorized as UnitTest, IntegrationTest and DistributedTest. True unit tests written using Mockito or JMock do result in meaningful, worthwhile tests both as tests and as design tools even for a code base such as Geode. These unit tests no less important than integration tests or end-to-end tests (both are still important and all three provide different types of complementary test coverage). Lastly a list of my favorite books (so far) on unit testing: Working Effectively with Legacy Code by Michael Feathers Practical Unit Testing with JUnit and Mockito by Tomek Kaczanowski Effective Unit Testing by Lasse Koskela Growing Object-Oriented Software, Guided by Tests by Steve Freeman and Nat Pryce Working Effectively with Legacy Code in particular covers best practices for breaking dependencies so you can isolate a class for testing it. -Kirk On Thu, Aug 13, 2015 at 12:55 PM, Roman Shaposhnik ro...@shaposhnik.org wrote: Hi! it seems that we're running into various spurious test failures. What's the plan to get it back to green again? Thanks, Roman.
Re: YCSB driver for Geode
Do you have a corresponding error message or indicator of it not being compatible? — Jacob Barrett Manager GemFire Advanced Customer Engineering (ACE) Pivotal jbarr...@pivotal.io 503-533-3763 For immediate support please contact Pivotal Support at http://support.pivotal.io/ On Thu, Aug 13, 2015 at 12:58 PM, Roman Shaposhnik r...@apache.org wrote: Hi! it seems that the gemfire's cache.xml is no longer compatible with Geode. Can someone please help me come up with the update that makes sense? The following seems to work, but I'm not sure how optimal it is: client-cache pool name=client subscription-enabled=true locator host=10.0.2.15 port=10334/ /pool region name=usertable region-attributes pool-name=client refid=CACHING_PROXY/ /region /client-cache Thanks, Roman.
Re: YCSB driver for Geode
On Thu, Aug 13, 2015 at 1:54 PM, William Markito wmark...@pivotal.io wrote: Hi Roman, There is the cache and client-cache xml. http://gemfire.docs.pivotal.io/latest/userguide/reference/topics/elements_ref.html Sure. What I'm asking is: what is the most reasonable content of cache.xml that gets shipped as part of YCSB. The current content clearly doesn't work. Thanks, Roman.
Re: Dynamic classloading in Geode
Thanks for the suggestions Mike. At this point we are just exploring ideas and putting them out for discussion. Regarding restricting access to this feature, we used the Geode Java client so standard security and authorizations would apply. Anthony On Aug 13, 2015, at 12:24 PM, Michael Stolz mst...@pivotal.io wrote: If this feature makes it into an actual release please make sure this option is not enabled by default and is securely turned off for environments where there are strong controls around releasing software into production. Also make sure that it is secured in terms of Authentication and Authorization via the Geode security framework when it is enabled, so that not just anyone can push code. -- Mike Stolz Principal Technical Account Manager Mobile: 631-835-4771 On Thu, Aug 13, 2015 at 1:57 PM, Anthony Baker aba...@pivotal.io mailto:aba...@pivotal.io wrote: Vito and I spent some time hacking up a prototype for dynamic and distributed classloading of Geode functions. Currently a user has to compile a function into a jar and deploy it using gfsh before it can be executed. If we could enable automatic deployment of functions across a running cluster it would speed up the development cycle for Geode applications and pave the way for other interesting features (like Java8 lambdas). Here’s how it works: A function wrapper (DynamicFunction) serializes the original function object and captures dependent classes as byte arrays. We generate an MD5 hash over the bytecode and use that as the key for storing the bytecode in a replicated region (“hackday”) within the cache. When the function is invoked, we call putIfAbsent() to distribute the byte code prior to executing the function across the cluster. During execution, we extend the TCCL with a new class loader that loads classes from our region while the original function is being deserialized. The original function is then executed in parallel on the cluster members. This allows an application developer to iteratively modify and test function code without any manual steps to upload class files. Obviously, there is a lot more thinking and design work to do around these ideas. Here’s our super-hacky code if you’re interested: https://gist.github.com/metatype/9b1f39a24e52f5c6f3e1 https://gist.github.com/metatype/9b1f39a24e52f5c6f3e1 https://gist.github.com/metatype/9b1f39a24e52f5c6f3e1 Caveats: 1) Currently we only capture static class dependencies. Any class dependencies present during method invocations are ignored. This could be addressed by doing byte code inspection (using ASM, javaassist, etc). 2) The region we use to cache class byte code should be automatically recreated as a metadata region, similar to how we store pdx types. We also need to configure eviction and expiration attributes to control resource usage and remove garbage. 3) We only injected the byte code caching hack into the code path for FunctionService.onServers(pool). Also, the putIfAbsent() call adds another network roundtrip. Anthony Vito
Re: YCSB driver for Geode
Hi Roman, There is the cache and client-cache xml. http://gemfire.docs.pivotal.io/latest/userguide/reference/topics/elements_ref.html On Thu, Aug 13, 2015 at 1:16 PM, Roman Shaposhnik ro...@shaposhnik.org wrote: On Thu, Aug 13, 2015 at 1:07 PM, Jacob Barrett jbarr...@pivotal.io wrote: Do you have a corresponding error message or indicator of it not being compatible? Exception in thread Thread-1 java.lang.IllegalStateException: You must use client-cache in the cache.xml when ClientCacheFactory is used. at com.gemstone.gemfire.internal.cache.xmlcache.CacheCreation.create(CacheCreation.java:379) at com.gemstone.gemfire.internal.cache.xmlcache.CacheXmlParser.create(CacheXmlParser.java:320) at com.gemstone.gemfire.internal.cache.GemFireCacheImpl.loadCacheXml(GemFireCacheImpl.java:4248) at com.gemstone.gemfire.internal.cache.GemFireCacheImpl.initializeDeclarativeCache(GemFireCacheImpl.java:1174) at com.gemstone.gemfire.internal.cache.GemFireCacheImpl.init(GemFireCacheImpl.java:1018) at com.gemstone.gemfire.internal.cache.GemFireCacheImpl.create(GemFireCacheImpl.java:678) at com.gemstone.gemfire.cache.client.ClientCacheFactory.basicCreate(ClientCacheFactory.java:207) at com.gemstone.gemfire.cache.client.ClientCacheFactory.create(ClientCacheFactory.java:162) at com.yahoo.ycsb.db.GemFireClient.init(GemFireClient.java:142) at com.yahoo.ycsb.DBWrapper.init(DBWrapper.java:63) at com.yahoo.ycsb.ClientThread.run(Client.java:195) -- William Markito Oliveira Enterprise Architect -- For questions about Apache Geode, please write to *dev@geode.incubator.apache.org dev@geode.incubator.apache.org*
Re: YCSB driver for Geode
On Thu, Aug 13, 2015 at 1:07 PM, Jacob Barrett jbarr...@pivotal.io wrote: Do you have a corresponding error message or indicator of it not being compatible? Exception in thread Thread-1 java.lang.IllegalStateException: You must use client-cache in the cache.xml when ClientCacheFactory is used. at com.gemstone.gemfire.internal.cache.xmlcache.CacheCreation.create(CacheCreation.java:379) at com.gemstone.gemfire.internal.cache.xmlcache.CacheXmlParser.create(CacheXmlParser.java:320) at com.gemstone.gemfire.internal.cache.GemFireCacheImpl.loadCacheXml(GemFireCacheImpl.java:4248) at com.gemstone.gemfire.internal.cache.GemFireCacheImpl.initializeDeclarativeCache(GemFireCacheImpl.java:1174) at com.gemstone.gemfire.internal.cache.GemFireCacheImpl.init(GemFireCacheImpl.java:1018) at com.gemstone.gemfire.internal.cache.GemFireCacheImpl.create(GemFireCacheImpl.java:678) at com.gemstone.gemfire.cache.client.ClientCacheFactory.basicCreate(ClientCacheFactory.java:207) at com.gemstone.gemfire.cache.client.ClientCacheFactory.create(ClientCacheFactory.java:162) at com.yahoo.ycsb.db.GemFireClient.init(GemFireClient.java:142) at com.yahoo.ycsb.DBWrapper.init(DBWrapper.java:63) at com.yahoo.ycsb.ClientThread.run(Client.java:195)
Re: YCSB driver for Geode
Yeah, I just trimmed it to the part that'd be different between the old and the new. So... does it make sense? Thanks, Roman. On Thu, Aug 13, 2015 at 1:55 PM, Jacob Barrett jbarr...@pivotal.io wrote: Your example you posted doesn’t have any scheme declarations. Did you trim them for the email? If not you need to make sure you include the scheme declaration in your xml. ?xml version=1.0? client-cache xmlns=http://schema.pivotal.io/gemfire/cache; xmlns:xsi=http://www.w3.org/2001/XMLSchema-instance; xsi:schemaLocation=http://schema.pivotal.io/gemfire/cache http://schema.pivotal.io/gemfire/cache/cache-8.1.xsd; version=8.1” /client-cache — Jacob Barrett Manager GemFire Advanced Customer Engineering (ACE) Pivotal jbarr...@pivotal.io 503-533-3763 For immediate support please contact Pivotal Support at http://support.pivotal.io/ On Thu, Aug 13, 2015 at 1:16 PM, Roman Shaposhnik ro...@shaposhnik.org wrote: On Thu, Aug 13, 2015 at 1:07 PM, Jacob Barrett jbarr...@pivotal.io wrote: Do you have a corresponding error message or indicator of it not being compatible? Exception in thread Thread-1 java.lang.IllegalStateException: You must use client-cache in the cache.xml when ClientCacheFactory is used. at com.gemstone.gemfire.internal.cache.xmlcache.CacheCreation.create(CacheCreation.java:379) at com.gemstone.gemfire.internal.cache.xmlcache.CacheXmlParser.create(CacheXmlParser.java:320) at com.gemstone.gemfire.internal.cache.GemFireCacheImpl.loadCacheXml(GemFireCacheImpl.java:4248) at com.gemstone.gemfire.internal.cache.GemFireCacheImpl.initializeDeclarativeCache(GemFireCacheImpl.java:1174) at com.gemstone.gemfire.internal.cache.GemFireCacheImpl.init(GemFireCacheImpl.java:1018) at com.gemstone.gemfire.internal.cache.GemFireCacheImpl.create(GemFireCacheImpl.java:678) at com.gemstone.gemfire.cache.client.ClientCacheFactory.basicCreate(ClientCacheFactory.java:207) at com.gemstone.gemfire.cache.client.ClientCacheFactory.create(ClientCacheFactory.java:162) at com.yahoo.ycsb.db.GemFireClient.init(GemFireClient.java:142) at com.yahoo.ycsb.DBWrapper.init(DBWrapper.java:63) at com.yahoo.ycsb.ClientThread.run(Client.java:195)
Re: Dynamic classloading in Geode
If this feature makes it into an actual release please make sure this option is not enabled by default and is securely turned off for environments where there are strong controls around releasing software into production. Also make sure that it is secured in terms of Authentication and Authorization via the Geode security framework when it is enabled, so that not just anyone can push code. -- Mike Stolz Principal Technical Account Manager Mobile: 631-835-4771 On Thu, Aug 13, 2015 at 1:57 PM, Anthony Baker aba...@pivotal.io wrote: Vito and I spent some time hacking up a prototype for dynamic and distributed classloading of Geode functions. Currently a user has to compile a function into a jar and deploy it using gfsh before it can be executed. If we could enable automatic deployment of functions across a running cluster it would speed up the development cycle for Geode applications and pave the way for other interesting features (like Java8 lambdas). Here’s how it works: A function wrapper (DynamicFunction) serializes the original function object and captures dependent classes as byte arrays. We generate an MD5 hash over the bytecode and use that as the key for storing the bytecode in a replicated region (“hackday”) within the cache. When the function is invoked, we call putIfAbsent() to distribute the byte code prior to executing the function across the cluster. During execution, we extend the TCCL with a new class loader that loads classes from our region while the original function is being deserialized. The original function is then executed in parallel on the cluster members. This allows an application developer to iteratively modify and test function code without any manual steps to upload class files. Obviously, there is a lot more thinking and design work to do around these ideas. Here’s our super-hacky code if you’re interested: https://gist.github.com/metatype/9b1f39a24e52f5c6f3e1 https://gist.github.com/metatype/9b1f39a24e52f5c6f3e1 Caveats: 1) Currently we only capture static class dependencies. Any class dependencies present during method invocations are ignored. This could be addressed by doing byte code inspection (using ASM, javaassist, etc). 2) The region we use to cache class byte code should be automatically recreated as a metadata region, similar to how we store pdx types. We also need to configure eviction and expiration attributes to control resource usage and remove garbage. 3) We only injected the byte code caching hack into the code path for FunctionService.onServers(pool). Also, the putIfAbsent() call adds another network roundtrip. Anthony Vito
[Spring CI] Spring Data GemFire Nightly-ApacheGeode #34 was SUCCESSFUL (with 1020 tests)
--- Spring Data GemFire Nightly-ApacheGeode #34 was successful. --- Scheduled 1024 tests in total. https://build.spring.io/browse/SGF-NAG-34/ -- This message is automatically generated by Atlassian Bamboo
Re: Geode and Java 8 Streams
This is pretty neat. Turn this into a feature request and I am sure people will find this valuable. Suds Sent from my iPhone On Aug 13, 2015, at 5:24 PM, Dan Smith dsm...@pivotal.io wrote: Along the same lines as Anthony's email on distributed classloading, QiHong and I also hacked up a prototype of shipping Java 8 stream operations to gemfire data stores. What we did: Java 8 allows people to do functional operations on a region using the new stream API: region.entrySet().stream() .filter(e - e.getKey() % 2 == 0) .map(e - e.getValue()) .reduce(1, Integer::sum); It would be much more efficient if the filtering, mapping, reducing, etc. happen on the members that actual host the data, rather than shipping all the data to a single member. So we implemented a new method, remoteStream, which collects the operations as they are applied to a stream. When a terminal operation like reduce is encountered, the whole pipeline is sent out to all members with a gemfire function, and the results of processing that pipeline on the data stores are brought back. What we did is basically a quick and dirty proof of concept. There's a lot to polish up if we want to make this a real geode feature. Is this something people would be interested in using? The proof of concept code is sitting in github if anyone is interested. Checkout the stream-prototype and look at the tests in StreamsP2PDUnitTest: https://github.com/upthewaterspout/incubator-geode
