Re: Review Request 37209: GEODE-17 : Integrated Security Code Merge
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/37209/ --- (Updated Aug. 14, 2015, 11:01 a.m.) Review request for geode, Amogh Shetkar, Jens Deppe, and Nilkanth Patel. Repository: geode Description (updated) --- GEODE-77 : Integrated Security Code Merge This is manual merge of code from int_security branch. Testing done : JMX RMI-connector testing done from JConsole, Gfsh interactive testing with different roles. DUnits are not yet integrated into open. Adding description about changes done JMX - Key Changes ManagementAgent.java Hooks managementInterceptor when security plugins are configured ManagementInterceptor.java Central interceptor for JMX RMI connector. Each JMX call go through interceptor via MBeanServerWrapper in following fashion jmx(mxbean.op()) - mbeanServerWrapper - interceptor - security plugin - back to wrapper - mxbean.op() ResourceOperationContext OperationContext for all mm resource operations. This returns operation code as RESOURCE (except for data commands) and has additional code called resourceOperationCode which return exact operation requested ResourceOperation This annotation is used to mark mxbean interfaces and commands to corresponding mm action JMXOperationContext describes mbean operation(getAttr,SetAttr,Op) in terms of ResourceOperationContext. Parses all MXBean annotation and build map used for mapping jmx calls to resource codes CLIOperationContext describes gfsh command(name, params) in terms of ResourceOperationContext Parses all Command annotation and build map used for mapping gfsh command calls to resource codes *MXBean and *Commands Changes Added ResourceOperation annotation REST ADMIN - Key Changes AuthManager gateway to authorize and authenticate REST ADMIN internal/web/controllers/AbstractCommandsController.java Changes for ADMIN REST to add authentication and authorization callbacks Pulse - Key Changes from gemfire side AccessControlMXBean/AccessControlContext This is hidden mbean which opens up authorization end-point for Pulse Pulse will access this mbean to know its authorization levels after connecting with given credentials Any JMX Client can use this mbean to know its (currrent jmx connection) authorization levels REST - Key changes gemfire-web-api - AbstractBaseController.java and other controller classes REST API changes for At Az DistributionConfig (its impl) New system properties token-service for REST TokenService New interface for REST endpoint which is supposed to give secured token when given Princial RestAPIsOperationContext OperationContext for REST API Diffs - gemfire-core/src/main/java/com/gemstone/gemfire/cache/operations/OperationContext.java d25063c gemfire-core/src/main/java/com/gemstone/gemfire/distributed/DistributedSystem.java b7b2cd8 gemfire-core/src/main/java/com/gemstone/gemfire/distributed/internal/AbstractDistributionConfig.java 472959d gemfire-core/src/main/java/com/gemstone/gemfire/distributed/internal/DistributionConfig.java 10094a9 gemfire-core/src/main/java/com/gemstone/gemfire/distributed/internal/DistributionConfigImpl.java b8dfeb3 gemfire-core/src/main/java/com/gemstone/gemfire/internal/i18n/LocalizedStrings.java f5ae3e5 gemfire-core/src/main/java/com/gemstone/gemfire/internal/security/AuthorizeRequest.java 8ba07a2 gemfire-core/src/main/java/com/gemstone/gemfire/management/CacheServerMXBean.java 59f6537 gemfire-core/src/main/java/com/gemstone/gemfire/management/DiskStoreMXBean.java f14d16c gemfire-core/src/main/java/com/gemstone/gemfire/management/DistributedSystemMXBean.java f0a0a79 gemfire-core/src/main/java/com/gemstone/gemfire/management/GatewayReceiverMXBean.java 3e5ba1a gemfire-core/src/main/java/com/gemstone/gemfire/management/GatewaySenderMXBean.java b6c5219 gemfire-core/src/main/java/com/gemstone/gemfire/management/LockServiceMXBean.java e53d50a gemfire-core/src/main/java/com/gemstone/gemfire/management/ManagerMXBean.java 04fda7e gemfire-core/src/main/java/com/gemstone/gemfire/management/MemberMXBean.java e935fcd gemfire-core/src/main/java/com/gemstone/gemfire/management/internal/ManagementAgent.java 43bfe73
Re: Review Request 37453: GEODE-77 adding Messenger statistics
On Aug. 14, 2015, 12:15 a.m., anilkumar gingade wrote: gemfire-core/src/main/java/com/gemstone/gemfire/distributed/internal/membership/gms/messenger/StatRecorder.java, line 39 https://reviews.apache.org/r/37453/diff/1/?file=1039666#file1039666line39 Is there a chance, where this is not called...We are not doing null checks for this.stats in other places. Stats are inserted into the StatRecorder in JGroupsMessenger.init(), before the JGroups channel is started. No message events are passed up or down the protocol stack until the channel is started in JGroupsMessenger.start(). - Bruce --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/37453/#review95358 --- On Aug. 13, 2015, 11:44 p.m., Bruce Schuchardt wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/37453/ --- (Updated Aug. 13, 2015, 11:44 p.m.) Review request for geode, anilkumar gingade, Jason Huynh, Jianxia Chen, and Qihong Chen. Repository: geode Description --- This adds the recording of statistics to DistributionStats for the Messenger component of the membership manager. Old statistics that were being recorded by the modified JGroups stack have been removed (stats with JG in their names, primarily). A new unit test ensures that the stat recorder is in place and is properly invoking DistributionStats methods. Diffs - gemfire-core/src/main/java/com/gemstone/gemfire/distributed/internal/membership/gms/messenger/StatRecorder.java PRE-CREATION gemfire-core/src/test/java/com/gemstone/gemfire/distributed/internal/membership/gms/membership/StatRecorderJUnitTest.java PRE-CREATION Diff: https://reviews.apache.org/r/37453/diff/ Testing --- New unit test Thanks, Bruce Schuchardt
Re: Review Request 37486: GEODE-214: better Azul support
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/37486/#review95470 --- Ship it! Ship It! gemfire-core/src/main/java/com/gemstone/gemfire/internal/SharedLibrary.java (line 112) https://reviews.apache.org/r/37486/#comment150432 I would recommend fixing the indentation of this method. Also need to make sure the assumption of compressed oops is not used when using Zing JVM. - Kirk Lund On Aug. 14, 2015, 5:22 p.m., Darrel Schneider wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/37486/ --- (Updated Aug. 14, 2015, 5:22 p.m.) Review request for geode and Kirk Lund. Bugs: GEODE-214 https://issues.apache.org/jira/browse/GEODE-214 Repository: geode Description --- - isTenured now looks for GenPauseless Old Gen for azul. - Fatal message about jvm version no longer logged for azul jvm. - Fatal log message about jvm version is now a warning since we continue to run. - Fixed a bug in ReflectionSingleObjectSizer in how it calculated the size of a field It was calling Field.getClass instead of Field.getType. Field.getClass always returns and instance of Field.class which the sizer always says is an objref size. getType will return a primitive class for primitive fields which is what we want. - Improved ObjectSizerJUnitTest. - The object header size and reference size are now correctly computed for azul. - Fix tests that are unintentionally spawning processes that use the default mcast-port. This fixes intermittent failures caused by finding another member of the wrong GemFire version. Diffs - gemfire-core/src/main/java/com/gemstone/gemfire/internal/SharedLibrary.java 59ab34e8c13e1e47cfb5b918f551b4175855409a gemfire-core/src/main/java/com/gemstone/gemfire/internal/cache/MinimumSystemRequirements.java d0a346faaf9d9aa3f12ccf289de2bdb6910e3d51 gemfire-core/src/main/java/com/gemstone/gemfire/internal/cache/control/HeapMemoryMonitor.java fda337b94f34d0ce2c8211152f82609325b29438 gemfire-core/src/main/java/com/gemstone/gemfire/internal/lang/SystemUtils.java f4f111ac0f108b326252786f925a669f85061634 gemfire-core/src/main/java/com/gemstone/gemfire/internal/size/ReflectionSingleObjectSizer.java 47b73a8de2db21abd415ba7e71d3fc4eb276eaef gemfire-core/src/test/java/com/gemstone/gemfire/distributed/LocatorLauncherRemoteFileJUnitTest.java b4814619b821f20d7d5c8cab65231f77f3eaf31e gemfire-core/src/test/java/com/gemstone/gemfire/distributed/LocatorLauncherRemoteJUnitTest.java 25aa23cc10dc4037d1b1277127c794be62e2cf79 gemfire-core/src/test/java/com/gemstone/gemfire/distributed/ServerLauncherLocalJUnitTest.java 1addfc89427223b330ce6f476d1b37de7f058dd7 gemfire-core/src/test/java/com/gemstone/gemfire/distributed/ServerLauncherRemoteFileJUnitTest.java 3b321edba497e7c2a3d4fb0802128b123218899e gemfire-core/src/test/java/com/gemstone/gemfire/distributed/ServerLauncherWithSpringJUnitTest.java dbefdbdc06cd4174321c6483216f8e38ae478cd3 gemfire-core/src/test/java/com/gemstone/gemfire/internal/size/ObjectSizerJUnitTest.java 3d6a73189924cfb9a4bb86c81a60be730589ec2a Diff: https://reviews.apache.org/r/37486/diff/ Testing --- precheckin Thanks, Darrel Schneider
Review Request 37486: GEODE-214: better Azul support
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/37486/ --- Review request for geode and Kirk Lund. Bugs: GEODE-214 https://issues.apache.org/jira/browse/GEODE-214 Repository: geode Description --- - isTenured now looks for GenPauseless Old Gen for azul. - Fatal message about jvm version no longer logged for azul jvm. - Fatal log message about jvm version is now a warning since we continue to run. - Fixed a bug in ReflectionSingleObjectSizer in how it calculated the size of a field It was calling Field.getClass instead of Field.getType. Field.getClass always returns and instance of Field.class which the sizer always says is an objref size. getType will return a primitive class for primitive fields which is what we want. - Improved ObjectSizerJUnitTest. - The object header size and reference size are now correctly computed for azul. - Fix tests that are unintentionally spawning processes that use the default mcast-port. This fixes intermittent failures caused by finding another member of the wrong GemFire version. Diffs - gemfire-core/src/main/java/com/gemstone/gemfire/internal/SharedLibrary.java 59ab34e8c13e1e47cfb5b918f551b4175855409a gemfire-core/src/main/java/com/gemstone/gemfire/internal/cache/MinimumSystemRequirements.java d0a346faaf9d9aa3f12ccf289de2bdb6910e3d51 gemfire-core/src/main/java/com/gemstone/gemfire/internal/cache/control/HeapMemoryMonitor.java fda337b94f34d0ce2c8211152f82609325b29438 gemfire-core/src/main/java/com/gemstone/gemfire/internal/lang/SystemUtils.java f4f111ac0f108b326252786f925a669f85061634 gemfire-core/src/main/java/com/gemstone/gemfire/internal/size/ReflectionSingleObjectSizer.java 47b73a8de2db21abd415ba7e71d3fc4eb276eaef gemfire-core/src/test/java/com/gemstone/gemfire/distributed/LocatorLauncherRemoteFileJUnitTest.java b4814619b821f20d7d5c8cab65231f77f3eaf31e gemfire-core/src/test/java/com/gemstone/gemfire/distributed/LocatorLauncherRemoteJUnitTest.java 25aa23cc10dc4037d1b1277127c794be62e2cf79 gemfire-core/src/test/java/com/gemstone/gemfire/distributed/ServerLauncherLocalJUnitTest.java 1addfc89427223b330ce6f476d1b37de7f058dd7 gemfire-core/src/test/java/com/gemstone/gemfire/distributed/ServerLauncherRemoteFileJUnitTest.java 3b321edba497e7c2a3d4fb0802128b123218899e gemfire-core/src/test/java/com/gemstone/gemfire/distributed/ServerLauncherWithSpringJUnitTest.java dbefdbdc06cd4174321c6483216f8e38ae478cd3 gemfire-core/src/test/java/com/gemstone/gemfire/internal/size/ObjectSizerJUnitTest.java 3d6a73189924cfb9a4bb86c81a60be730589ec2a Diff: https://reviews.apache.org/r/37486/diff/ Testing --- precheckin Thanks, Darrel Schneider
Re: Review Request 37485: Remove unnecessary wait-and-assert for process termination
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/37485/#review95444 --- Ship it! Ship It! - Darrel Schneider On Aug. 14, 2015, 9:52 a.m., Kirk Lund wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/37485/ --- (Updated Aug. 14, 2015, 9:52 a.m.) Review request for geode and Darrel Schneider. Bugs: GEODE-209 https://issues.apache.org/jira/browse/GEODE-209 Repository: geode Description --- The waiting for and asserting of process termination is taking more than 5 minutes on ASF infrastructure. This is not necessary for the test and I don't want to increase the wait time so I'm removing it from the test and replace it with a call to destroy the process. Diffs - gemfire-core/src/test/java/com/gemstone/gemfire/test/golden/FailWithTimeoutOfWaitForOutputToMatchJUnitTest.java 1b8a312 Diff: https://reviews.apache.org/r/37485/diff/ Testing --- FailWithTimeoutOfWaitForOutputToMatchJUnitTest GoldenTestFrameworkTestSuite check Thanks, Kirk Lund
Re: Review Request 37209: GEODE-17 : Integrated Security Code Merge
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/37209/#review95439 --- gemfire-core/src/main/java/com/gemstone/gemfire/cache/operations/OperationContext.java (line 82) https://reviews.apache.org/r/37209/#comment150374 whe was 22 skipped? gemfire-core/src/main/java/com/gemstone/gemfire/cache/operations/OperationContext.java (line 88) https://reviews.apache.org/r/37209/#comment150375 why 28? seems like it should be 26 (last ordinal + 1). gemfire-core/src/main/java/com/gemstone/gemfire/cache/operations/OperationContext.java (line 101) https://reviews.apache.org/r/37209/#comment150376 for new external APIs add @since Geode 1.0 gemfire-core/src/main/java/com/gemstone/gemfire/cache/operations/OperationContext.java (line 307) https://reviews.apache.org/r/37209/#comment150373 Destory should be Destroy. Also it is called delete in the javadocs and OP_DELETE_QUERY. They should be consistent. gemfire-core/src/main/java/com/gemstone/gemfire/distributed/DistributedSystem.java (line 592) https://reviews.apache.org/r/37209/#comment150377 I'm not sure what you are saying in this that requires to manage tokens but I think this might be better that manage tokens gemfire-core/src/main/java/com/gemstone/gemfire/distributed/DistributedSystem.java (line 594) https://reviews.apache.org/r/37209/#comment150380 I couldn't find the TokenService interface in this code review even though it is mentioned on the Description. gemfire-core/src/main/java/com/gemstone/gemfire/internal/i18n/LocalizedStrings.java (line 1288) https://reviews.apache.org/r/37209/#comment150378 Something seems wrong here. It says it is a method name but a method can not implement the TokenService interface. Is this a class name instead? The last sentence also talks about method name and static method so maybe I just don't understand. Put some of this info into the javadocs on the property (like for REST client verification) in DistributedSystem. Also add the before TokenService. - Darrel Schneider On Aug. 14, 2015, 4:01 a.m., Tushar Khairnar wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/37209/ --- (Updated Aug. 14, 2015, 4:01 a.m.) Review request for geode, Amogh Shetkar, Jens Deppe, and Nilkanth Patel. Repository: geode Description --- GEODE-77 : Integrated Security Code Merge This is manual merge of code from int_security branch. Testing done : JMX RMI-connector testing done from JConsole, Gfsh interactive testing with different roles. DUnits are not yet integrated into open. Adding description about changes done JMX - Key Changes ManagementAgent.java Hooks managementInterceptor when security plugins are configured ManagementInterceptor.java Central interceptor for JMX RMI connector. Each JMX call go through interceptor via MBeanServerWrapper in following fashion jmx(mxbean.op()) - mbeanServerWrapper - interceptor - security plugin - back to wrapper - mxbean.op() ResourceOperationContext OperationContext for all mm resource operations. This returns operation code as RESOURCE (except for data commands) and has additional code called resourceOperationCode which return exact operation requested ResourceOperation This annotation is used to mark mxbean interfaces and commands to corresponding mm action JMXOperationContext describes mbean operation(getAttr,SetAttr,Op) in terms of ResourceOperationContext. Parses all MXBean annotation and build map used for mapping jmx calls to resource codes CLIOperationContext describes gfsh command(name, params) in terms of ResourceOperationContext Parses all Command annotation and build map used for mapping gfsh command calls to resource codes *MXBean and *Commands Changes Added ResourceOperation annotation REST ADMIN - Key Changes AuthManager gateway to authorize and authenticate REST ADMIN internal/web/controllers/AbstractCommandsController.java Changes for ADMIN REST to add authentication and authorization callbacks Pulse - Key Changes from gemfire side AccessControlMXBean/AccessControlContext This is hidden mbean which opens up authorization end-point for Pulse Pulse will access this mbean to know its authorization levels after connecting with
Re: So is it time to dump Hipchat and move to Geode conversations to Slack?
Did we ever move to Slack? I don't see any further mention of it. It doesn't look like hipchat is being actively used for Geode anymore. -Kirk On Fri, Jul 17, 2015 at 11:26 AM, John Blum jb...@pivotal.io wrote: +1 for moving to Slack On Fri, Jul 17, 2015 at 11:13 AM, Bruce Schuchardt bschucha...@pivotal.io wrote: If we use Slack, please enable gateway access so we can plug it into Thunderbird. Le 7/17/2015 11:01 AM, Gregory Chase a écrit : Hi all, After than a less than satisfactory experience with HipChat, there's been talk about moving our chat line to Slack, so we can at least have a history. Are the contributors in agreement? -- -John 503-504-8657 john.blum10101 (skype)
Re: Geode now has a Zeppelin interpreter!
AWESOME! On Fri, Aug 14, 2015 at 2:02 PM, William Markito wmark...@pivotal.io wrote: Thanks to Christian, Geode now has a native OQL interpreter on Apache Zeppelin! Just tweeted a picture of a running example here: https://twitter.com/william_markito/status/632295226313314304 For more details please check the Zeppelin ticket - https://issues.apache.org/jira/browse/ZEPPELIN-189 I'll definitely be using this as part of upcoming QCon Rio presentation. -- William Markito Oliveira Enterprise Architect -- For questions about Apache Geode, please write to *dev@geode.incubator.apache.org dev@geode.incubator.apache.org* -- Greg Chase Director of Big Data Communities http://www.pivotal.io/big-data Pivotal Software http://www.pivotal.io/ 650-215-0477 @GregChase Blog: http://geekmarketing.biz/
Geode now has a Zeppelin interpreter!
Thanks to Christian, Geode now has a native OQL interpreter on Apache Zeppelin! Just tweeted a picture of a running example here: https://twitter.com/william_markito/status/632295226313314304 For more details please check the Zeppelin ticket - https://issues.apache.org/jira/browse/ZEPPELIN-189 I'll definitely be using this as part of upcoming QCon Rio presentation. -- William Markito Oliveira Enterprise Architect -- For questions about Apache Geode, please write to *dev@geode.incubator.apache.org dev@geode.incubator.apache.org*
Review Request 37492: GEODE-220: Use wait after off-heap destroyRegion
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/37492/ --- Review request for geode and Kirk Lund. Bugs: GEODE-220 https://issues.apache.org/jira/browse/GEODE-220 Repository: geode Description --- GEODE-220: Use wait after off-heap destroyRegion Diffs - gemfire-core/src/test/java/com/gemstone/gemfire/cache30/MultiVMRegionTestCase.java ecec93224743389198fa0e9e1b9096518c82339a Diff: https://reviews.apache.org/r/37492/diff/ Testing --- DistributedNoAckRegionOffHeapDUnitTest Thanks, Darrel Schneider
Re: Geode now has a Zeppelin interpreter!
Sweet! I look forward to playing with this. -Dan On Fri, Aug 14, 2015 at 2:02 PM, William Markito wmark...@pivotal.io wrote: Thanks to Christian, Geode now has a native OQL interpreter on Apache Zeppelin! Just tweeted a picture of a running example here: https://twitter.com/william_markito/status/632295226313314304 For more details please check the Zeppelin ticket - https://issues.apache.org/jira/browse/ZEPPELIN-189 I'll definitely be using this as part of upcoming QCon Rio presentation. -- William Markito Oliveira Enterprise Architect -- For questions about Apache Geode, please write to *dev@geode.incubator.apache.org dev@geode.incubator.apache.org*
Re: Geode now has a Zeppelin interpreter!
Awesome!! On 15 Aug 2015 7:03 am, William Markito wmark...@pivotal.io wrote: Thanks to Christian, Geode now has a native OQL interpreter on Apache Zeppelin! Just tweeted a picture of a running example here: https://twitter.com/william_markito/status/632295226313314304 For more details please check the Zeppelin ticket - https://issues.apache.org/jira/browse/ZEPPELIN-189 I'll definitely be using this as part of upcoming QCon Rio presentation. -- William Markito Oliveira Enterprise Architect -- For questions about Apache Geode, please write to *dev@geode.incubator.apache.org dev@geode.incubator.apache.org*
Re: Dynamic classloading in Geode
Is this going to be implementation for deploy jars. If yes then GEODE-17 (integrated security) will subject it for authorization scrutiny. On Fri, Aug 14, 2015 at 1:50 AM, Anthony Baker aba...@pivotal.io wrote: Thanks for the suggestions Mike. At this point we are just exploring ideas and putting them out for discussion. Regarding restricting access to this feature, we used the Geode Java client so standard security and authorizations would apply. Anthony On Aug 13, 2015, at 12:24 PM, Michael Stolz mst...@pivotal.io wrote: If this feature makes it into an actual release please make sure this option is not enabled by default and is securely turned off for environments where there are strong controls around releasing software into production. Also make sure that it is secured in terms of Authentication and Authorization via the Geode security framework when it is enabled, so that not just anyone can push code. -- Mike Stolz Principal Technical Account Manager Mobile: 631-835-4771 On Thu, Aug 13, 2015 at 1:57 PM, Anthony Baker aba...@pivotal.io mailto:aba...@pivotal.io wrote: Vito and I spent some time hacking up a prototype for dynamic and distributed classloading of Geode functions. Currently a user has to compile a function into a jar and deploy it using gfsh before it can be executed. If we could enable automatic deployment of functions across a running cluster it would speed up the development cycle for Geode applications and pave the way for other interesting features (like Java8 lambdas). Here’s how it works: A function wrapper (DynamicFunction) serializes the original function object and captures dependent classes as byte arrays. We generate an MD5 hash over the bytecode and use that as the key for storing the bytecode in a replicated region (“hackday”) within the cache. When the function is invoked, we call putIfAbsent() to distribute the byte code prior to executing the function across the cluster. During execution, we extend the TCCL with a new class loader that loads classes from our region while the original function is being deserialized. The original function is then executed in parallel on the cluster members. This allows an application developer to iteratively modify and test function code without any manual steps to upload class files. Obviously, there is a lot more thinking and design work to do around these ideas. Here’s our super-hacky code if you’re interested: https://gist.github.com/metatype/9b1f39a24e52f5c6f3e1 https://gist.github.com/metatype/9b1f39a24e52f5c6f3e1 https://gist.github.com/metatype/9b1f39a24e52f5c6f3e1 Caveats: 1) Currently we only capture static class dependencies. Any class dependencies present during method invocations are ignored. This could be addressed by doing byte code inspection (using ASM, javaassist, etc). 2) The region we use to cache class byte code should be automatically recreated as a metadata region, similar to how we store pdx types. We also need to configure eviction and expiration attributes to control resource usage and remove garbage. 3) We only injected the byte code caching hack into the code path for FunctionService.onServers(pool). Also, the putIfAbsent() call adds another network roundtrip. Anthony Vito
Jenkins build is back to normal : Geode-nightly #184
See https://builds.apache.org/job/Geode-nightly/184/changes
