JSON implementation
Hi,
I have been looking at Geode-s code and come across major performance killer
for JSON handling in Geode, namely implementation of
com.gemstone.gemfire.pdx.internal.PdxInstanceImpl#getObject
as you can see in the snipped below the code creates ObjectMapper every time it
needs to convert PDX instance into JSON. According to docs and examples on
Jackson’s site instances of ObjectMapper should be shared globally. Creating
it for every transaction is quite expensive in terms of CPU and garbage
collection.
Another thing is date format, in the JSON land it is pretty much settled to be
ISO 8661
https://weblog.west-wind.com/posts/2014/Jan/06/JavaScript-JSON-Date-Parsing-and-real-Dates
It would be nice to be able to have Geode’s JSON standard compliant, or have
this configurable.
public Object getObject() {
if (getPdxType().getNoDomainClass()) {
//In case of Developer Rest APIs, All PdxInstances converted from Json will
have a className =__GEMFIRE_JSON.
//Following code added to convert Json/PdxInstance into the Java object.
if(this.getClassName().equals(__GEMFIRE_JSON)){
//introspect the JSON, does the @type meta-data exist.
String className = extractTypeMetaData();
if(StringUtils.hasText(className)) {
try {
ObjectMapper mapper = new ObjectMapper();
mapper.setDateFormat(new SimpleDateFormat(MM/dd/));
mapper.configure(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES,
false);
mapper.configure(com.fasterxml.jackson.core.JsonParser.Feature.ALLOW_UNQUOTED_FIELD_NAMES,
true);
String JSON = JSONFormatter.toJSON(this);
Object classInstance = mapper.readValue(JSON,
ClassPathLoader.getLatest().forName(className));
return classInstance;
}catch(Exception e){
throw new PdxSerializationException(Could not deserialize as java
class type could not resolved, e);
}
}
}
return this;
}
Regards,
Konstantin Ignatyev
--
This message and any attachments are intended only for the use of the addressee
and may contain information that is privileged and confidential. If the reader
of the message is not the intended recipient or an authorized representative of
the intended recipient, you are hereby notified that any dissemination of this
communication is strictly prohibited. If you have received this communication
in error, notify the sender immediately by return email and delete the message
and any attachments from your system.
Re: JSON implementation
Thanks Konstantin!
Would you mind opening a JIRA about those ?
https://issues.apache.org/jira/browse/GEODE
On Mon, Aug 17, 2015 at 2:51 PM, Ignatyev, Konstantin ignat...@cobalt.com
wrote:
Hi,
I have been looking at Geode-s code and come across major performance
killer for JSON handling in Geode, namely implementation of
com.gemstone.gemfire.pdx.internal.PdxInstanceImpl#getObject
as you can see in the snipped below the code creates ObjectMapper every
time it needs to convert PDX instance into JSON. According to docs and
examples on Jackson’s site instances of ObjectMapper should be shared
globally. Creating it for every transaction is quite expensive in terms of
CPU and garbage collection.
Another thing is date format, in the JSON land it is pretty much settled
to be ISO 8661
https://weblog.west-wind.com/posts/2014/Jan/06/JavaScript-JSON-Date-Parsing-and-real-Dates
It would be nice to be able to have Geode’s JSON standard compliant, or
have this configurable.
public Object getObject() {
if (getPdxType().getNoDomainClass()) {
//In case of Developer Rest APIs, All PdxInstances converted from Json
will have a className =__GEMFIRE_JSON.
//Following code added to convert Json/PdxInstance into the Java
object.
if(this.getClassName().equals(__GEMFIRE_JSON)){
//introspect the JSON, does the @type meta-data exist.
String className = extractTypeMetaData();
if(StringUtils.hasText(className)) {
try {
ObjectMapper mapper = new ObjectMapper();
mapper.setDateFormat(new SimpleDateFormat(MM/dd/));
mapper.configure(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES, false);
mapper.configure(com.fasterxml.jackson.core.JsonParser.Feature.ALLOW_UNQUOTED_FIELD_NAMES,
true);
String JSON = JSONFormatter.toJSON(this);
Object classInstance = mapper.readValue(JSON,
ClassPathLoader.getLatest().forName(className));
return classInstance;
}catch(Exception e){
throw new PdxSerializationException(Could not deserialize as
java class type could not resolved, e);
}
}
}
return this;
}
Regards,
Konstantin Ignatyev
--
This message and any attachments are intended only for the use of the
addressee and may contain information that is privileged and confidential.
If the reader of the message is not the intended recipient or an authorized
representative of the intended recipient, you are hereby notified that any
dissemination of this communication is strictly prohibited. If you have
received this communication in error, notify the sender immediately by
return email and delete the message and any attachments from your system.
--
William Markito Oliveira
Enterprise Architect
-- For questions about Apache Geode, please write to
*dev@geode.incubator.apache.org
dev@geode.incubator.apache.org*
Re: So is it time to dump Hipchat and move to Geode conversations to Slack?
What about Gitter ? https://gitter.im/ On Sun, Aug 16, 2015 at 5:13 PM, Greg Chase gch...@pivotal.io wrote: Email killer is just first round social collaboration software marketing. They'll find something a bit more sophisticated to say once they realize email is not going away. This email encrypted by tiny buttons fat thumbs, beta voice recognition, and autocorrect on my iPhone. On Aug 16, 2015, at 4:25 PM, Roman Shaposhnik ro...@shaposhnik.org wrote: Actually as it happens Slack's desire to be an 'email killer' makes it rather unpopular with ASF as I found out. More details here: https://mail-archives.apache.org/mod_mbox/community-dev/201508.mbox/%3C0A45B62D-00FA-4F66-B357-E0240F9E65A1%40gmail.com%3E Thanks, Roman. On Fri, Aug 14, 2015 at 10:18 AM, Kirk Lund kl...@pivotal.io wrote: Did we ever move to Slack? I don't see any further mention of it. It doesn't look like hipchat is being actively used for Geode anymore. -Kirk On Fri, Jul 17, 2015 at 11:26 AM, John Blum jb...@pivotal.io wrote: +1 for moving to Slack On Fri, Jul 17, 2015 at 11:13 AM, Bruce Schuchardt bschucha...@pivotal.io wrote: If we use Slack, please enable gateway access so we can plug it into Thunderbird. Le 7/17/2015 11:01 AM, Gregory Chase a écrit : Hi all, After than a less than satisfactory experience with HipChat, there's been talk about moving our chat line to Slack, so we can at least have a history. Are the contributors in agreement? -- -John 503-504-8657 john.blum10101 (skype) -- William Markito Oliveira Enterprise Architect -- For questions about Apache Geode, please write to *dev@geode.incubator.apache.org dev@geode.incubator.apache.org*
Re: JSON implementation
Let's discuss this in tomorrow's Apache geode Clubhouse meeting
This email encrypted by tiny buttons fat thumbs, beta voice recognition, and
autocorrect on my iPhone.
On Aug 17, 2015, at 3:21 PM, Ignatyev, Konstantin ignat...@cobalt.com wrote:
Done:
https://issues.apache.org/jira/browse/GEODE-225
https://issues.apache.org/jira/browse/GEODE-226
Konstantin Ignatyev
Enterprise Architect
CDK Global
t 206.219.8381 | c 425.233.4536 | f 206.269.6789
konstantin.ignat...@cdk.commailto:konstantin.ignat...@cdk.com?subject=
cdk.comhttp://www.cdkglobal.com
On Aug 17, 2015, at 3:00 PM, William Markito
wmark...@pivotal.iomailto:wmark...@pivotal.io wrote:
Thanks Konstantin!
Would you mind opening a JIRA about those ?
https://issues.apache.org/jira/browse/GEODE
On Mon, Aug 17, 2015 at 2:51 PM, Ignatyev, Konstantin ignat...@cobalt.com
wrote:
Hi,
I have been looking at Geode-s code and come across major performance
killer for JSON handling in Geode, namely implementation of
com.gemstone.gemfire.pdx.internal.PdxInstanceImpl#getObject
as you can see in the snipped below the code creates ObjectMapper every
time it needs to convert PDX instance into JSON. According to docs and
examples on Jackson’s site instances of ObjectMapper should be shared
globally. Creating it for every transaction is quite expensive in terms of
CPU and garbage collection.
Another thing is date format, in the JSON land it is pretty much settled
to be ISO 8661
https://weblog.west-wind.com/posts/2014/Jan/06/JavaScript-JSON-Date-Parsing-and-real-Dates
It would be nice to be able to have Geode’s JSON standard compliant, or
have this configurable.
public Object getObject() {
if (getPdxType().getNoDomainClass()) {
//In case of Developer Rest APIs, All PdxInstances converted from Json
will have a className =__GEMFIRE_JSON.
//Following code added to convert Json/PdxInstance into the Java
object.
if(this.getClassName().equals(__GEMFIRE_JSON)){
//introspect the JSON, does the @type meta-data exist.
String className = extractTypeMetaData();
if(StringUtils.hasText(className)) {
try {
ObjectMapper mapper = new ObjectMapper();
mapper.setDateFormat(new SimpleDateFormat(MM/dd/));
mapper.configure(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES, false);
mapper.configure(com.fasterxml.jackson.core.JsonParser.Feature.ALLOW_UNQUOTED_FIELD_NAMES,
true);
String JSON = JSONFormatter.toJSON(this);
Object classInstance = mapper.readValue(JSON,
ClassPathLoader.getLatest().forName(className));
return classInstance;
}catch(Exception e){
throw new PdxSerializationException(Could not deserialize as
java class type could not resolved, e);
}
}
}
return this;
}
Regards,
Konstantin Ignatyev
--
This message and any attachments are intended only for the use of the
addressee and may contain information that is privileged and confidential.
If the reader of the message is not the intended recipient or an authorized
representative of the intended recipient, you are hereby notified that any
dissemination of this communication is strictly prohibited. If you have
received this communication in error, notify the sender immediately by
return email and delete the message and any attachments from your system.
--
William Markito Oliveira
Enterprise Architect
-- For questions about Apache Geode, please write to
*dev@geode.incubator.apache.org
dev@geode.incubator.apache.org*
--
This message and any attachments are intended only for the use of the
addressee and may contain information that is privileged and confidential. If
the reader of the message is not the intended recipient or an authorized
representative of the intended recipient, you are hereby notified that any
dissemination of this communication is strictly prohibited. If you have
received this communication in error, notify the sender immediately by return
email and delete the message and any attachments from your system.
Re: Weird networking behavior on Amazon EC via OSv deployment
No. ALL does not count. I remembered that I have to manually specify the range. On Mon, Aug 17, 2015 at 6:06 PM, Roman Shaposhnik ro...@shaposhnik.org wrote: On Mon, Aug 17, 2015 at 5:40 PM, Xiaojian Zhou gz...@pivotal.io wrote: This error msg actually is not the root cause of the issue. To avoid the error msg, you can modify sysctl.conf of your instance to add 2 lines: # unicast buffer size net.core.rmem_max = 2096304 net.core.wmem_max = 2096304 Thanks for the suggestion, but I am actually not running it on Linux (or UNIX for that matter). 2) You might not open enough ports for tcp and udp. That could be the real root cause. To do that, you can either turn the ports (1024-65535) from AWS GUI interface or via following commands: aws ec2 authorize-security-group-ingress --group-id your_security_group_id --protocol tcp --port 22 --cidr 0.0.0.0/00 aws ec2 authorize-security-group-ingress --group-id your_security_group_id --protocol tcp --port 1024-65535 --cidr 0.0.0.0/00 aws ec2 authorize-security-group-ingress --group-id your_security_group_id --protocol udp --port 1024-65535 --cidr 0.0.0.0/00 Hm. That's a good point, but the security group I'm running it from has ALL the ports open so I guess it shouldn't be a problem then, right? Thanks, Roman.
Re: JSON implementation
Done:
https://issues.apache.org/jira/browse/GEODE-225
https://issues.apache.org/jira/browse/GEODE-226
Konstantin Ignatyev
Enterprise Architect
CDK Global
t 206.219.8381 | c 425.233.4536 | f 206.269.6789
konstantin.ignat...@cdk.commailto:konstantin.ignat...@cdk.com?subject=
cdk.comhttp://www.cdkglobal.com
On Aug 17, 2015, at 3:00 PM, William Markito
wmark...@pivotal.iomailto:wmark...@pivotal.io wrote:
Thanks Konstantin!
Would you mind opening a JIRA about those ?
https://issues.apache.org/jira/browse/GEODE
On Mon, Aug 17, 2015 at 2:51 PM, Ignatyev, Konstantin ignat...@cobalt.com
wrote:
Hi,
I have been looking at Geode-s code and come across major performance
killer for JSON handling in Geode, namely implementation of
com.gemstone.gemfire.pdx.internal.PdxInstanceImpl#getObject
as you can see in the snipped below the code creates ObjectMapper every
time it needs to convert PDX instance into JSON. According to docs and
examples on Jackson’s site instances of ObjectMapper should be shared
globally. Creating it for every transaction is quite expensive in terms of
CPU and garbage collection.
Another thing is date format, in the JSON land it is pretty much settled
to be ISO 8661
https://weblog.west-wind.com/posts/2014/Jan/06/JavaScript-JSON-Date-Parsing-and-real-Dates
It would be nice to be able to have Geode’s JSON standard compliant, or
have this configurable.
public Object getObject() {
if (getPdxType().getNoDomainClass()) {
//In case of Developer Rest APIs, All PdxInstances converted from Json
will have a className =__GEMFIRE_JSON.
//Following code added to convert Json/PdxInstance into the Java
object.
if(this.getClassName().equals(__GEMFIRE_JSON)){
//introspect the JSON, does the @type meta-data exist.
String className = extractTypeMetaData();
if(StringUtils.hasText(className)) {
try {
ObjectMapper mapper = new ObjectMapper();
mapper.setDateFormat(new SimpleDateFormat(MM/dd/));
mapper.configure(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES, false);
mapper.configure(com.fasterxml.jackson.core.JsonParser.Feature.ALLOW_UNQUOTED_FIELD_NAMES,
true);
String JSON = JSONFormatter.toJSON(this);
Object classInstance = mapper.readValue(JSON,
ClassPathLoader.getLatest().forName(className));
return classInstance;
}catch(Exception e){
throw new PdxSerializationException(Could not deserialize as
java class type could not resolved, e);
}
}
}
return this;
}
Regards,
Konstantin Ignatyev
--
This message and any attachments are intended only for the use of the
addressee and may contain information that is privileged and confidential.
If the reader of the message is not the intended recipient or an authorized
representative of the intended recipient, you are hereby notified that any
dissemination of this communication is strictly prohibited. If you have
received this communication in error, notify the sender immediately by
return email and delete the message and any attachments from your system.
--
William Markito Oliveira
Enterprise Architect
-- For questions about Apache Geode, please write to
*dev@geode.incubator.apache.org
dev@geode.incubator.apache.org*
--
This message and any attachments are intended only for the use of the addressee
and may contain information that is privileged and confidential. If the reader
of the message is not the intended recipient or an authorized representative of
the intended recipient, you are hereby notified that any dissemination of this
communication is strictly prohibited. If you have received this communication
in error, notify the sender immediately by return email and delete the message
and any attachments from your system.
Re: July Incubator Report
Awesome! I've sent a note to the IPMC. Thanks, Roman. On Mon, Aug 17, 2015 at 1:02 PM, William Markito wmark...@pivotal.io wrote: Folks, since it looks like we should have actually sent our report on July and also to make it easier for everyone to track the project status and numbers, I've just updated our Community Development wiki page. https://cwiki.apache.org/confluence/display/GEODE/Community+development Thanks. -- William Markito Oliveira Enterprise Architect -- For questions about Apache Geode, please write to *dev@geode.incubator.apache.org dev@geode.incubator.apache.org*
[Spring CI] Spring Data GemFire Nightly-ApacheGeode #38 was SUCCESSFUL (with 1058 tests)
--- Spring Data GemFire Nightly-ApacheGeode #38 was successful. --- Scheduled 1062 tests in total. https://build.spring.io/browse/SGF-NAG-38/ -- This message is automatically generated by Atlassian Bamboo
Re: Weird networking behavior on Amazon EC via OSv deployment
On Mon, Aug 17, 2015 at 5:40 PM, Xiaojian Zhou gz...@pivotal.io wrote: This error msg actually is not the root cause of the issue. To avoid the error msg, you can modify sysctl.conf of your instance to add 2 lines: # unicast buffer size net.core.rmem_max = 2096304 net.core.wmem_max = 2096304 Thanks for the suggestion, but I am actually not running it on Linux (or UNIX for that matter). 2) You might not open enough ports for tcp and udp. That could be the real root cause. To do that, you can either turn the ports (1024-65535) from AWS GUI interface or via following commands: aws ec2 authorize-security-group-ingress --group-id your_security_group_id --protocol tcp --port 22 --cidr 0.0.0.0/00 aws ec2 authorize-security-group-ingress --group-id your_security_group_id --protocol tcp --port 1024-65535 --cidr 0.0.0.0/00 aws ec2 authorize-security-group-ingress --group-id your_security_group_id --protocol udp --port 1024-65535 --cidr 0.0.0.0/00 Hm. That's a good point, but the security group I'm running it from has ALL the ports open so I guess it shouldn't be a problem then, right? Thanks, Roman.
Re: Review Request 37209: GEODE-17 : Integrated Security Code Merge
On Aug. 14, 2015, 10:22 p.m., Darrel Schneider wrote: gemfire-core/src/main/java/com/gemstone/gemfire/distributed/DistributedSystem.java, line 592 https://reviews.apache.org/r/37209/diff/1/?file=1034140#file1034140line592 I'm not sure what you are saying in this that requires to manage tokens but I think this might be better that manage tokens Tushar Khairnar wrote: will confirm with Neelkanth. This part was added by me while I was involved with the project initially (before Geode). The javadoc template was copied from that of an exising property (see security-client-auth-init above). Yes, it should be changed to 'that is required to manage tokens' or 'that manage tokens'. On Aug. 14, 2015, 10:22 p.m., Darrel Schneider wrote: gemfire-core/src/main/java/com/gemstone/gemfire/internal/i18n/LocalizedStrings.java, line 1288 https://reviews.apache.org/r/37209/diff/1/?file=1034144#file1034144line1288 Something seems wrong here. It says it is a method name but a method can not implement the TokenService interface. Is this a class name instead? The last sentence also talks about method name and static method so maybe I just don't understand. Put some of this info into the javadocs on the property (like for REST client verification) in DistributedSystem. Also add the before TokenService. Tushar Khairnar wrote: I think Neelkanth meant either its fully classified name of class implementing TokenService or static method returning an instance of TokenService. Again, the format of this statement was copied from another existing StringId constants (e.g. AbstractDistributionConfig_SECURITY_CLIENT_AUTH_INIT_NAME_0) for consistency. But it should not have been ambiguous. Yes, it should really be ... fully qualified method name of a class implementing the TokenService ... Legal values can be any fully qualified \name\ of a static method that returns an instance of the class, present in the classpath. - Amogh --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/37209/#review95439 --- On Aug. 14, 2015, 4:31 p.m., Tushar Khairnar wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/37209/ --- (Updated Aug. 14, 2015, 4:31 p.m.) Review request for geode, Amogh Shetkar, Jens Deppe, and Nilkanth Patel. Repository: geode Description --- GEODE-77 : Integrated Security Code Merge This is manual merge of code from int_security branch. Testing done : JMX RMI-connector testing done from JConsole, Gfsh interactive testing with different roles. DUnits are not yet integrated into open. Adding description about changes done JMX - Key Changes ManagementAgent.java Hooks managementInterceptor when security plugins are configured ManagementInterceptor.java Central interceptor for JMX RMI connector. Each JMX call go through interceptor via MBeanServerWrapper in following fashion jmx(mxbean.op()) - mbeanServerWrapper - interceptor - security plugin - back to wrapper - mxbean.op() ResourceOperationContext OperationContext for all mm resource operations. This returns operation code as RESOURCE (except for data commands) and has additional code called resourceOperationCode which return exact operation requested ResourceOperation This annotation is used to mark mxbean interfaces and commands to corresponding mm action JMXOperationContext describes mbean operation(getAttr,SetAttr,Op) in terms of ResourceOperationContext. Parses all MXBean annotation and build map used for mapping jmx calls to resource codes CLIOperationContext describes gfsh command(name, params) in terms of ResourceOperationContext Parses all Command annotation and build map used for mapping gfsh command calls to resource codes *MXBean and *Commands Changes Added ResourceOperation annotation REST ADMIN - Key Changes AuthManager gateway to authorize and authenticate REST ADMIN internal/web/controllers/AbstractCommandsController.java Changes for ADMIN REST to add authentication and authorization callbacks Pulse - Key Changes from gemfire side AccessControlMXBean/AccessControlContext This is hidden mbean which opens up authorization end-point for Pulse Pulse will access this mbean to know its
Re: Upcoming: Apache Geode Clubhouse Tuesday, August 18th
Reminder: Apache Geode Clubhouse is tomorrow at 9AM PDT. Hope you can join! -Marie On Mon, Aug 10, 2015 at 2:46 PM, Marie Nedich mned...@pivotal.io wrote: Hello Apache Geode Community, There will be another Clubhouse next week on Tuesday, August 18th. If you were not able to attend the one earlier this month, you can access a recording of William Markito’s presentation on Apache Geode Docker: https://youtu.be/DinKpmP9jwo Apache Geode Clubhouse Tuesday, August 18th Start Time: 9AM PDT End Time: 10AM PDT Agenda: Open community roundtable to discuss current concerns, open issues, identify areas for new contributors to get started, and any other topics you would like to bring up. Everyone will be given a chance to talk or ask questions. Please join at least 20 minutes prior to 9AM so we can test mics and cameras. We'd like to get as many talking heads in this roundtable as possible! JOIN: http://bit.ly/1JSmjeN The first time you share your screen, camera, or mic you will be prompted to grant access permission to the player for the camera/microphone in a pop up dialog box. New to Adobe Connect? Test your browser here: http://bit.ly/1fQ4MvW *Sponsored by Pivotal Software Cheers, Marie
