[google-appengine] Re: Datastore Small Operations - datastore.get(key)
I settled with Generic DAO classes for Objectify wrapped by Jersey. I am using Backbone for the frontend MVC - so far, I have to admit that it is shaping up well. Jerff - thanks for the library. On Apr 20, 8:48 am, Jeff Schnitzer j...@infohazard.org wrote: No, get-by-key that reaches the datastore is a Read Operation. However, memcache hits are free. Jeff On Fri, Apr 20, 2012 at 1:56 AM, Peter Han devifr...@gmail.com wrote: However, you can convert queries into keys-only queries followed by batch get()s; this will cost small datastore operations for cache hits rather than full read operations keys-only query is a small datastore operation thats is clear, but batch get of entities by key also? Thank you. On Apr 18, 6:49 pm, Jeff Schnitzer j...@infohazard.org wrote: On Wed, Apr 18, 2012 at 3:36 AM, cloudpre pbx.ku...@gmail.com wrote: Jeff - thanks for the note. I have been trying to add few items in memcache manually. Can put operations also be saved? Let's say I am updating the same query again after few seconds. I'm not sure what you mean here. The cache covers get(), put(), and delete() operations. Queries do not affect the cache at all. However, you can convert queries into keys-only queries followed by batch get()s; this will cost small datastore operations for cache hits rather than full read operations. Objectify4 will actually do this for you. Does it work flawlessly in the production? The last thing I want to see is our thousands of customers coming back and complaining. There have not been any complaints for the current cache code (3.1+), and it is fairly widely used. Older versions of the cache code (3.0 and prior) had synchronization issues, but for 3.1 I rewrote it with some help from Ari and Alfred (Google). It should be transactionally safe - even under heavy contention - as long as you don't hit DeadlineExceededExceptions. Jeff -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group athttp://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: Datastore Small Operations - datastore.get(key)
However, you can convert queries into keys-only queries followed by batch get()s; this will cost small datastore operations for cache hits rather than full read operations keys-only query is a small datastore operation thats is clear, but batch get of entities by key also? Thank you. On Apr 18, 6:49 pm, Jeff Schnitzer j...@infohazard.org wrote: On Wed, Apr 18, 2012 at 3:36 AM, cloudpre pbx.ku...@gmail.com wrote: Jeff - thanks for the note. I have been trying to add few items in memcache manually. Can put operations also be saved? Let's say I am updating the same query again after few seconds. I'm not sure what you mean here. The cache covers get(), put(), and delete() operations. Queries do not affect the cache at all. However, you can convert queries into keys-only queries followed by batch get()s; this will cost small datastore operations for cache hits rather than full read operations. Objectify4 will actually do this for you. Does it work flawlessly in the production? The last thing I want to see is our thousands of customers coming back and complaining. There have not been any complaints for the current cache code (3.1+), and it is fairly widely used. Older versions of the cache code (3.0 and prior) had synchronization issues, but for 3.1 I rewrote it with some help from Ari and Alfred (Google). It should be transactionally safe - even under heavy contention - as long as you don't hit DeadlineExceededExceptions. Jeff -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: Datastore Small Operations - datastore.get(key)
No, get-by-key that reaches the datastore is a Read Operation. However, memcache hits are free. Jeff On Fri, Apr 20, 2012 at 1:56 AM, Peter Han devifr...@gmail.com wrote: However, you can convert queries into keys-only queries followed by batch get()s; this will cost small datastore operations for cache hits rather than full read operations keys-only query is a small datastore operation thats is clear, but batch get of entities by key also? Thank you. On Apr 18, 6:49 pm, Jeff Schnitzer j...@infohazard.org wrote: On Wed, Apr 18, 2012 at 3:36 AM, cloudpre pbx.ku...@gmail.com wrote: Jeff - thanks for the note. I have been trying to add few items in memcache manually. Can put operations also be saved? Let's say I am updating the same query again after few seconds. I'm not sure what you mean here. The cache covers get(), put(), and delete() operations. Queries do not affect the cache at all. However, you can convert queries into keys-only queries followed by batch get()s; this will cost small datastore operations for cache hits rather than full read operations. Objectify4 will actually do this for you. Does it work flawlessly in the production? The last thing I want to see is our thousands of customers coming back and complaining. There have not been any complaints for the current cache code (3.1+), and it is fairly widely used. Older versions of the cache code (3.0 and prior) had synchronization issues, but for 3.1 I rewrote it with some help from Ari and Alfred (Google). It should be transactionally safe - even under heavy contention - as long as you don't hit DeadlineExceededExceptions. Jeff -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: Datastore Small Operations - datastore.get(key)
Hi,
Memcache is a cache, so you should not depend on it being there. If
you need something persisted, you will need to write it to the
datastore. You can stick it in memcache at write time, which will
help your reads though.
Robert
On Wed, Apr 18, 2012 at 02:36, cloudpre pbx.ku...@gmail.com wrote:
Jeff - thanks for the note. I have been trying to add few items in
memcache manually.
Can put operations also be saved? Let's say I am updating the same
query again after few seconds.
Does it work flawlessly in the production? The last thing I want to
see is our thousands of customers coming back and complaining.
Cheers.
On Apr 18, 12:17 am, Jeff Schnitzer j...@infohazard.org wrote:
Get-by-key is a read operation, not a small operation. Small
operations are things like count() index walks and keys-only queries
(although there is still 1 read operation per query).
Most read-heavy apps can benefit significantly from memcache. If
you're using the low-level api, try this:
http://code.google.com/p/objectify-appengine/wiki/MemcacheStandalone
Jeff
On Wed, Apr 18, 2012 at 2:27 AM, cloudpre pbx.ku...@gmail.com wrote:
Hi
We have a pretty big app which has not been optimized till date. We
spend over $40/day just on the reads.
This is our current stats
Datastore Read Operations 46.74 Million Ops 46.69
$0.70/ Million
Ops $32.69
Datastore Small Operations 0.00 Million Ops 0.00
$0.10/ Million
Ops $0.00
I have been reading that small operations are much cheaper and they
are done based on key.
The following function is used at many places - but I do not see
counter moving from zero for small operations.
Is datastore.get(key) a small operation? If not, what should I change
the function to make it count towards small operations.
Thanks.
// Get Entity
public static Entity getEntityFromId(String entityName, String id)
{
// Get Datastore
DatastoreService datastore =
DatastoreServiceFactory.getDatastoreService();
// Get Entity
try
{
Key key = KeyFactory.stringToKey(id);
return datastore.get(key);
} catch (Exception e)
{
return null;
}
}
--
You received this message because you are subscribed to the Google Groups
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group
athttp://groups.google.com/group/google-appengine?hl=en.
--
You received this message because you are subscribed to the Google Groups
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.
--
You received this message because you are subscribed to the Google Groups
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: Datastore Small Operations - datastore.get(key)
On Wed, Apr 18, 2012 at 3:36 AM, cloudpre pbx.ku...@gmail.com wrote: Jeff - thanks for the note. I have been trying to add few items in memcache manually. Can put operations also be saved? Let's say I am updating the same query again after few seconds. I'm not sure what you mean here. The cache covers get(), put(), and delete() operations. Queries do not affect the cache at all. However, you can convert queries into keys-only queries followed by batch get()s; this will cost small datastore operations for cache hits rather than full read operations. Objectify4 will actually do this for you. Does it work flawlessly in the production? The last thing I want to see is our thousands of customers coming back and complaining. There have not been any complaints for the current cache code (3.1+), and it is fairly widely used. Older versions of the cache code (3.0 and prior) had synchronization issues, but for 3.1 I rewrote it with some help from Ari and Alfred (Google). It should be transactionally safe - even under heavy contention - as long as you don't hit DeadlineExceededExceptions. Jeff -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: Datastore Small Operations - datastore.get(key)
Thanks Robert. Jeff - thanks for chipping in. I wanted to check if puts are also cached - eg: entity with same values being written over and over again - will they be actually be written to datastore. I will use 3.1 - I liked the Global cache very much. Twig kept the syntax simpler but I do not like the lack of caching which will affect the high-volume apps. On Apr 18, 9:49 am, Jeff Schnitzer j...@infohazard.org wrote: On Wed, Apr 18, 2012 at 3:36 AM, cloudpre pbx.ku...@gmail.com wrote: Jeff - thanks for the note. I have been trying to add few items in memcache manually. Can put operations also be saved? Let's say I am updating the same query again after few seconds. I'm not sure what you mean here. The cache covers get(), put(), and delete() operations. Queries do not affect the cache at all. However, you can convert queries into keys-only queries followed by batch get()s; this will cost small datastore operations for cache hits rather than full read operations. Objectify4 will actually do this for you. Does it work flawlessly in the production? The last thing I want to see is our thousands of customers coming back and complaining. There have not been any complaints for the current cache code (3.1+), and it is fairly widely used. Older versions of the cache code (3.0 and prior) had synchronization issues, but for 3.1 I rewrote it with some help from Ari and Alfred (Google). It should be transactionally safe - even under heavy contention - as long as you don't hit DeadlineExceededExceptions. Jeff -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: Datastore Small Operations - datastore.get(key)
On Wed, Apr 18, 2012 at 2:45 PM, cloudpre pbx.ku...@gmail.com wrote: Thanks Robert. Jeff - thanks for chipping in. I wanted to check if puts are also cached - eg: entity with same values being written over and over again - will they be actually be written to datastore. The cache is write-through; all put()s go straight to the datastore and then expire the cache content. The cache is refreshed on a subsequent get(). I will use 3.1 - I liked the Global cache very much. Twig kept the syntax simpler but I do not like the lack of caching which will affect the high-volume apps. If you're just using the low-level api, the CachingDatstoreService in the Objectify 3.1 jar is the same as what's in trunk (Objectify4). If you're actually thinking of using Objectify, grab trunk - Objectify4 has a more Twig-ish syntax and has folded in many of Twig's features (with John Patterson's design help). Jeff -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: Datastore Small Operations - datastore.get(key)
Thanks. I will use CachingDatastoreService - As a suggestion, you should put a list of customers using it so that they know that it's proven. On Apr 18, 12:22 pm, Jeff Schnitzer j...@infohazard.org wrote: On Wed, Apr 18, 2012 at 2:45 PM, cloudpre pbx.ku...@gmail.com wrote: Thanks Robert. Jeff - thanks for chipping in. I wanted to check if puts are also cached - eg: entity with same values being written over and over again - will they be actually be written to datastore. The cache is write-through; all put()s go straight to the datastore and then expire the cache content. The cache is refreshed on a subsequent get(). I will use 3.1 - I liked the Global cache very much. Twig kept the syntax simpler but I do not like the lack of caching which will affect the high-volume apps. If you're just using the low-level api, the CachingDatstoreService in the Objectify 3.1 jar is the same as what's in trunk (Objectify4). If you're actually thinking of using Objectify, grab trunk - Objectify4 has a more Twig-ish syntax and has folded in many of Twig's features (with John Patterson's design help). Jeff -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: Datastore Small Operations - datastore.get(key)
Thanks... but I don't have customers. Customers pay. This is free software - I don't really know who is using it, I just take wild guesses based on the # of people who have subscribed to the mailing list and the # of questions on stackoverflow. Jeff On Wed, Apr 18, 2012 at 10:38 PM, cloudpre pbx.ku...@gmail.com wrote: Thanks. I will use CachingDatastoreService - As a suggestion, you should put a list of customers using it so that they know that it's proven. On Apr 18, 12:22 pm, Jeff Schnitzer j...@infohazard.org wrote: On Wed, Apr 18, 2012 at 2:45 PM, cloudpre pbx.ku...@gmail.com wrote: Thanks Robert. Jeff - thanks for chipping in. I wanted to check if puts are also cached - eg: entity with same values being written over and over again - will they be actually be written to datastore. The cache is write-through; all put()s go straight to the datastore and then expire the cache content. The cache is refreshed on a subsequent get(). I will use 3.1 - I liked the Global cache very much. Twig kept the syntax simpler but I do not like the lack of caching which will affect the high-volume apps. If you're just using the low-level api, the CachingDatstoreService in the Objectify 3.1 jar is the same as what's in trunk (Objectify4). If you're actually thinking of using Objectify, grab trunk - Objectify4 has a more Twig-ish syntax and has folded in many of Twig's features (with John Patterson's design help). Jeff -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
