[appengine-java] Deploying two apps to one datastore memcache
I currently have an app which has two distinct parts. One is a public facing part which relies on basic servlets and very few dependencies - it is dynamic and uses the datastore and memcache extensively but I need this part to scale quickly and efficiently with as low a cost as possible. The second part is essentially an administration interface, it's much heavier, makes use of Guice, Cambridge templates, Objectify and I'm just about to add a fairly heavy library. I can make the first part load on app engine in just over a second, but once I add in the extra features needed for the second part, start- up takes over 10 seconds. This is probably fine, but I'd like to serve people quicker than that, and I might get lots of traffic very suddenly which could cause lots of new instances to start at once. So, is it possible to separate these but have them access the same datastore and memcache? Can I use different versions of the app for this? Is that the only way? Will different versions load separately in a way that will work correctly for scaling? -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
[appengine-java] Re: Users changing data
I can now confirm that I do not see this behaviour in production -
only on the dev server.
On Nov 4, 9:01 am, Matthew Jaggard matt...@jaggard.org.uk wrote:
Sure, thanks Ikai, I should have done this first time around.
When putting the entity (code from around my app stuck together)
log(Storing user {0}, us.getCurrentUser().getUserId());
DatastoreServiceConfig DATASTORE_CONFIG =
DatastoreServiceConfig.Builder.withDeadline(5);
DATASTORE_CONFIG.readPolicy(new ReadPolicy(ReadPolicy.Consistency.EVENTUAL));
DatastoreService ds =
DatastoreServiceFactory.getDatastoreService(DATASTORE_CONFIG)
Entity e = new Entity(UserForTesting);
e.setProperty(User, us.getCurrentUser());
ds.put(e);
In a separate request (and I am sure that I'm looking at the same
object!) (Dull Java code removed)
PreparedQuery pq = ds.prepare(new Query(UserForTesting));
...
pq.asIterable()
...
e.getProperties()
...
log(u.getUserId());
log('/');
log(u.getNickname());
...
On 4 November 2011 01:05, Ikai Lan (Google) ika...@google.com wrote:
Can you post code? It's not clear to me what you're doing.
--
Ikai Lan
Developer Programs Engineer, Google App Engine
plus.ikailan.com | twitter.com/ikai
On Thu, Nov 3, 2011 at 4:39 PM, Mat Jaggard matt...@jaggard.org.uk wrote:
I have an entity that stores a user, however the user gets changed.
When I put the entity in to the local datastore, the ID is like this
18530476822013922411 but when I get it out the ID is like this
-1403876245. I haven't tried on production.
Any ideas what's causing the issue? Have I done something wrong?
Thanks,
Mat.
--
You received this message because you are subscribed to the Google Groups
Google App Engine for Java group.
To post to this group, send email to
google-appengine-java@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine-java+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine-java?hl=en.
--
You received this message because you are subscribed to the Google Groups
Google App Engine for Java group.
To post to this group, send email to google-appengine-java@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine-java+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine-java?hl=en.
--
You received this message because you are subscribed to the Google Groups
Google App Engine for Java group.
To post to this group, send email to google-appengine-java@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine-java+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine-java?hl=en.
[appengine-java] Users changing data
I have an entity that stores a user, however the user gets changed. When I put the entity in to the local datastore, the ID is like this 18530476822013922411 but when I get it out the ID is like this -1403876245. I haven't tried on production. Any ideas what's causing the issue? Have I done something wrong? Thanks, Mat. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
[appengine-java] Re: Error 500 Expecting a stackmap frame at branch target...
I resolved this error by manually deleting all my class files (eclipse wasn't always doing this on a Clean for some reason) and then recompiling and running using JRE6 instead of JRE7. I have no idea why that was required - maybe some bug in the eclipse java compiler? I wish they'd just use the real java compiler. :-( On Oct 20, 3:39 pm, Mat Jaggard matt...@jaggard.org.uk wrote: I've recently switched to eclipse for my GAE development from Netbeans. I find it quite hard to change my mindset - I got Ant files and eclipse's compilation methods make no sense to me. Anyway, I'm not really looking for a discussion on that, I've changed because the Netbeans support for GAE is a bit lacking - especially when it comes to deploying without a reload. I also use Windows and Linux with the same project at different times. On windows with both eclipse and Netbeans, I currently get a runtime error when running my project using GAE SDK 1.5.5 and accessing a URL. Full stack trace is here:http://pastebin.com/Jqed6gz4 Any suggestions as to what could cause this? Thanks, Mat. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
[appengine-java] Error 500 Expecting a stackmap frame at branch target...
I've recently switched to eclipse for my GAE development from Netbeans. I find it quite hard to change my mindset - I got Ant files and eclipse's compilation methods make no sense to me. Anyway, I'm not really looking for a discussion on that, I've changed because the Netbeans support for GAE is a bit lacking - especially when it comes to deploying without a reload. I also use Windows and Linux with the same project at different times. On windows with both eclipse and Netbeans, I currently get a runtime error when running my project using GAE SDK 1.5.5 and accessing a URL. Full stack trace is here: http://pastebin.com/Jqed6gz4 Any suggestions as to what could cause this? Thanks, Mat. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
[appengine-java] Re: GAE - Vote counting system
Sorry to bring this thread back up again, but I've noticed quite a lot of issues being posted to this and other groups about the task queue system failing - and scrolling back through the issues page (http:// code.google.com/status/appengine) it's always the taskqueue that has problems. Those of you who use it - are you finding it safe enough for production use with the kind of volumes and importance that's been mentioned here? Just wondering whether to re-factor my code to this pattern or continue writing too much to the datastore on each request. Thanks, Mat. On Oct 3, 7:34 pm, Jeff Schnitzer j...@infohazard.org wrote: On Mon, Oct 3, 2011 at 9:24 AM, Mat Jaggard matjagg...@gmail.com wrote: Jeff - I'm a bit confused. I thought that the whole idea of the datastore was that you could read or write as much as you want, as fast as you want as long as they are not related? So one datastore write per vote (and being written to different entity groups) should be fine? I thought that the system just split tablets if they were being accessed too much - so as long as the traffic didn't suddenly increase, there'd be no scalability issues apart from cost. apart from cost he says :-) The OP posited millions of users and millions of things to vote for. Each million votes will cost you (at minimum) $1.70 for one write + one read, but it'll probably be more depending on how many page views you have and what caching strategy you have. Still, maybe this is no big deal. The bigger problem though is that vote traffic is likely to be focused on a handful of items. Popular things might get thousands of votes per second, unpopular things won't be voted for at all. It's hard to come up with a sharding strategy that works well for this - you probably don't want 1k shards for everything, storage costs go up and expense/latency of calculating totals goes up. I have to deal with a similar problem myself right now (with the added constraint that I need an instantaneously precise count). I'm considering a system that automatically tracks latency and increases the shard count when it crosses a threshold. It's not a pretty problem to solve. Jeff -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
[appengine-java] Re: Help with sharded counters and loading results.
This is really similar to the thread entitled GAE - Vote counting system - have a look and see if it fits with your use case. http://groups.google.com/group/google-appengine-java/browse_thread/thread/4812840ad76c73b On Oct 5, 6:53 pm, dnkoutso dnkou...@gmail.com wrote: I have an Android client that deals with product items and I would like to create an interface for displaying the most popular programs at any given time. I have read and used shard counters to achieve highly scalable and parallel counting. This has been working well as far as counting is concerned. However, the problem starts when it comes the time to calculate the top 10 most popular product items for a single request, I have to fetch them all product entities first, fetch the shard counters of each and add them up and then finally sort them to get the most popular ones. The problem here is that in order to find out whats the most popular I have to recalculate all shard counters. Multiply that by 1 product items and my request for a single user becomes slow as hell. I need to very quickly calculate the top 10 product items in a LIVE manner. Is this even conceivably possible? I've thought the idea of using a cron job to calculate the result and store that instead. Would you recommend me going that way? Has anyone else dealt with a similar situation? Thanks! -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
[appengine-java] Using the AsyncDatastoreService what happens to the DatastoreUnavailableException?
Do we have to use the Cababilities API instead of getting this exception? Do we get a null result from the future.get(); in this case? Is that the same as the entity not being present? -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
[appengine-java] ***MODERATOR*** Please delete my previous message, I posted it to the wrong group.
***MODERATOR*** Please delete my previous message, I posted it to the wrong group. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
[appengine-java] Re: Server Code vs DataStore
Yes, although you're still limited by the datastore - you can't save images bigger than 1MB due to the Blob size limit (I'm not sure if it would work to spread the data over more than one Blob, but I think I'd avoid it. I might try to avoid storing image data in the datastore at all to be honest - the Blobstore is there for that purpose. Mat. On Oct 5, 5:38 am, Kb kb.kb.1...@gmail.com wrote: Thanks . Can i display images with Objectify Framework ? i.e get output data (including image) with .get(key) method etc and then Display to client based on his input . -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
[appengine-java] Re: GAE - Vote counting system
Jeff - I'm a bit confused. I thought that the whole idea of the datastore was that you could read or write as much as you want, as fast as you want as long as they are not related? So one datastore write per vote (and being written to different entity groups) should be fine? I thought that the system just split tablets if they were being accessed too much - so as long as the traffic didn't suddenly increase, there'd be no scalability issues apart from cost. Have I misunderstood? Thanks, Mat. On Sep 30, 9:15 pm, Jeff Schnitzer j...@infohazard.org wrote: Assuming the goal is massive write throughput, you don't want to do 1 write per vote. You need to aggregate writes into a batch - you can do that with pull queues, but then you're limited to the maximum throughput of a pull queue. And the biggest batch size is 1,000 which might actually be votes for 1,000 different things, which means you're back to 1-vote-1-write. Peter, you can certainly build a system whereby all vote totals are tracked in RAM in a backend but now you're putting higher memory requirements on the backend and life gets more complicated when you deal with sharding. Depending on how exact you need the counts to be, you can always use increment() on memcache in addition to incrementing the backend. The only catch is that bootstrapping the initial memcache value will be a little tricky - you'll need to use CAS and query the backends for any existing deltas. Or just not care if it's off by a few. Jeff On Thu, Sep 29, 2011 at 5:56 AM, Mat Jaggard matjagg...@gmail.com wrote: Will that end up being cheaper than the following? Put a new entity in the datastore for each vote - Kind: Vote, ID: Auto generated, Vote for: Item S Have a task queue query all the votes and delete them then write the count of votes to a global object. Cost = 1 datastore read + 1 datastore write + some fairly minor task processing per vote. On Sep 29, 1:47 pm, Peter Dev dev133...@gmail.com wrote: Price: - with backends lets say 3 B2 machines = 350USD/Month - UrlFetch Data Sent/Received 0,15USD/GB Limit: - URL Fetch Daily Limit 46,000,000 calls this can be a problem...but I see it is possible to request an increase Write data parallel in DB: Task Queue with rate every 30second could be a solution (check timestamps in cache and write in DB) RESET counters = empty cache in Backends reset counter of object in DB Backends cache = HashMap with shared counter values or counter values without sharding (just increment value in java hashmap is fast enough) With backends we don’t need sharding I thinkwhat do you think? Thx. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group athttp://groups.google.com/group/google-appengine-java?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
[appengine-java] Re: GAE - Vote counting system
Will that end up being cheaper than the following? Put a new entity in the datastore for each vote - Kind: Vote, ID: Auto generated, Vote for: Item S Have a task queue query all the votes and delete them then write the count of votes to a global object. Cost = 1 datastore read + 1 datastore write + some fairly minor task processing per vote. On Sep 29, 1:47 pm, Peter Dev dev133...@gmail.com wrote: Price: - with backends lets say 3 B2 machines = 350USD/Month - UrlFetch Data Sent/Received 0,15USD/GB Limit: - URL Fetch Daily Limit 46,000,000 calls this can be a problem...but I see it is possible to request an increase Write data parallel in DB: Task Queue with rate every 30second could be a solution (check timestamps in cache and write in DB) RESET counters = empty cache in Backends reset counter of object in DB Backends cache = HashMap with shared counter values or counter values without sharding (just increment value in java hashmap is fast enough) With backends we don’t need sharding I thinkwhat do you think? Thx. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
[appengine-java] Re: Disable 'refresh' in a page
Someone tried to do this in my old company. The short answer is No, the long answer is This is a really bad idea. We set up an event which fired when the page was unloaded, this had the effect that when someone pressed F5 or refresh, they got an error message saying Please be patient, however it did still refresh, and all that happened was people disabled Javascript in their browser when visiting our site so that they didn't get the box when they refreshed. I wrote a greasemonkey script change the page to avoid having to disable javascript. So, as I say - no, don't do it. Mat. On Sep 25, 7:00 am, Shoubhik sbos...@gmail.com wrote: I need to DISABLE 'refresh' in a page (all possible ways, Browser buttons, right click - refresh and F5 ) Please provide me some solutions. Thanks in advance, Shoubhik Bose -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
