[google-appengine] Least Expensive Way to Delete Entities
Hi, I have a bit of old data that I want to clean out. I tried to write a fanout operation to clean out some of the old data, but it's actually manged to run out my budget in a few hours. I'm curious if there would be a way to delete huge amounts of old data that isn't needed anymore. Is there a tool or something that i can use to efficiently purge entity groups from my application? Any help would be greatly appreciated. Thanks, Patrick. -- Patrick H. Twohig. Namazu Studios P.O. Box 34161 San Diego, CA 92163-4161 -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [appengine-java] Re: cost effective way to store PDF and word docs in datastore / blobstore service
Just because billing is enabled that doesn't mean that you are billed for using it. It just uncaps your quotas beyond the free amount. Merry Chrismas! Sent from my G1 phone. On Dec 24, 2010 8:21 AM, dreamer venugopal.vasire...@gmail.com wrote: This is what I found in documentation Note: The Blobstore API is only available for apps with billing enabled. You still get an amount of quota for free, but billing must be enabled to use it. In development phase I don't want to be worried about billing. we can live with 1GB restriction while in development across for storage requirements. Any body from Google help to relax this restriction ? On Dec 24, 7:53 am, dreamer venugopal.vasire...@gmail.com wrote: Hi, I need to store pdf ... -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-j...@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
[appengine-java] maven-gae-plugin
Hi, I was having difficulty debugging my application using Maven plugin for App Engine. As soon as I connect the debugger the process stops. Is anybody else experiencing this issue? Or, alternatively, is there any other means to get maven+gae+eclipse and the debugger working? Thanks, Patrick. -- Patrick H. Twohig. Namazu Studios P.O. Box 34161 San Diego, CA 92163-4161 -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-j...@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [google-appengine] Re: BIlling Settings?
Works now. Thanks Ikai. On Wed, Dec 22, 2010 at 3:05 PM, Ikai Lan (Google) ikai.l+gro...@google.comikai.l%2bgro...@google.com wrote: I talked to the team about this, and this was the result of some code we pushed. The change has been rolled back. We'll fix it before we push it again. Try updating your settings now and let me know if it's still broken. -- Ikai Lan Developer Programs Engineer, Google App Engine Blogger: http://googleappengine.blogspot.com Reddit: http://www.reddit.com/r/appengine Twitter: http://twitter.com/app_engine On Wed, Dec 22, 2010 at 1:54 PM, Ikai Lan (Google) ikai.l+gro...@google.com ikai.l%2bgro...@google.com wrote: Can you guys send me your application IDs as well as reproduction steps to get the error 500? -- Ikai Lan Developer Programs Engineer, Google App Engine Blogger: http://googleappengine.blogspot.com Reddit: http://www.reddit.com/r/appengine Twitter: http://twitter.com/app_engine On Wed, Dec 22, 2010 at 10:50 AM, saidimu apale said...@gmail.comwrote: I've also had this same issue for the last 2-3 days. On Wed, Dec 22, 2010 at 4:07 AM, sarp sarpcen...@gmail.com wrote: We also have this problem. We're exceeding our quotas and can't change our billing settings. It's very frustrating! On Dec 22, 7:01 am, Patrick Twohig patr...@namazustudios.com wrote: Hi, I'm consistently getting an error 500 when trying to adjust my app's billing settings. Anybody else having this problem? Thanks, Patrick. -- Patrick H. Twohig. Namazu Studios P.O. Box 34161 San Diego, CA 92163-4161 -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.comgoogle-appengine%2bunsubscr...@googlegroups.com . For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.comgoogle-appengine%2bunsubscr...@googlegroups.com . For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.comgoogle-appengine%2bunsubscr...@googlegroups.com . For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] BIlling Settings?
Hi, I'm consistently getting an error 500 when trying to adjust my app's billing settings. Anybody else having this problem? Thanks, Patrick. -- Patrick H. Twohig. Namazu Studios P.O. Box 34161 San Diego, CA 92163-4161 -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] GAE/J And Task Queue Exceptions
Hi, I'm using the task queue to write a task that will iterate over all entities of a certain kind in the datastore. The task needs to perform an operation on each entity once, so I basically do a two-pass approach. ON the first pass, I actually iterate the entities enqueueing a separate task for each entity. Following that, the actual work gets executed for each entity. I'm using named tasks to ensure that each entity is processed in its own sub-task once. My question is, what happens when I try to enqueue two separate tasks with the same name to the same queue? Do I end up with an exception or does it fail silently? And before anybody says anything, yes, I do know that there's a chance that with named tasks it's possible that it can actually get enqueued twice. The documentation leads me to believe that this is extremely rare and for this particular use case a hard guarantee is not necessary. Cheers, Patrick. -- Patrick H. Twohig. Namazu Studios P.O. Box 34161 San Diego, CA 92163-4161 -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: GAE/J And Task Queue Exceptions
Whoops. I guess I'm a dumb ass and I forgot to read the docs fully: http://code.google.com/appengine/docs/java/javadoc/com/google/appengine/api/taskqueue/TaskAlreadyExistsException.html ...oh it's been a long night On Mon, Dec 20, 2010 at 11:13 PM, Patrick Twohig patr...@namazustudios.comwrote: Hi, I'm using the task queue to write a task that will iterate over all entities of a certain kind in the datastore. The task needs to perform an operation on each entity once, so I basically do a two-pass approach. ON the first pass, I actually iterate the entities enqueueing a separate task for each entity. Following that, the actual work gets executed for each entity. I'm using named tasks to ensure that each entity is processed in its own sub-task once. My question is, what happens when I try to enqueue two separate tasks with the same name to the same queue? Do I end up with an exception or does it fail silently? And before anybody says anything, yes, I do know that there's a chance that with named tasks it's possible that it can actually get enqueued twice. The documentation leads me to believe that this is extremely rare and for this particular use case a hard guarantee is not necessary. Cheers, Patrick. -- Patrick H. Twohig. Namazu Studios P.O. Box 34161 San Diego, CA 92163-4161 -- Patrick H. Twohig. Namazu Studios P.O. Box 34161 San Diego, CA 92163-4161 -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] GAE/J and Maven
Hi, I'm having some trouble getting a GAE/J build to work with Maven. I've added the appropriate Project Nature in my pom.xml and when I open it up in eclipse, I am getting the following error. I was curious if anybody else had the same error and would have any insight as to how to solve it. The App Engine SDK '/~/.m2/repository/com/google/appengine' on the project's build path is not valid - Google App Engine Problem Thanks, Patrick. -- Patrick H. Twohig. Namazu Studios P.O. Box 34161 San Diego, CA 92163-4161 -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Java POJO Mapper
A few months ago I went ahead and wrote a POJO mapper for AppEngine and I just now got around to putting it up on Google Code. If anybody wants to try it out here it is. I hope somebody can find it useful. http://code.google.com/p/appengine-catfish/ It's similar to Objectify and Twig. Let me know what y'all think. -- Patrick H. Twohig. Namazu Studios P.O. Box 34161 San Diego, CA 92163-4161 -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Login page for mobile?
*** I don't know if this violates Google's terms of service, so use this method at your own risk. *** You can POST some parameters to, I don't know what they are off hand. https://www.google.com/accounts/ServiceLoginAuth And then store the cookies. If you're on iPhone you can pass a cookie storage to NSURLConnection which will automatically persist the cookies for subsequent requests. Entering the wrong password too many times will lock out the user's account and you'll have to direct the user to some page with a CAPTCHA to unlock it. I'm pretty sure the bulkloader.py script uses this method to authenticate users. This post may help. Just translate the CURL/PHP to the equivalent mobile code. http://curl.haxx.se/mail/curlphp-2006-01/0014.html Cheers. On Mon, Jun 21, 2010 at 12:05 PM, Harshal p.hars...@gmail.com wrote: Hi, The app relies on google accounts ( http://code.google.com/appengine/docs/python/users/) using app engine's API. When visited from mobile it gives the normal google login page. Is there any way to get a mobile friendly google login page (similar to whats used by gmail or something?) -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.comgoogle-appengine%2bunsubscr...@googlegroups.com . For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Patrick H. Twohig. Namazu Studios P.O. Box 34161 San Diego, CA 92163-4161 -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Deadline?
I was under the impression that the deadline for requests was 30 seconds, but my app is routinely throwing deadline exceptions after 10 seconds or so. Is there a reason for this? -- Patrick H. Twohig. Namazu Studios P.O. Box 34161 San Diego, CA 92163-4161 -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: Deadline?
Ah, that would make sense. On Mon, Jun 7, 2010 at 1:17 PM, Mike Wesner mike.wes...@webfilings.comwrote: The ones that fail right at 10 seconds are usually due to the simultaneous dynamic request limit. Do the logs show this? Request was aborted after waiting too long to attempt to service your request because you have reached your simultaneous dynamic request limit. This is almost always due to excessively high latency in your app. Please see http://code.google.com/appengine/kb/general.html#activerequests for more details. -mike On Jun 7, 2:20 pm, Patrick Twohig patr...@namazustudios.com wrote: I was under the impression that the deadline for requests was 30 seconds, but my app is routinely throwing deadline exceptions after 10 seconds or so. Is there a reason for this? -- Patrick H. Twohig. Namazu Studios P.O. Box 34161 San Diego, CA 92163-4161 -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.comgoogle-appengine%2bunsubscr...@googlegroups.com . For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Patrick H. Twohig. Namazu Studios P.O. Box 34161 San Diego, CA 92163-4161 Office: 619.862.2890 x100 Cell: 619.453.5075 Twitter: @svm_invictvs IRC: svm_invic...@irc.freenode.net ##java, #android-dev, #iphonedev, #appengine http://www.namazustudios.com/ This communication, and any attachments, shall be considered confidential and proprietary information of Namazu Studios LLC. This message, and attachments, are intended for the listed recipients only. If you are not one of the intended recipients, please destroy all copies of this communication. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: Email Service with 10K+ addresses in the cc list
If you send a bulk email like that, does it count as one email towards your quota or does it count as each recipient. I wrote a task that sends a single email to my entire subscriber base. WOuld I be saving money by using the BCC field? On Mon, Jun 7, 2010 at 8:34 AM, Sandeep Arneja sandee...@gmail.com wrote: Has anyone experimented with shorter email bcc groups? Whats the maximum number of people I can have in a group without having the task fail? On Jun 3, 3:00 am, Aurelian aurelian.vacar...@googlemail.com wrote: I was trying to send 1200 emails using a task queue, but it doesn't work, giving a error. I think you should split the list of receivers every 5000, and send 2 o 3 emails. Happy coding. On Jun 3, 7:45 am, Sandeep Arneja sandee...@gmail.com wrote: I am trying to send an email with about 10K+ addresses in the cc list. This is not for spam but in case i have maintainence on my application i send an email to all users informing them about the status. How should i design this? I see that google app engine in dev stops responding when i use Transport.send(msg); with 10K+ emails in cc even when the method is supposed to return asynchronously. Should i divide the email into smaller tasks and put on the queue or would the engine not hang in PROD? -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.comgoogle-appengine%2bunsubscr...@googlegroups.com . For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Patrick H. Twohig. Namazu Studios P.O. Box 34161 San Diego, CA 92163-4161 -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Effectively Parallelizing Fetches (with pictures, yay!)
Ah, thanks Nick! I actually started to implement some of those changes, but ended up getting sidetracked with other things, but I'm starting again on it. Will probably have more questions later :) On Tue, Apr 20, 2010 at 4:23 AM, Nick Johnson (Google) nick.john...@google.com wrote: Hi Patrick, Good questions! On Tue, Apr 20, 2010 at 12:57 AM, Patrick Twohig patr...@namazustudios.com wrote: Hi All, As I understand it, the process of performing a single fetch (call to get()) from the dastastore using a key basically involves finding the host housing the entity, opening a socket, fetching the data, and then cleaning up the connection. So to fetch something like 30 entities from the datastore, you're repeating the process 30 times over in serial, each time incurring whatever overhead is involved. I also read that if you perform bulk fetches, (ie passing multiple keys at once) you can eliminate a great deal of that overhead. In one of the videos I watched from Google I/0 2009, the presenter (whose name I forget - d'oh) said that performing a bulk fetch actually performs the fetches in parallel from the data store and you shoudl see requests noticeably faster. Currently I have a few situations where the app performs many fetches from the data store in serially, rather than in bulk, and I believe it is the result of these requests being extremely slow and CPU intensive. Where possible, I put into place as much bulk fetches as I can but I'm a little stuck in a few places. I'm basing the fetch latency on today's numbers -- http://code.google.com/status/appengine/detail/datastore/2010/04/19. Anomalies aside, It looks like the get latency somewhere between 80ms and 160ms, let's spit difference and just say that it's 120 milliseconds. Additionally, the query latency is somewhere between 250ms and 500ms. Splitting the difference, that's 375ms. I'm just going to use those numbers as a ballpark estimate for fetching multiple entities from the data store, feel free to correct me if any of my reasoning is flawed or incorrect. The figures shown by the status site seem to be on the high side at the moment - they represent worst cases. In my own apps, gets are observed to be more on the order of 10-20ms, while queries vary widely depending on returned data, but average about 100-300ms. Example 1: http://imagepaste.nullnetwork.net/viewimage.php?id=830 Given the above example, I'm assuming that if I performed an ancestor query with Foo(A) as the ancestor it would effectively bulk-fetch the entire entity group. I could then use the result of that query to get the data I need. That would make the fetch from the datastore one query, 375 milliseconds versus (7entities * 160ms) or 1120ms. So long as you need 3 or more entities (3 * 160) it would stand to reason that you're just better off just fetching the whole thing. In some simple tests I did, that seemed to be the case, the query approach was faster, and that's great if you know everything is in the same entity group. Example 2: http://imagepaste.nullnetwork.net/viewimage.php?id=831 Given the above example, none of the entities are in the same entity group, but I would want to try to perform bulk fetches wherever possible. I would first fetch Foo(A). I would then see that it has two key properties pointing to Bar(B) and Bar(C), perform a fetch of those two entities at once. Finally, I would see that Bar(B) and Bar(C) each reference two more entities -- Baz(D), Baz(E), Baz(F), and Baz(G) for a total of four. In the worst case, I would fetch each entity individually taking, once again, 1120ms. In the best case and I perform 3 fetches, (fetch A first, then fetch B and C, then lastly fetch D, E, F, and G), it would be more in the neighborhood of 480 milliseconds. It's still an improvement over fetching each entity individually, but not much. Very similar to this is the 'referenceproperty prefetching' pattern - see http://blog.notdot.net/2010/01/ReferenceProperty-prefetching-in-App-Engine http://blog.notdot.net/2010/01/ReferenceProperty-prefetching-in-App-Engine So I was thinking of ways to improve this, the second example in particular, because I have a few places in my app where that exact thing is happening. Right now it's actually implemented with individual fetches, but it backed by memcache in many circumstances so that definitely helps. So given that, here's my questions... - When serializing the objects, would it be worthwhile adding some sort of metadata in the entity that would tell me what other entities it references (either directly or indirectly) so that I could fetch the whole thing with one or two API calls? I was thinking that an entity could have child entities with all the keys it references directly or indirectly. This would be a huge pain to implement, and I'm not sure it would make a noticeable performance boost
[google-appengine] Deleting entities in bulk.
Recently, I have refactored my code such that I no longer need entities of a particular kind. I was curious how I would go about writing a task to delete these entities in bulk. I have set up a task that performs a query specifying only the kind, and then using a cursor to delete the entities on the task queue. However, when I run the task it seems to complete prematurely rather than delete all entities. Here's the process I use: - I perform a query for all entities of a particular kind. For instance SELECT * FROM SomeKind or new Query(SomeKind) in Java. - I delete as many entities as I can using the results of the query until I hit a soft deadline. I set that up myself to be 10 seconds, or so. - If there are more results left, I generate a cursor and pick up again and re-queue the task to execute again. - If there are no results left, I just leave finish the task logging the result. The problem is, the task seems to finish prematurely and it appears to either delete only a few hundred entities, or it simply skips a few here and there. Running it in the debugger, it all seems to work well, but on the AppEngine it doesn't seem to work quite as well. Any thoughts or suggestions? Thanks, Patrick. -- Patrick H. Twohig. Namazu Studios P.O. Box 34161 San Diego, CA 92163-4161 -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Deleting entities in bulk.
Thanks Jawaad!
On Mon, May 3, 2010 at 8:26 PM, Jawaad Mahmood jawaad.mahm...@gmail.comwrote:
I had to deal with the same issue myself.
I did this:
class DeleteFull():
def execute(self):
deleting = model_class_name.all().order('__key__').fetch(100)
while deleting:
a = []
key = deleting[-1].key()
for item in deleting:
a.append(item)
db.delete(a)
deleting = model_class_name.all().filter('__key__ ',
key).order('__key__').fetch(100)
This purged everything, but it took a hell of a long time.
On Tue, May 4, 2010 at 11:08 AM, Patrick Twohig
patr...@namazustudios.com wrote:
Recently, I have refactored my code such that I no longer need entities
of a
particular kind. I was curious how I would go about writing a task to
delete these entities in bulk. I have set up a task that performs a
query
specifying only the kind, and then using a cursor to delete the entities
on
the task queue. However, when I run the task it seems to complete
prematurely rather than delete all entities.
Here's the process I use:
I perform a query for all entities of a particular kind. For instance
SELECT * FROM SomeKind or new Query(SomeKind) in Java.
I delete as many entities as I can using the results of the query until I
hit a soft deadline. I set that up myself to be 10 seconds, or so.
If there are more results left, I generate a cursor and pick up again and
re-queue the task to execute again.
If there are no results left, I just leave finish the task logging the
result.
The problem is, the task seems to finish prematurely and it appears to
either delete only a few hundred entities, or it simply skips a few here
and
there. Running it in the debugger, it all seems to work well, but on the
AppEngine it doesn't seem to work quite as well. Any thoughts or
suggestions?
Thanks,
Patrick.
--
Patrick H. Twohig.
Namazu Studios
P.O. Box 34161
San Diego, CA 92163-4161
--
You received this message because you are subscribed to the Google Groups
Google App Engine group.
To post to this group, send email to google-appeng...@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.comgoogle-appengine%2bunsubscr...@googlegroups.com
.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.
--
Sincerely yours,
Jawaad Mahmood
http://www.jawaadmahmood.com
080-4204-7198
--
You received this message because you are subscribed to the Google Groups
Google App Engine group.
To post to this group, send email to google-appeng...@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.comgoogle-appengine%2bunsubscr...@googlegroups.com
.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.
--
Patrick H. Twohig.
Namazu Studios
P.O. Box 34161
San Diego, CA 92163-4161
--
You received this message because you are subscribed to the Google Groups
Google App Engine group.
To post to this group, send email to google-appeng...@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Database Security
I just read this article, and I was curious what/if there was anything that GAE does (or is capable of doing) to comply with this sort of law.. http://www.darkreading.com/database_security/security/app-security/showArticle.jhtml?articleID=223200019cid=RSSfeed_DR_News -- Patrick H. Twohig. Namazu Studios P.O. Box 34161 San Diego, CA 92163-4161 -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: Counting query results 1000
For whatever it's worth, just getting all the keys with a fetch limit of the largest possible value for an integer seems to work. I have approximately 10,000 entities and counting the keys seems to work and not hit a deadline exception. Cheers! On Sat, Apr 10, 2010 at 7:31 PM, 风笑雪 kea...@gmail.com wrote: def count(model, limit): result = model.count(limit) if limit 1000 and result == 1000: result = len(model.all(keys_only=True).fetch(limit)) return result -- keakon 2010/4/10 Patrick Twohig patr...@namazustudios.com: I can't use a cursor, it has to happen in one request. I tried using a sharded counter to provide an estimation of ranking, but that has proven unsuccessful. On Thu, Apr 8, 2010 at 11:55 PM, Tim Hoffman zutes...@gmail.com wrote: Hi You will need to use a cursor combined with a keys_only query (for efficiency) and keep resubmitting the cursor until you get all the results. If you plan to do this in appengine process rather than via the remote_api you will probably need to use a chain of tasks to achieve the result if you have a lot of entities. In python it looks something like the following (I don't do any work with java on app engine) def count(query): i = 0 while True: result = query.fetch(1000) i = i + len(result) if len(result) 1000: break cursor = query.cursor() query.with_cursor(cursor) return i myquery = mymodels.MyModel.all(keys_only=True) x = count(myquery) Hope this helps Rgds T On Apr 9, 7:36 am, Patrick Twohig patr...@namazustudios.com wrote: I'm trying to figure out a way to count entities matching a certain criteria. I was thinking of doing a key-only query and using PreparedQuery.countEntities() (Java). However, when I do that, it always returns a value no higher than 1000. Is there a way to get an accurate count of all entities matching a query without having to do something like int count = datastore.prepare( query.setKeysOnly() ).asList(FetchOptions.Builder.limit(Integer.MAX_VALUE)).size() ? Thanks, Patrick. -- Patrick H. Twohig. Namazu Studios P.O. Box 34161 San Diego, CA 92163-4161 -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.comgoogle-appengine%2bunsubscr...@googlegroups.com . For more options, visit this group athttp://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.comgoogle-appengine%2bunsubscr...@googlegroups.com . For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Patrick H. Twohig. Namazu Studios P.O. Box 34161 San Diego, CA 92163-4161 -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.comgoogle-appengine%2bunsubscr...@googlegroups.com . For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.comgoogle-appengine%2bunsubscr...@googlegroups.com . For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Patrick H. Twohig. Namazu Studios P.O. Box 34161 San Diego, CA 92163-4161 -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Effectively Parallelizing Fetches (with pictures, yay!)
Hi All, As I understand it, the process of performing a single fetch (call to get()) from the dastastore using a key basically involves finding the host housing the entity, opening a socket, fetching the data, and then cleaning up the connection. So to fetch something like 30 entities from the datastore, you're repeating the process 30 times over in serial, each time incurring whatever overhead is involved. I also read that if you perform bulk fetches, (ie passing multiple keys at once) you can eliminate a great deal of that overhead. In one of the videos I watched from Google I/0 2009, the presenter (whose name I forget - d'oh) said that performing a bulk fetch actually performs the fetches in parallel from the data store and you shoudl see requests noticeably faster. Currently I have a few situations where the app performs many fetches from the data store in serially, rather than in bulk, and I believe it is the result of these requests being extremely slow and CPU intensive. Where possible, I put into place as much bulk fetches as I can but I'm a little stuck in a few places. I'm basing the fetch latency on today's numbers -- http://code.google.com/status/appengine/detail/datastore/2010/04/19. Anomalies aside, It looks like the get latency somewhere between 80ms and 160ms, let's spit difference and just say that it's 120 milliseconds. Additionally, the query latency is somewhere between 250ms and 500ms. Splitting the difference, that's 375ms. I'm just going to use those numbers as a ballpark estimate for fetching multiple entities from the data store, feel free to correct me if any of my reasoning is flawed or incorrect. Example 1: http://imagepaste.nullnetwork.net/viewimage.php?id=830 Given the above example, I'm assuming that if I performed an ancestor query with Foo(A) as the ancestor it would effectively bulk-fetch the entire entity group. I could then use the result of that query to get the data I need. That would make the fetch from the datastore one query, 375 milliseconds versus (7entities * 160ms) or 1120ms. So long as you need 3 or more entities (3 * 160) it would stand to reason that you're just better off just fetching the whole thing. In some simple tests I did, that seemed to be the case, the query approach was faster, and that's great if you know everything is in the same entity group. Example 2: http://imagepaste.nullnetwork.net/viewimage.php?id=831 Given the above example, none of the entities are in the same entity group, but I would want to try to perform bulk fetches wherever possible. I would first fetch Foo(A). I would then see that it has two key properties pointing to Bar(B) and Bar(C), perform a fetch of those two entities at once. Finally, I would see that Bar(B) and Bar(C) each reference two more entities -- Baz(D), Baz(E), Baz(F), and Baz(G) for a total of four. In the worst case, I would fetch each entity individually taking, once again, 1120ms. In the best case and I perform 3 fetches, (fetch A first, then fetch B and C, then lastly fetch D, E, F, and G), it would be more in the neighborhood of 480 milliseconds. It's still an improvement over fetching each entity individually, but not much. So I was thinking of ways to improve this, the second example in particular, because I have a few places in my app where that exact thing is happening. Right now it's actually implemented with individual fetches, but it backed by memcache in many circumstances so that definitely helps. So given that, here's my questions... - When serializing the objects, would it be worthwhile adding some sort of metadata in the entity that would tell me what other entities it references (either directly or indirectly) so that I could fetch the whole thing with one or two API calls? I was thinking that an entity could have child entities with all the keys it references directly or indirectly. This would be a huge pain to implement, and I'm not sure it would make a noticeable performance boost. - Is there something under the covers of the API that actually makes more efficient usage of resources that I don't know about? - Is there something in the API that I don't know about that could make the second example faster w/o much effort? - Is my design just bad and I should figure out a better way of doing it? If so, how would I go about doing that? Alright, that's all for now. Thanks, Patrick. -- Patrick H. Twohig. Namazu Studios P.O. Box 34161 San Diego, CA 92163-4161 -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] iPhone / AppEngine / Java Servlet / Authentication
I use HTTP Basic auth over SSL, works quite well for me. On Tue, Apr 13, 2010 at 1:37 PM, hiawatha bhh...@gmail.com wrote: Hello, I'm looking for help with, example code, or a tutorial on doing session management and user authentication (not Google user) from an iPhone App using App Engine as the back end. Any and all relevant pointers appreciated. Thanks much. - H -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.comgoogle-appengine%2bunsubscr...@googlegroups.com . For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Patrick H. Twohig. Namazu Studios P.O. Box 34161 San Diego, CA 92163-4161 http://www.namazustudios.com/ http://www.iduel-online.com/ -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: Counting query results 1000
I can't use a cursor, it has to happen in one request. I tried using a sharded counter to provide an estimation of ranking, but that has proven unsuccessful. On Thu, Apr 8, 2010 at 11:55 PM, Tim Hoffman zutes...@gmail.com wrote: Hi You will need to use a cursor combined with a keys_only query (for efficiency) and keep resubmitting the cursor until you get all the results. If you plan to do this in appengine process rather than via the remote_api you will probably need to use a chain of tasks to achieve the result if you have a lot of entities. In python it looks something like the following (I don't do any work with java on app engine) def count(query): i = 0 while True: result = query.fetch(1000) i = i + len(result) if len(result) 1000: break cursor = query.cursor() query.with_cursor(cursor) return i myquery = mymodels.MyModel.all(keys_only=True) x = count(myquery) Hope this helps Rgds T On Apr 9, 7:36 am, Patrick Twohig patr...@namazustudios.com wrote: I'm trying to figure out a way to count entities matching a certain criteria. I was thinking of doing a key-only query and using PreparedQuery.countEntities() (Java). However, when I do that, it always returns a value no higher than 1000. Is there a way to get an accurate count of all entities matching a query without having to do something like int count = datastore.prepare( query.setKeysOnly() ).asList(FetchOptions.Builder.limit(Integer.MAX_VALUE)).size() ? Thanks, Patrick. -- Patrick H. Twohig. Namazu Studios P.O. Box 34161 San Diego, CA 92163-4161 -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.comgoogle-appengine%2bunsubscr...@googlegroups.com . For more options, visit this group athttp:// groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.comgoogle-appengine%2bunsubscr...@googlegroups.com . For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Patrick H. Twohig. Namazu Studios P.O. Box 34161 San Diego, CA 92163-4161 -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Counting query results 1000
I'm trying to figure out a way to count entities matching a certain criteria. I was thinking of doing a key-only query and using PreparedQuery.countEntities() (Java). However, when I do that, it always returns a value no higher than 1000. Is there a way to get an accurate count of all entities matching a query without having to do something like int count = datastore.prepare( query.setKeysOnly() ).asList(FetchOptions.Builder.limit(Integer.MAX_VALUE)).size() ? Thanks, Patrick. -- Patrick H. Twohig. Namazu Studios P.O. Box 34161 San Diego, CA 92163-4161 -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: Huzzah! My App's Finally Live.
Ikai, THe Android version is in the works, but the last Android app we put out bombed miserably so we went to iPhone. Though since it's Java, I can probably copy a bunch of the stuff over to the Android client and turn it around much faster than the iPhone version. You also may be interested to note is that we advertised the application at PAXE in Boston this past weekend. People definitely responded to the ad and our usage ramped up quite high and AppEngine kept up with few issues. On that note, how do I feature it in the App gallery? Derek: It's all Polling over HTTP. The original design used push notifications combined with polling, but that ended up not happening and we just went with polling. It's not ideal, and I'd like to take the comet approach at some point to make it a bit more responsive. Pat. On Tue, Mar 30, 2010 at 12:23 PM, A1programmer derrick.simp...@gmail.comwrote: Wow, it looks amazing! I'm curious as to how you implemented the communication, did you use polling over HTTP, or XMPP ? - Derrick On Mar 30, 2:47 pm, Patrick Twohig patr...@namazustudios.com wrote: Last Thursday my app went live and I thought I'd share it with everybody on here. Despite the frustrations with GAE/J, we're reasonably pleased with it. If anybody wants to check it out, download iDuel from the AppStore. http://itunes.apple.com/us/app/iduel-online/id360729665?mt=8 -- Patrick H. Twohig. Namazu Studios P.O. Box 34161 San Diego, CA 92163-4161 -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.comgoogle-appengine%2bunsubscr...@googlegroups.com . For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Patrick H. Twohig. Namazu Studios P.O. Box 34161 San Diego, CA 92163-4161 -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Stuck Index?
I have an index that appears to be stuck, how do I unstick it? -- Patrick H. Twohig. Namazu Studios P.O. Box 34161 San Diego, CA 92163-4161 -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Huzzah! My App's Finally Live.
Last Thursday my app went live and I thought I'd share it with everybody on here. Despite the frustrations with GAE/J, we're reasonably pleased with it. If anybody wants to check it out, download iDuel from the AppStore. http://itunes.apple.com/us/app/iduel-online/id360729665?mt=8 -- Patrick H. Twohig. Namazu Studios P.O. Box 34161 San Diego, CA 92163-4161 -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: data center location
Just FYI, I think the GAE datacenter is in SF, not in Mountain View. I'm not 100% sure if that's positive. On Mon, Mar 29, 2010 at 12:31 PM, nickmilon nickmi...@gmail.com wrote: As Googlers have posted some time ago here NO there is not such option now, but G has the infrastracture and know how to do it, may be some time in the future. Nick On Mar 29, 7:09 pm, Ilya Biryukov mobiled...@gmail.com wrote: Hi guys. Can I specify where to deploy my app geographically? Say I want my data to stay in Europe only. Is this possible? If so, what are the available locations? Thanks -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.comgoogle-appengine%2bunsubscr...@googlegroups.com . For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Patrick H. Twohig. Namazu Studios P.O. Box 34161 San Diego, CA 92163-4161 -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: Reading a file in GAE Java
Are they in war or on the classpath? On Tue, Mar 23, 2010 at 4:49 AM, Iain Flynn trancedsha...@gmail.com wrote: Thanks for that, but it's still getting null. I've tried changing the URI and the file's location around (and triple-checking that I've spelled it correctly), but still nothing. I've included these lines in the appengine-web.xml file, as the tutorial says. Can you please say if you see anything wrong with them? static-files exclude path=/**.xsl / /static-files resource-files include path=/**.xsl / /resource-files Thank you again, - Iain. On Mar 23, 2:06 am, Patrick Twohig patr...@namazustudios.com wrote: I use Class.getResourceAsStream(/xml/myschema.xsd); Cheers! On Mon, Mar 22, 2010 at 10:57 AM, Iain Flynn trancedsha...@gmail.com wrote: Hi there. I have an application that requires me to read in and transform an XSLT stylesheet, which I keep in the directory /xml in / war. I've read the tutorials, and I've included it as a resource file (more specifically, set that all xsl files are included as resource files), but when I try to read it in my Java servlet (using the URI /xml/ file.xsl), I get the FilePermission error. I presume I'm getting this error because it can't find the file (I know that writing files is not allowed, but I'm opening it in read-mode). I've looked through all the tutorials and I can't see where I'm going wrong here. I can use files with my HTML, but getting Java on them seems to elude me. I get the feeling I've made a very elementary mistake, but can't for the life of me figure it out. Thanks for your time, and sorry if this is the wrong place for this. - Iain -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com . To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.comgoogle-appengine%2bunsubscr...@googlegroups.comgoogle-appengine%2Bunsubscrib e...@googlegroups.com . For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Patrick H. Twohig. Namazu Studios P.O. Box 34161 San Diego, CA 92163-4161 -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.comgoogle-appengine%2bunsubscr...@googlegroups.com . For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Patrick H. Twohig. Namazu Studios P.O. Box 34161 San Diego, CA 92163-4161 -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: Will GAE ever be open source?
On Mon, Mar 22, 2010 at 5:47 AM, Wooble geoffsp...@gmail.com wrote: On Mar 22, 12:36 am, Iap iap...@gmail.com wrote: My concern is that the open-source GAE would also expose the weakness of GAE. That makes the GAE more vulnerable under attacking. I hope you're not coding in a programming language that's open source. Just imagine the vulnerabilities. Of course, with python and java eliminated, how are you using App Engine at all? -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.comgoogle-appengine%2bunsubscr...@googlegroups.com . For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. 1995 called. They want their rhetoric back. -- Patrick H. Twohig. Namazu Studios P.O. Box 34161 San Diego, CA 92163-4161 -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: ~7 GB of ghost data???
Hey Nick, Just out of curiosity, how many properties would it take to get that amount of wasted space in overhead? Are we talking about entities in the orders of magnitudes of tens/thousands/hundreds? On Mon, Mar 22, 2010 at 9:07 AM, homunq jameson.qu...@gmail.com wrote: OK, I guess I'm guilty on all counts. Clearly, I can fix that moving forward, though it will cost me a lot of CPU to fix the data I've already entered. But as a short-term stopgap, is there any way to delete entire default indexes for a given property? (I mean, anything besides setting indexed=False and then touching each entity one-by-one). You can vacuum custom indexes - can you do it with indexes created by default? Thanks, Jameson On 22 mar, 03:42, Nick Johnson (Google) nick.john...@google.com wrote: Hi, The discrepancy between datastore stats volume and stored data is generally due to indexing overhead, which is not included in the datastore stats. This can be very high for entities with many properties, or with long entity and property names or entity keys. Do you have reason to suppose that's not the case in your situation? -Nick Johnson On Sun, Mar 21, 2010 at 3:39 AM, homunq jameson.qu...@gmail.com wrote: Something is wrong. My app is showing with 7.42GB of total stored data, but only 615 MB of datastore. There is only one version string uploaded, which is almost 150MB, and nothing in the blobstore. This discrepancy has been getting worse - several hours ago (longer than the period since datastore statistics were updated, if you're wondering), there were the same 615 MB in the datastore, and only 3.09GB of total stored data. (at that time, my theory was that it was old uploads of tweaks to the same version - but the numbers have gone far, far beyond that explanation now.) It's not some exploding index; the only non-default index I have is on an entity type with just 33 entities. Here's the line from my dashboard: Total Stored Data$0.005/GByte-day82% 7.42 of 9.00 GBytes $0.04 / $0.04 And here is the word from my datastore statistics: Last updatedTotal number of entitiesSize of all entities 1:32:13 ago 232,867 615 MBytes (metadata 11%, if that matters) Please, can someone help me figure out this issue? I'd be happy to share any info or code which would help track this down. My app id is vulahealth. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com . To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.comgoogle-appengine%2bunsubscr...@googlegroups.comgoogle-appengine%2Bunsubscrib e...@googlegroups.com . For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Nick Johnson, Developer Programs Engineer, App Engine Google Ireland Ltd. :: Registered in Dublin, Ireland, Registration Number: 368047 -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.comgoogle-appengine%2bunsubscr...@googlegroups.com . For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Patrick H. Twohig. Namazu Studios P.O. Box 34161 San Diego, CA 92163-4161 -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: ~7 GB of ghost data???
I'd use a cursor on the task queue. Do bulk deletes in blocks of 500 (I think that's the most keys you can pass to delete on a single call) and it shouldn't be that hard to wipe it out. Cheers! On Mon, Mar 22, 2010 at 1:45 PM, homunq jameson.qu...@gmail.com wrote: OK, after hashing it out on IRC, I see that I have to erase my data and start again. Since it took me 3 days of CPU quota to add the data, I want to know if I can erase it quickly. 1. Is the overhead for erasing data (and thus whittling down indexes) over half the overhead from adding it? Under 10%? Or what? (I don't need exact numbers, just approximates. 2. If it's more like half - is there some way to just nuke all my data and start over? Thanks, Jameson On 22 mar, 03:42, Nick Johnson (Google) nick.john...@google.com wrote: Hi, The discrepancy between datastore stats volume and stored data is generally due to indexing overhead, which is not included in the datastore stats. This can be very high for entities with many properties, or with long entity and property names or entity keys. Do you have reason to suppose that's not the case in your situation? -Nick Johnson On Sun, Mar 21, 2010 at 3:39 AM, homunq jameson.qu...@gmail.com wrote: Something is wrong. My app is showing with 7.42GB of total stored data, but only 615 MB of datastore. There is only one version string uploaded, which is almost 150MB, and nothing in the blobstore. This discrepancy has been getting worse - several hours ago (longer than the period since datastore statistics were updated, if you're wondering), there were the same 615 MB in the datastore, and only 3.09GB of total stored data. (at that time, my theory was that it was old uploads of tweaks to the same version - but the numbers have gone far, far beyond that explanation now.) It's not some exploding index; the only non-default index I have is on an entity type with just 33 entities. Here's the line from my dashboard: Total Stored Data$0.005/GByte-day82% 7.42 of 9.00 GBytes $0.04 / $0.04 And here is the word from my datastore statistics: Last updatedTotal number of entitiesSize of all entities 1:32:13 ago 232,867 615 MBytes (metadata 11%, if that matters) Please, can someone help me figure out this issue? I'd be happy to share any info or code which would help track this down. My app id is vulahealth. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com . To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.comgoogle-appengine%2bunsubscr...@googlegroups.comgoogle-appengine%2Bunsubscrib e...@googlegroups.com . For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Nick Johnson, Developer Programs Engineer, App Engine Google Ireland Ltd. :: Registered in Dublin, Ireland, Registration Number: 368047 -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.comgoogle-appengine%2bunsubscr...@googlegroups.com . For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Patrick H. Twohig. Namazu Studios P.O. Box 34161 San Diego, CA 92163-4161 -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Reading a file in GAE Java
I use Class.getResourceAsStream(/xml/myschema.xsd); Cheers! On Mon, Mar 22, 2010 at 10:57 AM, Iain Flynn trancedsha...@gmail.comwrote: Hi there. I have an application that requires me to read in and transform an XSLT stylesheet, which I keep in the directory /xml in / war. I've read the tutorials, and I've included it as a resource file (more specifically, set that all xsl files are included as resource files), but when I try to read it in my Java servlet (using the URI /xml/ file.xsl), I get the FilePermission error. I presume I'm getting this error because it can't find the file (I know that writing files is not allowed, but I'm opening it in read-mode). I've looked through all the tutorials and I can't see where I'm going wrong here. I can use files with my HTML, but getting Java on them seems to elude me. I get the feeling I've made a very elementary mistake, but can't for the life of me figure it out. Thanks for your time, and sorry if this is the wrong place for this. - Iain -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.comgoogle-appengine%2bunsubscr...@googlegroups.com . For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Patrick H. Twohig. Namazu Studios P.O. Box 34161 San Diego, CA 92163-4161 -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Session Data must be in DataStore?
For whatever its' worth, I have HTTP Basic auth implemented in my web service and it fetches the user's account details from the datastore each time a request is made and it doesn't seem to be a huge problem. On Wed, Mar 17, 2010 at 4:41 PM, Guy Smith g...@multiniche.org wrote: “If a web application wants to store session data, then it must be stored in the DataStore rather than static variables or MemCache.” - is that correct? I understand the reasoning behind it – that separate requests in a session may go to different servers, which will not have the same values in static variables or MemCache, only the DataStore is shared between servers. But it is expensive (in developer time and DataStore writes), so I’d just like to double check. NB: I have a fair bit of session data – encoding it in the cookie is not feasible. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.comgoogle-appengine%2bunsubscr...@googlegroups.com . For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Patrick H. Twohig. Namazu Studios P.O. Box 34161 San Diego, CA 92163-4161 -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] More bulkloader headache :(
I'm getting this when trying to do a nightly backup of my app from a cron job... No handlers could be found for logger google.appengine.tools.appengine_rpc Please go to https://www.google.com/accounts/DisplayUnlockCaptcha. Upon entering my account information with the correct password at CAPTCHA, it tells me the username/account isn't correct. Furthermore, I'm able to log the same account into gmail just fine. The Google account I'm using is a Google Apps account. Any suggestions? Thanks, -- Patrick H. Twohig. Namazu Studios P.O. Box 34161 San Diego, CA 92163-4161 -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] No api proxy found for service datastore_v3
I'm trying to bulkload data from my app, but it fails on this call. I browsed through the archives and found a post by Nick Johnson earlier and he suggested changing the fetch limit per request, but that didn't seem to see much. Any suggestions? I'm thinking that I just forgot to set some environment variable or something. Thanks, Pat. Full trace: patr...@gentoo ~/test/test_gae_bulkload $ bulkloader.py --batch_size=2 --dry_run --filename=backup.csv --dump --url= http://1.latest.myapp.appspot.com/remote_api /home/patrick/gae/google_appengine-1.3.1/google/appengine/ext/remote_api/remote_api_stub.py:64: DeprecationWarning: the sha module is deprecated; use the hashlib module instead import sha /home/patrick/gae/google_appengine-1.3.1/google/appengine/tools/dev_appserver_login.py:33: DeprecationWarning: the md5 module is deprecated; use hashlib instead import md5 [INFO] Logging to bulkloader-log-20100315.162913 [INFO] Throttling transfers: [INFO] Bandwidth: 25 bytes/second [INFO] HTTP connections: 8/second [INFO] Entities inserted/fetched/modified: 20/second [INFO] Opening database: bulkloader-progress-20100315.162913.sql3 [INFO] Opening database: bulkloader-results-20100315.162913.sql3 [INFO] Running in dry run mode, skipping remote_api setup [INFO] Connecting to 1.latest.myapp.appspot.com/remote_api Traceback (most recent call last): File /home/patrick/gae/current/bulkloader.py, line 68, in module run_file(__file__, globals()) File /home/patrick/gae/current/bulkloader.py, line 64, in run_file execfile(script_path, globals_) File /home/patrick/gae/google_appengine-1.3.1/google/appengine/tools/bulkloader.py, line 4031, in module sys.exit(main(sys.argv)) File /home/patrick/gae/google_appengine-1.3.1/google/appengine/tools/bulkloader.py, line 4027, in main return _PerformBulkload(arg_dict) File /home/patrick/gae/google_appengine-1.3.1/google/appengine/tools/bulkloader.py, line 3913, in _PerformBulkload return_code = app.Run() File /home/patrick/gae/google_appengine-1.3.1/google/appengine/tools/bulkloader.py, line 3170, in Run kinds = self.RunPostAuthentication() File /home/patrick/gae/google_appengine-1.3.1/google/appengine/tools/bulkloader.py, line 3314, in RunPostAuthentication return self.request_manager.GetSchemaKinds() File /home/patrick/gae/google_appengine-1.3.1/google/appengine/tools/bulkloader.py, line 1222, in GetSchemaKinds global_stat = stats.GlobalStat.all().get() File /home/patrick/gae/google_appengine-1.3.1/google/appengine/ext/db/__init__.py, line 1564, in get results = self.fetch(1, rpc=rpc) File /home/patrick/gae/google_appengine-1.3.1/google/appengine/ext/db/__init__.py, line 1616, in fetch raw = raw_query.Get(limit, offset, rpc=rpc) File /home/patrick/gae/google_appengine-1.3.1/google/appengine/api/datastore.py, line 1183, in Get limit=limit, offset=offset, prefetch_count=limit, **kwargs)._Get(limit) File /home/patrick/gae/google_appengine-1.3.1/google/appengine/api/datastore.py, line 1110, in _Run datastore_pb.QueryResult(), rpc) File /home/patrick/gae/google_appengine-1.3.1/google/appengine/api/datastore.py, line 173, in _MakeSyncCall rpc = CreateRPC(service) File /home/patrick/gae/google_appengine-1.3.1/google/appengine/api/datastore.py, line 192, in CreateRPC return DatastoreRPC(service, deadline, callback) File /home/patrick/gae/google_appengine-1.3.1/google/appengine/api/apiproxy_stub_map.py, line 345, in __init__ self.__rpc = CreateRPC(service) File /home/patrick/gae/google_appengine-1.3.1/google/appengine/api/apiproxy_stub_map.py, line 54, in CreateRPC assert stub, 'No api proxy found for service %s' % service AssertionError: No api proxy found for service datastore_v3 -- Patrick H. Twohig. Namazu Studios P.O. Box 34161 San Diego, CA 92163-4161 -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] No api proxy found for service datastore_v3
Still having that issue :( I'm gonna grok the docs a little more. On Mon, Mar 15, 2010 at 5:06 PM, Eli Jones eli.jo...@gmail.com wrote: to be sure.. you should probably run it using python2.5 to run the bulkloader. on my machine which came with 2.6 automatically installed, i just did: apt-get install python2.5 and then I run --dumps like so: # python2.5 bulkloader.py --dump --url=http://myapp.appspot.com/remote_api--filename=myapp.dump --batch_size=100 On Mon, Mar 15, 2010 at 7:57 PM, Patrick Twohig patr...@namazustudios.com wrote: I'm trying to bulkload data from my app, but it fails on this call. I browsed through the archives and found a post by Nick Johnson earlier and he suggested changing the fetch limit per request, but that didn't seem to see much. Any suggestions? I'm thinking that I just forgot to set some environment variable or something. Thanks, Pat. Full trace: patr...@gentoo ~/test/test_gae_bulkload $ bulkloader.py --batch_size=2 --dry_run --filename=backup.csv --dump --url= http://1.latest.myapp.appspot.com/remote_api /home/patrick/gae/google_appengine-1.3.1/google/appengine/ext/remote_api/remote_api_stub.py:64: DeprecationWarning: the sha module is deprecated; use the hashlib module instead import sha /home/patrick/gae/google_appengine-1.3.1/google/appengine/tools/dev_appserver_login.py:33: DeprecationWarning: the md5 module is deprecated; use hashlib instead import md5 [INFO] Logging to bulkloader-log-20100315.162913 [INFO] Throttling transfers: [INFO] Bandwidth: 25 bytes/second [INFO] HTTP connections: 8/second [INFO] Entities inserted/fetched/modified: 20/second [INFO] Opening database: bulkloader-progress-20100315.162913.sql3 [INFO] Opening database: bulkloader-results-20100315.162913.sql3 [INFO] Running in dry run mode, skipping remote_api setup [INFO] Connecting to 1.latest.myapp.appspot.com/remote_api Traceback (most recent call last): File /home/patrick/gae/current/bulkloader.py, line 68, in module run_file(__file__, globals()) File /home/patrick/gae/current/bulkloader.py, line 64, in run_file execfile(script_path, globals_) File /home/patrick/gae/google_appengine-1.3.1/google/appengine/tools/bulkloader.py, line 4031, in module sys.exit(main(sys.argv)) File /home/patrick/gae/google_appengine-1.3.1/google/appengine/tools/bulkloader.py, line 4027, in main return _PerformBulkload(arg_dict) File /home/patrick/gae/google_appengine-1.3.1/google/appengine/tools/bulkloader.py, line 3913, in _PerformBulkload return_code = app.Run() File /home/patrick/gae/google_appengine-1.3.1/google/appengine/tools/bulkloader.py, line 3170, in Run kinds = self.RunPostAuthentication() File /home/patrick/gae/google_appengine-1.3.1/google/appengine/tools/bulkloader.py, line 3314, in RunPostAuthentication return self.request_manager.GetSchemaKinds() File /home/patrick/gae/google_appengine-1.3.1/google/appengine/tools/bulkloader.py, line 1222, in GetSchemaKinds global_stat = stats.GlobalStat.all().get() File /home/patrick/gae/google_appengine-1.3.1/google/appengine/ext/db/__init__.py, line 1564, in get results = self.fetch(1, rpc=rpc) File /home/patrick/gae/google_appengine-1.3.1/google/appengine/ext/db/__init__.py, line 1616, in fetch raw = raw_query.Get(limit, offset, rpc=rpc) File /home/patrick/gae/google_appengine-1.3.1/google/appengine/api/datastore.py, line 1183, in Get limit=limit, offset=offset, prefetch_count=limit, **kwargs)._Get(limit) File /home/patrick/gae/google_appengine-1.3.1/google/appengine/api/datastore.py, line 1110, in _Run datastore_pb.QueryResult(), rpc) File /home/patrick/gae/google_appengine-1.3.1/google/appengine/api/datastore.py, line 173, in _MakeSyncCall rpc = CreateRPC(service) File /home/patrick/gae/google_appengine-1.3.1/google/appengine/api/datastore.py, line 192, in CreateRPC return DatastoreRPC(service, deadline, callback) File /home/patrick/gae/google_appengine-1.3.1/google/appengine/api/apiproxy_stub_map.py, line 345, in __init__ self.__rpc = CreateRPC(service) File /home/patrick/gae/google_appengine-1.3.1/google/appengine/api/apiproxy_stub_map.py, line 54, in CreateRPC assert stub, 'No api proxy found for service %s' % service AssertionError: No api proxy found for service datastore_v3 -- Patrick H. Twohig. Namazu Studios P.O. Box 34161 San Diego, CA 92163-4161 -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.comgoogle-appengine%2bunsubscr...@googlegroups.com . For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message
Re: [google-appengine] Re: GAE/J Performance (Slowness)
Scratch one cause off of the list. I found out that the client would always send each request without credentials, the server would kick back a 401 and then the client would resubmit with credentials every time! It was nuts, two requests for each useful request :( Since doing that the app has been performing much better. On Fri, Mar 12, 2010 at 11:55 PM, Tim Hoffman zutes...@gmail.com wrote: Hi Have you checked that startup times on a lightly used system are not inflating your response times ? T On Mar 13, 3:05 pm, Patrick Twohig patr...@namazustudios.com wrote: Hi, I asked once before, but I'm getting some painfully slow requests in my app. Even on queries that I am certain only fetch a few entities, (5-10 max under absolutely worst case circumstances). I also take advantage of memory caching wherever possible, and I have confirmed (via logging) that I'm pulling from the cache. Some requests are taking upwards of 2-3 seconds. And given that my app right now is mostly a web service and not running in a browser, there's quite a few things that are perplexing me. SO I have a few questions: - I'm running JAX-RS (Jersey), a little Groovy, low level API (no JDO or JPA), and Guice (yep, I'm a Google fanboy). I have identified where Groovy is being slow, and came up with a workaround for that, so I'm not too concerned. - What's a reasonable number of entities I should fetch from the datastore per request? I currently fetch at least one per request to validate a user's login credentials, so no matter what I'm fetching at least one per request. Adding up the numbers in my head, I don't think that - Even with aggressive caching, (all of my data objects get put in memcache), it still behaves rather slowly? What is the penalty speed-wise for using a transaction? I can't use caching with a transaction obviously because I can't be guaranteed any consistency of data. If I start a transaction and and commit it or roll it back and perform no operations in it, what is the penalty? I'm certain this happens in my app, albeit very infrequently. - What's up with queries? They just tend to be really really slow. - I know loading requests are a problem, but I'm not hitting many of those according to the logs, so I can rule those out as being a contributing factor (unless there's something I don't know). I also don't use JSPs, most of it is static. - How much of a performance difference does a bulk fetch/put cost versus a bunch of individual ones? I haven't seen much of a difference, but maybe I'm doing it wrong. - There are zero use cases in my application where more than two users write to the same entity group, except for a counter for which I use sharding offloaded to the task queue. Is there a penalty for many users reading from a single entity group at the same time? Thanks, -- Patrick H. Twohig. Namazu Studios P.O. Box 34161 San Diego, CA 92163-4161 -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.comgoogle-appengine%2bunsubscr...@googlegroups.com . For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Patrick H. Twohig. Namazu Studios P.O. Box 34161 San Diego, CA 92163-4161 -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Need help with a query...
So, the business requirements of my application need a query which needs to search for a property (let's call it foo) and a date property such that it will filter out all objects older than a certain date. I guess in GAE/J this would be something like Query q = new Query(MyKind) .addFilter(foo, GREATER_THAN, 0) .addFilter(bar, LESS_THAN, 42) .addFilter(date, GREATER_THAN, someDate); But, I'm guessing that wouldn't work because it's got inequality filters on multiple properties. Would there be a way to get a similar result without loading a huge amount of entities into memory and filtering them out manually? -- Patrick H. Twohig. Namazu Studios P.O. Box 34161 San Diego, CA 92163-4161 -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] GAE/J Performance (Slowness)
Hi, I asked once before, but I'm getting some painfully slow requests in my app. Even on queries that I am certain only fetch a few entities, (5-10 max under absolutely worst case circumstances). I also take advantage of memory caching wherever possible, and I have confirmed (via logging) that I'm pulling from the cache. Some requests are taking upwards of 2-3 seconds. And given that my app right now is mostly a web service and not running in a browser, there's quite a few things that are perplexing me. SO I have a few questions: - I'm running JAX-RS (Jersey), a little Groovy, low level API (no JDO or JPA), and Guice (yep, I'm a Google fanboy). I have identified where Groovy is being slow, and came up with a workaround for that, so I'm not too concerned. - What's a reasonable number of entities I should fetch from the datastore per request? I currently fetch at least one per request to validate a user's login credentials, so no matter what I'm fetching at least one per request. Adding up the numbers in my head, I don't think that - Even with aggressive caching, (all of my data objects get put in memcache), it still behaves rather slowly? What is the penalty speed-wise for using a transaction? I can't use caching with a transaction obviously because I can't be guaranteed any consistency of data. If I start a transaction and and commit it or roll it back and perform no operations in it, what is the penalty? I'm certain this happens in my app, albeit very infrequently. - What's up with queries? They just tend to be really really slow. - I know loading requests are a problem, but I'm not hitting many of those according to the logs, so I can rule those out as being a contributing factor (unless there's something I don't know). I also don't use JSPs, most of it is static. - How much of a performance difference does a bulk fetch/put cost versus a bunch of individual ones? I haven't seen much of a difference, but maybe I'm doing it wrong. - There are zero use cases in my application where more than two users write to the same entity group, except for a counter for which I use sharding offloaded to the task queue. Is there a penalty for many users reading from a single entity group at the same time? Thanks, -- Patrick H. Twohig. Namazu Studios P.O. Box 34161 San Diego, CA 92163-4161 -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: Selling python modules ?
I'm inclined to agree with Ikai on this one. I also think you should be able to monetize your application any way you wish, providing it's legal. -- Patrick H. Twohig. Namazu Studios P.O. Box 34161 San Diego, CA 92163-4161 -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Datastore Viewer throwing Server Error (500)
I think the size limit per entity is 1 megabyte. I could be wrong though. -- Patrick H. Twohig. Namazu Studios P.O. Box 34161 San Diego, CA 92163-4161 -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Datastore Viewer throwing Server Error (500)
Or rather that may apply to the size of the entity group as a whole. Are you sure that you're not tipping that limit? On Mon, Mar 8, 2010 at 10:10 AM, Patrick Twohig patr...@namazustudios.comwrote: I think the size limit per entity is 1 megabyte. I could be wrong though. -- Patrick H. Twohig. Namazu Studios P.O. Box 34161 San Diego, CA 92163-4161 -- Patrick H. Twohig. Namazu Studios P.O. Box 34161 San Diego, CA 92163-4161 -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: 1500 ms query latency
On this note, I'm not actually able to see the query latency chart. Is that something broken for me or everybody else? http://code.google.com/status/appengine/detail/datastore/2010/03/07#ae-trust-detail-datastore-query-latency On Fri, Mar 5, 2010 at 11:05 AM, Ikai L (Google) ika...@google.com wrote: Patrick, let us know if you do find any glaring differences. Nothing we've done yet has demonstrated that JDO/JPA introduces anything beyond negligible overhead to queries, but we're always in need of more good data. On Thu, Mar 4, 2010 at 11:47 PM, Patrick Twohig patr...@namazustudios.com wrote: Ikai, Off hand, no. Since I started using the low-level API, I've added lots of caching and the like so anything I could pull out of my code wouldn't exactly be a fair comparison. If I come across anything, I'll post it. When I was using JDO, I didn't see much that let me tune how aggressively it fetched entities from the datastore, which may have contributed to the frustrated I was having. I have had reasonably good success being able to adjust the fetching options manually. Pat. On Thu, Mar 4, 2010 at 11:56 AM, Ikai L (Google) ika...@google.com wrote: Patrick, do you have any examples of queries implemented using JDO/JPA versus the low-level API that can demonstrate that JDO/JPA queries introduce overhead? If you can post them, the datastore team will take a look at it. On Thu, Mar 4, 2010 at 9:49 AM, Patrick Twohig patr...@namazustudios.com wrote: Just a heads up, if you're using GAE/J and I've had much better performance with the low-level API over JDO/JPA. Pat. -- Patrick H. Twohig. Namazu Studios P.O. Box 34161 San Diego, CA 92163-4161 -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.comgoogle-appengine%2bunsubscr...@googlegroups.com . For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Ikai Lan Developer Programs Engineer, Google App Engine http://googleappengine.blogspot.com | http://twitter.com/app_engine -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.comgoogle-appengine%2bunsubscr...@googlegroups.com . For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Patrick H. Twohig. Namazu Studios P.O. Box 34161 San Diego, CA 92163-4161 -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.comgoogle-appengine%2bunsubscr...@googlegroups.com . For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Ikai Lan Developer Programs Engineer, Google App Engine http://googleappengine.blogspot.com | http://twitter.com/app_engine -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.comgoogle-appengine%2bunsubscr...@googlegroups.com . For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Patrick H. Twohig. Namazu Studios P.O. Box 34161 San Diego, CA 92163-4161 -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: 1500 ms query latency
Just a heads up, if you're using GAE/J and I've had much better performance with the low-level API over JDO/JPA. Pat. -- Patrick H. Twohig. Namazu Studios P.O. Box 34161 San Diego, CA 92163-4161 -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: 1500 ms query latency
Ikai, Off hand, no. Since I started using the low-level API, I've added lots of caching and the like so anything I could pull out of my code wouldn't exactly be a fair comparison. If I come across anything, I'll post it. When I was using JDO, I didn't see much that let me tune how aggressively it fetched entities from the datastore, which may have contributed to the frustrated I was having. I have had reasonably good success being able to adjust the fetching options manually. Pat. On Thu, Mar 4, 2010 at 11:56 AM, Ikai L (Google) ika...@google.com wrote: Patrick, do you have any examples of queries implemented using JDO/JPA versus the low-level API that can demonstrate that JDO/JPA queries introduce overhead? If you can post them, the datastore team will take a look at it. On Thu, Mar 4, 2010 at 9:49 AM, Patrick Twohig patr...@namazustudios.com wrote: Just a heads up, if you're using GAE/J and I've had much better performance with the low-level API over JDO/JPA. Pat. -- Patrick H. Twohig. Namazu Studios P.O. Box 34161 San Diego, CA 92163-4161 -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.comgoogle-appengine%2bunsubscr...@googlegroups.com . For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Ikai Lan Developer Programs Engineer, Google App Engine http://googleappengine.blogspot.com | http://twitter.com/app_engine -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.comgoogle-appengine%2bunsubscr...@googlegroups.com . For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Patrick H. Twohig. Namazu Studios P.O. Box 34161 San Diego, CA 92163-4161 -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: 1500 ms query latency
I've been having trouble with exceedingly long queries as well...I've noticed that on a good day it appears that a query is taking 500 milliseconds. I have one or two complex requests that make several queries and that adds up quickly. Caching has helped immensely, but if data frequently changes it doesn't help much... On Wed, Mar 3, 2010 at 8:15 PM, Thomas Johansson prenc...@gmail.com wrote: This is not app specific. Look at the appengine status for datastore queries: http://coI've been having issues with high latency issues.de.google.com/status/appengine/detail/datastore/2010/03/03#ae-trust-detail-datastore-query-latencyhttp://code.google.com/status/appengine/detail/datastore/2010/03/03#ae-trust-detail-datastore-query-latency . We are seeing the same on the app we are developing, along with long cold startup times the last few days (which otherwise had improved greatly after the downtime). On Mar 4, 1:57 am, Ikai L (Google) ika...@google.com wrote: Robert, what's your application ID? On Wed, Mar 3, 2010 at 3:35 PM, Robert Lancer robert.lan...@gmail.com wrote: If you monitor App Engine Status you will notice that query latency has been floating around 1500 ms today. How can we deliver a product on top of app engine with this dismal level of performance? Google are you going to fix this issue anytime soon? I would appreciate if you treated these performance issues with the same level of concern as an outage. I have spent the past five months developing my app. http://code.google.com/status/appengine/detail/datastore/2010/03/03#a. .. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com . To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.comgoogle-appengine%2bunsubscr...@googlegroups.com google-appengine%2bunsubscr...@googlegroups.comgoogle-appengine%252bunsubscr...@googlegroups.com . For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Ikai Lan Developer Programs Engineer, Google App Enginehttp:// googleappengine.blogspot.com|http://twitter.com/app_engine -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.comgoogle-appengine%2bunsubscr...@googlegroups.com . For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Patrick H. Twohig. Namazu Studios P.O. Box 34161 San Diego, CA 92163-4161 -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] GAE/J Low Level API Transaction management.
FWIW, it looks like that particular issue is a bug with the local development server and not on the actual app engine. I tried it out in my app and it seems that it finds all objects properly. Pat. On Thu, Feb 25, 2010 at 6:24 PM, Jeff Schnitzer j...@infohazard.org wrote: Looks good to me, but I don't work for Google so the only help I can offer is starring the issue :-( Jeff On Thu, Feb 25, 2010 at 5:44 PM, Patrick Twohig patr...@namazustudios.com wrote: Jeff, I made an issue regarding it, let me know if my unit test left anything out. Thanks, Patrick. http://code.google.com/p/googleappengine/issues/detail?id=2879 On Fri, Feb 19, 2010 at 1:16 AM, Patrick Twohig patr...@namazustudios.com wrote: I had that suspicion, but, I wasn't 100% sure. I wonder if it's an issue that shows up in production or just local datastore. Pat. On Thu, Feb 18, 2010 at 1:34 PM, Jeff Schnitzer j...@infohazard.org wrote: I just created a unit test for this case (there wasn't one before) and sure enough, it fails. Looks like a bug in appengine. Create an issue, I'll star it. The failing unit test is the last one in this file: http://code.google.com/p/objectify-appengine/source/browse/trunk/src/com/googlecode/ob jectify/test/QueryTests.java FWIW, our implementation is the ancestor() method: http://code.google.com/p/objectify-appengine/source/browse/trunk/src/com/googlecode/objectify/impl/QueryImpl.java Javadocs are on the interface class: http://objectify-appengine.googlecode.com/svn/trunk/javadoc/com/googlecode/objectify/Query.html Jeff On Thu, Feb 18, 2010 at 11:22 AM, Patrick Twohig patr...@namazustudios.com wrote: Duly noted. However, I'm not about to rewrite a bulk of my code over a single issue I'm having. At the time I started this, I hadn't realized Objectify existed and tried using JDO which turned out to be a gigantic nightmare, so I slimmed it down and wrote my own wrapper similar to Objectify. Right now the only pressing issue is that ancestor queries aren't working as expected and it's driving me nuts. Could you perhaps point me to some source in Objectify that executes an ancestor query that I may be able to see? On Wed, Feb 17, 2010 at 10:05 PM, Jeff Schnitzer j...@infohazard.org wrote: The documentation I wrote up here might help: http://code.google.com/p/objectify-appengine/wiki/Concepts You might consider using something like Objectify (or Twig, or SimpleDS, etc) instead of the Low-Level API. Jeff On Wed, Feb 17, 2010 at 9:23 PM, Patrick Twohig patr...@namazustudios.com wrote: Aside from the Javadocs, does there exist any further documentation on the GAE/J low-level API. I've had a handfull of issues with it so far and I'm at a bit of a loss. I'm having trouble with ancestor queries. Specifically, I'm not getting any child entities when I look for objects with no kind specified and just an ancestor. However, the equivalent code in python performs as expected. I was also curious how GAE/J organizes transactions. From what I gather in the documentation, each transaction is organized in a ThreadLocal stack. Every time you call DatastoreService.beginTransaction() it creates a new transaction, pushes it on the thread local stack then when it's done, it's popped of. I was curious if it may be possible to add a method that can provide the current transaction given a particular key somehow. -- Patrick H. Twohig. Namazu Studios P.O. Box 34161 San Diego, CA 92163-4161 -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.comgoogle-appengine%2bunsubscr...@googlegroups.com . For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.comgoogle-appengine%2bunsubscr...@googlegroups.com . For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Patrick H. Twohig. Namazu Studios P.O. Box 34161 San Diego, CA 92163-4161 -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email
[google-appengine] Probably a bug in the AppEngine SDK...
I got this with a query in production just now. I searched for it but I didn't find much. Apologies in advance if this has been brought up before. Unable to determine what index is missing. This is probably a bug in the App Engine SDK. com.google.appengine.api.datastore.DatastoreNeedIndexException: no matching index found.. Unable to determine what index is missing. This is probably a bug in the App Engine SDK. at com.google.appengine.api.datastore.DatastoreApiHelper.translateError(DatastoreApiHelper.java:39) at com.google.appengine.api.datastore.DatastoreApiHelper.makeSyncCall(DatastoreApiHelper.java:60) at com.google.appengine.api.datastore.PreparedQueryImpl.countEntities(PreparedQueryImpl.java:106) -- Patrick H. Twohig. Namazu Studios P.O. Box 34161 San Diego, CA 92163-4161 -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] GAE/J Low Level API Transaction management.
Jeff, I made an issue regarding it, let me know if my unit test left anything out. Thanks, Patrick. http://code.google.com/p/googleappengine/issues/detail?id=2879 On Fri, Feb 19, 2010 at 1:16 AM, Patrick Twohig patr...@namazustudios.comwrote: I had that suspicion, but, I wasn't 100% sure. I wonder if it's an issue that shows up in production or just local datastore. Pat. On Thu, Feb 18, 2010 at 1:34 PM, Jeff Schnitzer j...@infohazard.orgwrote: I just created a unit test for this case (there wasn't one before) and sure enough, it fails. Looks like a bug in appengine. Create an issue, I'll star it. The failing unit test is the last one in this file: http://code.google.com/p/objectify-appengine/source/browse/trunk/src/com/googlecode/ob jectify/test/QueryTests.javahttp://code.google.com/p/objectify-appengine/source/browse/trunk/src/com/googlecode/objectify/test/QueryTests.java FWIW, our implementation is the ancestor() method: http://code.google.com/p/objectify-appengine/source/browse/trunk/src/com/googlecode/objectify/impl/QueryImpl.java Javadocs are on the interface class: http://objectify-appengine.googlecode.com/svn/trunk/javadoc/com/googlecode/objectify/Query.html Jeff On Thu, Feb 18, 2010 at 11:22 AM, Patrick Twohig patr...@namazustudios.com wrote: Duly noted. However, I'm not about to rewrite a bulk of my code over a single issue I'm having. At the time I started this, I hadn't realized Objectify existed and tried using JDO which turned out to be a gigantic nightmare, so I slimmed it down and wrote my own wrapper similar to Objectify. Right now the only pressing issue is that ancestor queries aren't working as expected and it's driving me nuts. Could you perhaps point me to some source in Objectify that executes an ancestor query that I may be able to see? On Wed, Feb 17, 2010 at 10:05 PM, Jeff Schnitzer j...@infohazard.org wrote: The documentation I wrote up here might help: http://code.google.com/p/objectify-appengine/wiki/Concepts You might consider using something like Objectify (or Twig, or SimpleDS, etc) instead of the Low-Level API. Jeff On Wed, Feb 17, 2010 at 9:23 PM, Patrick Twohig patr...@namazustudios.com wrote: Aside from the Javadocs, does there exist any further documentation on the GAE/J low-level API. I've had a handfull of issues with it so far and I'm at a bit of a loss. I'm having trouble with ancestor queries. Specifically, I'm not getting any child entities when I look for objects with no kind specified and just an ancestor. However, the equivalent code in python performs as expected. I was also curious how GAE/J organizes transactions. From what I gather in the documentation, each transaction is organized in a ThreadLocal stack. Every time you call DatastoreService.beginTransaction() it creates a new transaction, pushes it on the thread local stack then when it's done, it's popped of. I was curious if it may be possible to add a method that can provide the current transaction given a particular key somehow. -- Patrick H. Twohig. Namazu Studios P.O. Box 34161 San Diego, CA 92163-4161 -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.comgoogle-appengine%2bunsubscr...@googlegroups.com . For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com . To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.comgoogle-appengine%2bunsubscr...@googlegroups.com . For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Patrick H. Twohig. Namazu Studios P.O. Box 34161 San Diego, CA 92163-4161 -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.comgoogle-appengine%2bunsubscr...@googlegroups.com . For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.comgoogle-appengine%2bunsubscr...@googlegroups.com . For more options, visit
Re: [google-appengine] Feature suggestion: Paying Ststem
Google Checkout API provides a payment system http://code.google.com/apis/checkout/ If you're not a fan of that, look into Paypal or talk to your business's bank. My bank has a payment system we can use if we so desire. On Wed, Feb 17, 2010 at 12:38 AM, kang areyouloo...@gmail.com wrote: I think Google App Engine need to give us a paying system like App Store and Android Market or give us some API about paying. We need a web app store :-) Thanks -- Stay hungry,Stay foolish. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.comgoogle-appengine%2bunsubscr...@googlegroups.com . For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Patrick H. Twohig. Namazu Studios P.O. Box 34161 San Diego, CA 92163-4161 -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] GAE/J Low Level API Transaction management.
I had that suspicion, but, I wasn't 100% sure. I wonder if it's an issue that shows up in production or just local datastore. Pat. On Thu, Feb 18, 2010 at 1:34 PM, Jeff Schnitzer j...@infohazard.org wrote: I just created a unit test for this case (there wasn't one before) and sure enough, it fails. Looks like a bug in appengine. Create an issue, I'll star it. The failing unit test is the last one in this file: http://code.google.com/p/objectify-appengine/source/browse/trunk/src/com/googlecode/ob jectify/test/QueryTests.java FWIW, our implementation is the ancestor() method: http://code.google.com/p/objectify-appengine/source/browse/trunk/src/com/googlecode/objectify/impl/QueryImpl.java Javadocs are on the interface class: http://objectify-appengine.googlecode.com/svn/trunk/javadoc/com/googlecode/objectify/Query.html Jeff On Thu, Feb 18, 2010 at 11:22 AM, Patrick Twohig patr...@namazustudios.com wrote: Duly noted. However, I'm not about to rewrite a bulk of my code over a single issue I'm having. At the time I started this, I hadn't realized Objectify existed and tried using JDO which turned out to be a gigantic nightmare, so I slimmed it down and wrote my own wrapper similar to Objectify. Right now the only pressing issue is that ancestor queries aren't working as expected and it's driving me nuts. Could you perhaps point me to some source in Objectify that executes an ancestor query that I may be able to see? On Wed, Feb 17, 2010 at 10:05 PM, Jeff Schnitzer j...@infohazard.org wrote: The documentation I wrote up here might help: http://code.google.com/p/objectify-appengine/wiki/Concepts You might consider using something like Objectify (or Twig, or SimpleDS, etc) instead of the Low-Level API. Jeff On Wed, Feb 17, 2010 at 9:23 PM, Patrick Twohig patr...@namazustudios.com wrote: Aside from the Javadocs, does there exist any further documentation on the GAE/J low-level API. I've had a handfull of issues with it so far and I'm at a bit of a loss. I'm having trouble with ancestor queries. Specifically, I'm not getting any child entities when I look for objects with no kind specified and just an ancestor. However, the equivalent code in python performs as expected. I was also curious how GAE/J organizes transactions. From what I gather in the documentation, each transaction is organized in a ThreadLocal stack. Every time you call DatastoreService.beginTransaction() it creates a new transaction, pushes it on the thread local stack then when it's done, it's popped of. I was curious if it may be possible to add a method that can provide the current transaction given a particular key somehow. -- Patrick H. Twohig. Namazu Studios P.O. Box 34161 San Diego, CA 92163-4161 -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.comgoogle-appengine%2bunsubscr...@googlegroups.com . For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.comgoogle-appengine%2bunsubscr...@googlegroups.com . For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Patrick H. Twohig. Namazu Studios P.O. Box 34161 San Diego, CA 92163-4161 -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.comgoogle-appengine%2bunsubscr...@googlegroups.com . For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.comgoogle-appengine%2bunsubscr...@googlegroups.com . For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Patrick H. Twohig. Namazu Studios P.O. Box 34161 San Diego, CA 92163-4161 -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email
Re: [google-appengine] Re: Lucene/Compass
I run it in my app that's on the App Engine right now, app's in staging so I have it blocked off to people not in my domain. Shay Banon has a video clip of it running on his blog, and then I submitted a patch to the compass project that prevents transactions from butting heads when writing the index to the datastore. I've seen it working with mine own two eyes, as they say. It's also listed as working in the will it play in app engine page. Unfortunately, I'm concerned about the scalability of it. It seems to take quite a while to index a single object and every so often the search index gets corrupted because a request to index an object gets shut down by a deadline exceeded exception. I try to recover by rolling back the work in progress and releasing all locks, but the request still gets terminated before that can happen. This is with 4-5 users using it just to do testing/tuning of the app. I can't imagine how it would work under any actual load. It's also using quite a bit of CPU usage. On Wed, Feb 17, 2010 at 7:21 AM, uo yon...@googlemail.com wrote: I too encountered challenges and I still cante believe anyone runs compass on gae... on close inspection of some of the core compass classes like org.compass.core.Compass implements javax.naming.referenceable which isnt in Google's whitelist. On Feb 16, 3:33 am, Patrick Twohig patr...@namazustudios.com wrote: Yeah. I know. The beta version of Compass supports GAE On Mon, Feb 15, 2010 at 1:13 AM, Tonny 12br...@gmail.com wrote: Not having access to the file system on GAE will prevent you from using the out-of-the-box Lucene Index implementations (which uses the filesystem). On Feb 13, 12:40 am, Patrick Twohig patr...@namazustudios.com wrote: I was curious if anybody is using Lucene and Compass and how well it works in the app engine. I've seen some other posts indicating that it actually manages to handle quite a bit of data, but I was curious if anybody else out there could attest to how well it works. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com . To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.comgoogle-appengine%2bunsubscr...@googlegroups.com google-appengine%2bunsubscr...@googlegroups.comgoogle-appengine%252bunsubscr...@googlegroups.com . For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Patrick H. Twohig. Namazu Studios P.O. Box 34161 San Diego, CA 92163-4161 -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.comgoogle-appengine%2bunsubscr...@googlegroups.com . For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Patrick H. Twohig. Namazu Studios P.O. Box 34161 San Diego, CA 92163-4161 -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] GAE/J Low Level API Transaction management.
Duly noted. However, I'm not about to rewrite a bulk of my code over a single issue I'm having. At the time I started this, I hadn't realized Objectify existed and tried using JDO which turned out to be a gigantic nightmare, so I slimmed it down and wrote my own wrapper similar to Objectify. Right now the only pressing issue is that ancestor queries aren't working as expected and it's driving me nuts. Could you perhaps point me to some source in Objectify that executes an ancestor query that I may be able to see? On Wed, Feb 17, 2010 at 10:05 PM, Jeff Schnitzer j...@infohazard.orgwrote: The documentation I wrote up here might help: http://code.google.com/p/objectify-appengine/wiki/Concepts You might consider using something like Objectify (or Twig, or SimpleDS, etc) instead of the Low-Level API. Jeff On Wed, Feb 17, 2010 at 9:23 PM, Patrick Twohig patr...@namazustudios.com wrote: Aside from the Javadocs, does there exist any further documentation on the GAE/J low-level API. I've had a handfull of issues with it so far and I'm at a bit of a loss. I'm having trouble with ancestor queries. Specifically, I'm not getting any child entities when I look for objects with no kind specified and just an ancestor. However, the equivalent code in python performs as expected. I was also curious how GAE/J organizes transactions. From what I gather in the documentation, each transaction is organized in a ThreadLocal stack. Every time you call DatastoreService.beginTransaction() it creates a new transaction, pushes it on the thread local stack then when it's done, it's popped of. I was curious if it may be possible to add a method that can provide the current transaction given a particular key somehow. -- Patrick H. Twohig. Namazu Studios P.O. Box 34161 San Diego, CA 92163-4161 -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.comgoogle-appengine%2bunsubscr...@googlegroups.com . For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.comgoogle-appengine%2bunsubscr...@googlegroups.com . For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Patrick H. Twohig. Namazu Studios P.O. Box 34161 San Diego, CA 92163-4161 -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] GAE/J Low Level API Transaction management.
Aside from the Javadocs, does there exist any further documentation on the GAE/J low-level API. I've had a handfull of issues with it so far and I'm at a bit of a loss. I'm having trouble with ancestor queries. Specifically, I'm not getting any child entities when I look for objects with no kind specified and just an ancestor. However, the equivalent code in python performs as expected. I was also curious how GAE/J organizes transactions. From what I gather in the documentation, each transaction is organized in a ThreadLocal stack. Every time you call DatastoreService.beginTransaction() it creates a new transaction, pushes it on the thread local stack then when it's done, it's popped of. I was curious if it may be possible to add a method that can provide the current transaction given a particular key somehow. -- Patrick H. Twohig. Namazu Studios P.O. Box 34161 San Diego, CA 92163-4161 -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: Lucene/Compass
Yeah. I know. The beta version of Compass supports GAE On Mon, Feb 15, 2010 at 1:13 AM, Tonny 12br...@gmail.com wrote: Not having access to the file system on GAE will prevent you from using the out-of-the-box Lucene Index implementations (which uses the filesystem). On Feb 13, 12:40 am, Patrick Twohig patr...@namazustudios.com wrote: I was curious if anybody is using Lucene and Compass and how well it works in the app engine. I've seen some other posts indicating that it actually manages to handle quite a bit of data, but I was curious if anybody else out there could attest to how well it works. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.comgoogle-appengine%2bunsubscr...@googlegroups.com . For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Patrick H. Twohig. Namazu Studios P.O. Box 34161 San Diego, CA 92163-4161 -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Google Calendar Sharing
Wrong group for this sort of thing. You need the group for Google Apps not Google App Engine. On Thu, Feb 11, 2010 at 3:15 AM, Rajalakshmi Subramanian raji.sm...@gmail.com wrote: Hi everydody I'm working with Google app for domain.Admin want to share user A's calendar to user B is it possible do this without A's idpassword. Regards Rajalakshmi -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.comgoogle-appengine%2bunsubscr...@googlegroups.com . For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Patrick H. Twohig. Namazu Studios P.O. Box 34161 San Diego, CA 92163-4161 -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Lucene/Compass
I was curious if anybody is using Lucene and Compass and how well it works in the app engine. I've seen some other posts indicating that it actually manages to handle quite a bit of data, but I was curious if anybody else out there could attest to how well it works. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] GAE/J recommended best practices.
Thanks Ikai, I found out I had forgot to turn on memory caching for some thigns in my app and the sluggishness went away. I still get the loading request every once in a while, but once it goes fully live (out of beta) that probably wont' be an issue as we're expecting fairly regular traffic. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] GAE/J recommended best practices.
I'm running a GAE/J application right now and it's behaving a bit sluggishly. Is there a set of guidelines or best practices that I may be able to look at to boost the performance? -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: How memcache work?
I believe that the memcache service is implemented on top of this. I could be wrong, though: http://memcached.org/ On Mon, Feb 1, 2010 at 10:31 AM, Satoshi satoshi.nakaj...@gmail.com wrote: There is no transaction mechanism around memcache. Although it is technically possible to use an entity group (or groups) on the datastore to create a transaction mechainsm on top of memcache, it does not make sense to use slower mechanism (datastore) on top of faster mechanism (memcache). Generally speaking, you should use memcache to return a snapshot of data to the client. If you need transactions, go directly to the datastore. Satoshi On Feb 1, 6:35 am, masterzim masterzim@googlemail.com wrote: In official documentation I didn't find nothing about memcache transaction. What happens if I access to memcache from different request( this is save? ) , is transactions atomic? This will be great if somebody can explain this. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.comgoogle-appengine%2bunsubscr...@googlegroups.com . For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Patrick H. Twohig. Namazu Studios P.O. Box 34161 San Diego, CA 92163-4161 Office: 619.862.2890 x100 Cell: 619.453.5075 Twitter: @svm_invictvs IRC: svm_invic...@irc.freenode.net ##java, #android-dev, #iphonedev, #appengine http://www.namazustudios.com/ This communication, and any attachments, shall be considered confidential and proprietary information of Namazu Studios LLC. This message, and attachments, are intended for the listed recipients only. If you are not one of the intended recipients, please destroy all copies of this communication. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] security in task queue servlets
I believe that the App Engine injects the pertinent information from the
Users service. For instance, when you call
HttpServletRequest.getUserPrinicpal(), you're getting values injected by the
UsersService.
On Mon, Feb 1, 2010 at 11:02 AM, Eli Jones eli.jo...@gmail.com wrote:
If you have a compelling reason for really locking down the task queue url
(and Require Admin login isn't enough), you could create a mechanism that
creates a task name for each queued task.. and the task verifies that its
name is correct.
You could have the task use the X-AppEngine-TaskName header to check its
name..
So.. when you add a task to the queue.. you do something like this:
taskName = getUniqueTaskName()
nameHash = getHash(taskName)
taskqueue.add(url= '/myTaskQueue', countdown = 0,
name = taskName,
params = {'nameHash' : nameHash})
and.. in the first part of the /myTaskQueue code.. you could have it verify
that the 'nameHash' param is equal to getHash() of the TaskName you grab
from the header..
On Sat, Jan 30, 2010 at 4:07 PM, Patrick Linskey plins...@gmail.comwrote:
Hi,
I'd like to programmatically ensure that my task queue servlets are
only invoked via the task queue. I've got a security constraint in my
web.xml, but I'd like to also check in code to avoid any potential mis-
configuration in the future.
Is there any supported means to do such a check?
I tried looking at the contents of the HttpServletRequest (isUserInRole
(), getAuthType(), getUserPrincipal(), getRemoteName()), to no avail.
I also tried UserServiceFactory.getUserService().isAdmin(), but
received an exception informing me that no user was logged in.
I can see that there are a number of task queue-specific HTTP headers.
Currently, I'm checking that X-AppEngine-TaskRetryCount is present,
and if so, assuming that the request has come from the task queue and
that it's therefore safe to process. Empirically, it looks like GAE
strips out the X-AppEngine-TaskRetryCount header when I specify it in
a curl-sourced request. Is this a safe assumption to rely on? Are
there plans to document a reliable way to ensure servlet security in a
task queue environment? Is there something else that I'm missing?
Also, in an ideal world, it'd be nice if request.isUserInRole(admin)
would return true at the appropriate times.
Thanks,
-Patrick
--
You received this message because you are subscribed to the Google Groups
Google App Engine group.
To post to this group, send email to google-appeng...@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.comgoogle-appengine%2bunsubscr...@googlegroups.com
.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.
--
You received this message because you are subscribed to the Google Groups
Google App Engine group.
To post to this group, send email to google-appeng...@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.comgoogle-appengine%2bunsubscr...@googlegroups.com
.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.
--
Patrick H. Twohig.
Namazu Studios
P.O. Box 34161
San Diego, CA 92163-4161
Office: 619.862.2890 x100
Cell: 619.453.5075
Twitter: @svm_invictvs
IRC: svm_invic...@irc.freenode.net ##java, #android-dev, #iphonedev,
#appengine
http://www.namazustudios.com/
This communication, and any attachments, shall be considered confidential
and proprietary information of Namazu Studios LLC. This message, and
attachments, are intended for the listed recipients only. If you are not
one of the intended recipients, please destroy all copies of this
communication.
--
You received this message because you are subscribed to the Google Groups
Google App Engine group.
To post to this group, send email to google-appeng...@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Ancestor Queries...
I've got a snippet of code that looks something like this
public getSubEntities(Transaction transaction, Key parent) {
com.google.appengine.api.datastore.Query query;
com.google.appengine.api.datastore.PreparedQuery pquery;
MapKey, Entity out = new HashMapKey, Entity();
query = new
com.google.appengine.api.datastore.Query().setAncestor(parent);
pquery = datastore.prepare(transaction, query);
for (Entity e : pquery.asQueryResultIterable()) {
out.put(e.getKey(), e);
}
return out;
}
...but when I run it, it never finds any sub entities of the given key. I'm
guessing I'm missing somthing? I'm trying to find all the direct
sub-entities of the given entity key. I've checked that they're in the
datastore, but for whatever reason the loop never hits. Any suggestions? I
haven't tried this in the app engine, just the local data store.
--
Patrick H. Twohig.
Namazu Studios
P.O. Box 34161
San Diego, CA 92163-4161
Office: 619.862.2890 x100
Cell: 619.453.5075
Twitter: @svm_invictvs
IRC: svm_invic...@irc.freenode.net ##java, #android-dev, #iphonedev,
#appengine
http://www.namazustudios.com/
This communication, and any attachments, shall be considered confidential
and proprietary information of Namazu Studios LLC. This message, and
attachments, are intended for the listed recipients only. If you are not
one of the intended recipients, please destroy all copies of this
communication.
--
You received this message because you are subscribed to the Google Groups
Google App Engine group.
To post to this group, send email to google-appeng...@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Stolen laptops
I perfer spam only served with eggs. Sent from my G1 phone. On Jan 17, 2010 4:54 PM, OvermindDL1 overmind...@gmail.com wrote: On Sun, Jan 17, 2010 at 3:05 PM, Stolen bike lady jjkjhjkhhkjhy...@live.com wrote: Stolen laptop... So was this made on appengine or something? -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.comgoogle-appengine%2bunsubscr...@googlegroups.com . For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: PersistenceManager.getObjectsById(*)
*Mendokusai.* I'm moving over to the low-level API. On Fri, Jan 15, 2010 at 11:25 AM, Erem ehb...@gmail.com wrote: Bump. Particularly on the future plans to implement. On Jan 14, 12:31 pm, Patrick Twohig patr...@namazustudios.com wrote: Why doesn't the family of methods PersistenceManager.getObjectsById actually perform a bulk fetch from the datastore? Are there future plans to implement it as a bulk fetch? -- Patrick H. Twohig. Namazu Studios P.O. Box 34161 San Diego, CA 92163-4161 Office: 619.862.2890 x100 Cell: 619.453.5075 Twitter: @svm_invictvs IRC: svm_invic...@irc.freenode.net ##java, #android-dev, #iphonedev, #appengine http://www.namazustudios.com/ This communication, and any attachments, shall be considered confidential and proprietary information of Namazu Studios LLC. This message, and attachments, are intended for the listed recipients only. If you are not one of the intended recipients, please destroy all copies of this communication. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.comgoogle-appengine%2bunsubscr...@googlegroups.com . For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Patrick H. Twohig. Namazu Studios P.O. Box 34161 San Diego, CA 92163-4161 Office: 619.862.2890 x100 Cell: 619.453.5075 Twitter: @svm_invictvs IRC: svm_invic...@irc.freenode.net ##java, #android-dev, #iphonedev, #appengine http://www.namazustudios.com/ This communication, and any attachments, shall be considered confidential and proprietary information of Namazu Studios LLC. This message, and attachments, are intended for the listed recipients only. If you are not one of the intended recipients, please destroy all copies of this communication. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: PersistenceManager.getObjectsById(*)
Sure, I can. I've also noticed that calling PersistenceManager.getObjectById(Object), as opposed to PersistenceManager.getObjectById(Class?, Object), behaves funny as well. Let me get an exception trace when I run my unit tests again and add that to an issue. On Fri, Jan 15, 2010 at 12:19 PM, Ikai L (Google) ika...@google.com wrote: Can you file an issue in the issue tracker? http://code.google.com/p/googleappengine/issues/list http://code.google.com/p/googleappengine/issues/list On Fri, Jan 15, 2010 at 12:01 PM, Patrick Twohig patr...@namazustudios.com wrote: *Mendokusai.* I'm moving over to the low-level API. On Fri, Jan 15, 2010 at 11:25 AM, Erem ehb...@gmail.com wrote: Bump. Particularly on the future plans to implement. On Jan 14, 12:31 pm, Patrick Twohig patr...@namazustudios.com wrote: Why doesn't the family of methods PersistenceManager.getObjectsById actually perform a bulk fetch from the datastore? Are there future plans to implement it as a bulk fetch? -- Patrick H. Twohig. Namazu Studios P.O. Box 34161 San Diego, CA 92163-4161 Office: 619.862.2890 x100 Cell: 619.453.5075 Twitter: @svm_invictvs IRC: svm_invic...@irc.freenode.net ##java, #android-dev, #iphonedev, #appengine http://www.namazustudios.com/ This communication, and any attachments, shall be considered confidential and proprietary information of Namazu Studios LLC. This message, and attachments, are intended for the listed recipients only. If you are not one of the intended recipients, please destroy all copies of this communication. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.comgoogle-appengine%2bunsubscr...@googlegroups.com . For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Patrick H. Twohig. Namazu Studios P.O. Box 34161 San Diego, CA 92163-4161 Office: 619.862.2890 x100 Cell: 619.453.5075 Twitter: @svm_invictvs IRC: svm_invic...@irc.freenode.net ##java, #android-dev, #iphonedev, #appengine http://www.namazustudios.com/ This communication, and any attachments, shall be considered confidential and proprietary information of Namazu Studios LLC. This message, and attachments, are intended for the listed recipients only. If you are not one of the intended recipients, please destroy all copies of this communication. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.comgoogle-appengine%2bunsubscr...@googlegroups.com . For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Ikai Lan Developer Programs Engineer, Google App Engine -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.comgoogle-appengine%2bunsubscr...@googlegroups.com . For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Patrick H. Twohig. Namazu Studios P.O. Box 34161 San Diego, CA 92163-4161 Office: 619.862.2890 x100 Cell: 619.453.5075 Twitter: @svm_invictvs IRC: svm_invic...@irc.freenode.net ##java, #android-dev, #iphonedev, #appengine http://www.namazustudios.com/ This communication, and any attachments, shall be considered confidential and proprietary information of Namazu Studios LLC. This message, and attachments, are intended for the listed recipients only. If you are not one of the intended recipients, please destroy all copies of this communication. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Bulk Updates
Ah yeah, deleting in bulk in a cron job or ongoing task would definitely work. I dunno why that didn't dawn on me earlier. Thanks Nick! On Thu, Jan 14, 2010 at 12:21 AM, Nick Johnson (Google) nick.john...@google.com wrote: Hi, On Thu, Jan 14, 2010 at 6:07 AM, Patrick Twohig patr...@namazustudios.com wrote: Perhaps I didn't explain my use case entirely well enough. Basically it's a leaderboard (high scores table) for an online game. After thinking a little more GAEly I came up with the following solution. I was curious if anybody had any comments or critiques for my approach. Create a WeeklyLeaderboardEntry (or similarly named kind). That basically contains the player's score and the player's unique name. And use it as follows: Make some code that establishes a sequence of weeks. Let's say the number of weeks since the Epoch. For instance if it's January 3, 1970, the current week is 0. If it's January 8, 1970 it's week 1 and so on. When player finishes game: 1) Look for WeeklyLeaderboardEntry with the current. If it exists, skip to step 4. (Maybe name the entitye PlayerName-Sequence). 2) Create WeeklyLeaderboardEntry for the current week. 3) Queue the entity to be deleted a week from now. 4) Set the player's current high score to that entity. 5) Persist object to be updated later in case the player plays another game When displaying the leaderboard: 1) Query for WeeklyLeaderboardEntries 2) Filter out ones not pertaining to the current week 3) Sort highest score to lowest This way the following holds true: - When the user is updating his/her high score he/she is only working a single entity. I never need to do bulk updates in a cron job or anything. - I don't need to write a cron job to go through a whole mess of leaderboard entities trying to reset the counter for everbody at once. - When the week rolls over it appears to happen atomically because I just restrict the query to the current week. There may be stale entry objects but those will get deleted at he app engine's leisure. My only questions/concerns: - Is it okay to queue something to be deleted after a week? Will the task queue accept a timeout for that length of time? Timeouts can be for up to a month. Queueing a task per entry is somewhat inefficient, however - you'd be better running a series of tasks at the end of the week that query for old entries and delete them in batches. - - Will the task queue keep up with clearing out stale entities as fast as they can be created That depends on the rate of creation. Batch operations will allow you to sustain a much higher rate with fewer resources. -Nick Johnson - ? On Wed, Jan 13, 2010 at 7:28 PM, Eric Ka Ka Ng ngk...@gmail.com wrote: how about batch update using db.put()? following is extracted from http://googleappengine.blogspot.com/2009/06/10-things-you-probably-didnt-know-about.html For example, take a look at this common pattern: for entity in MyModel.all().filter(color =, old_favorite).fetch(100): entity.color = new_favorite entity.put() Doing the update this way requires one datastore round trip for the query, plus one additional round trip for each updated entity - for a total of up to 101 round trips! In comparison, take a look at this example: updated = [] for entity in MyModel.all().filter(color =, old_favorite).fetch(100): entity.color = new_favorite updated.append(entity) db.put(updated) By adding two lines, we've reduced the number of round trips required from 101 to just 2! - eric 2010/1/14 Patrick Twohig patr...@namazustudios.com: So I'm looking at trying to reset large amounts of data en masse. Say I want to reset a counter on every account each week, how would I go about implementing something like that? Would I have to go through each object, update it, and store it? If that's the case, how do I go about doing so without killing my quota or running up my usage extremely high? Thanks, Patrick. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com . To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.comgoogle-appengine%2bunsubscr...@googlegroups.com . For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.comgoogle-appengine%2bunsubscr...@googlegroups.com . For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Patrick H. Twohig. Namazu Studios P.O. Box
[google-appengine] PersistenceManager.getObjectsById(*)
Why doesn't the family of methods PersistenceManager.getObjectsById actually perform a bulk fetch from the datastore? Are there future plans to implement it as a bulk fetch? -- Patrick H. Twohig. Namazu Studios P.O. Box 34161 San Diego, CA 92163-4161 Office: 619.862.2890 x100 Cell: 619.453.5075 Twitter: @svm_invictvs IRC: svm_invic...@irc.freenode.net ##java, #android-dev, #iphonedev, #appengine http://www.namazustudios.com/ This communication, and any attachments, shall be considered confidential and proprietary information of Namazu Studios LLC. This message, and attachments, are intended for the listed recipients only. If you are not one of the intended recipients, please destroy all copies of this communication. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] ImagesService and Alpha blending.
I have several images that I want to composite together. Each image has a completely transparent background, however when I use the ImagesSErvice to composite them, I end up with the last drawn image on top with a white background. It apepars that the ImagesService doesn't support alpha blending? Is there a workaround for it? -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Bulk Updates
So I'm looking at trying to reset large amounts of data *en masse*. Say I want to reset a counter on every account each week, how would I go about implementing something like that? Would I have to go through each object, update it, and store it? If that's the case, how do I go about doing so without killing my quota or running up my usage extremely high? Thanks, Patrick. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Bulk Updates
Perhaps I didn't explain my use case entirely well enough. Basically it's a leaderboard (high scores table) for an online game. After thinking a little more GAEly I came up with the following solution. I was curious if anybody had any comments or critiques for my approach. Create a WeeklyLeaderboardEntry (or similarly named kind). That basically contains the player's score and the player's unique name. And use it as follows: Make some code that establishes a sequence of weeks. Let's say the number of weeks since the Epoch. For instance if it's January 3, 1970, the current week is 0. If it's January 8, 1970 it's week 1 and so on. When player finishes game: 1) Look for WeeklyLeaderboardEntry with the current. If it exists, skip to step 4. (Maybe name the entitye PlayerName-Sequence). 2) Create WeeklyLeaderboardEntry for the current week. 3) Queue the entity to be deleted a week from now. 4) Set the player's current high score to that entity. 5) Persist object to be updated later in case the player plays another game When displaying the leaderboard: 1) Query for WeeklyLeaderboardEntries 2) Filter out ones not pertaining to the current week 3) Sort highest score to lowest This way the following holds true: - When the user is updating his/her high score he/she is only working a single entity. I never need to do bulk updates in a cron job or anything. - I don't need to write a cron job to go through a whole mess of leaderboard entities trying to reset the counter for everbody at once. - When the week rolls over it appears to happen atomically because I just restrict the query to the current week. There may be stale entry objects but those will get deleted at he app engine's leisure. My only questions/concerns: - Is it okay to queue something to be deleted after a week? Will the task queue accept a timeout for that length of time? - Will the task queue keep up with clearing out stale entities as fast as they can be created? On Wed, Jan 13, 2010 at 7:28 PM, Eric Ka Ka Ng ngk...@gmail.com wrote: how about batch update using db.put()? following is extracted from http://googleappengine.blogspot.com/2009/06/10-things-you-probably-didnt-know-about.html For example, take a look at this common pattern: for entity in MyModel.all().filter(color =, old_favorite).fetch(100): entity.color = new_favorite entity.put() Doing the update this way requires one datastore round trip for the query, plus one additional round trip for each updated entity - for a total of up to 101 round trips! In comparison, take a look at this example: updated = [] for entity in MyModel.all().filter(color =, old_favorite).fetch(100): entity.color = new_favorite updated.append(entity) db.put(updated) By adding two lines, we've reduced the number of round trips required from 101 to just 2! - eric 2010/1/14 Patrick Twohig patr...@namazustudios.com: So I'm looking at trying to reset large amounts of data en masse. Say I want to reset a counter on every account each week, how would I go about implementing something like that? Would I have to go through each object, update it, and store it? If that's the case, how do I go about doing so without killing my quota or running up my usage extremely high? Thanks, Patrick. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.comgoogle-appengine%2bunsubscr...@googlegroups.com . For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.comgoogle-appengine%2bunsubscr...@googlegroups.com . For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Patrick H. Twohig. Namazu Studios P.O. Box 34161 San Diego, CA 92163-4161 Office: 619.862.2890 x100 Cell: 619.453.5075 Twitter: @svm_invictvs IRC: svm_invic...@irc.freenode.net ##java, #android-dev, #iphonedev, #appengine http://www.namazustudios.com/ This communication, and any attachments, shall be considered confidential and proprietary information of Namazu Studios LLC. This message, and attachments, are intended for the listed recipients only. If you are not one of the intended recipients, please destroy all copies of this communication. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email
Re: [google-appengine] Re: This App Engine is a JOKE
To the original poster: Get Pissed. It helps. In all seriousness, though. I think that GAE has its fair share of limitations, but has a lot of potential. Like everything else, I think it best you read the documentation, do a little prototyping and assess weather or not it meets your needs. If you still think it's a JOKE then, nobody is holding a gun to your head forcing you to use it. If you are stuck using it and it doesn't meet your use cases, then I'm sorry but flaming the mailing list isn't going to get you anywhere. On Tue, Jan 12, 2010 at 4:47 PM, nickmilon nickmi...@gmail.com wrote: This GAE is a joke keeps coming in the group from time to time so it is a joke by itself by now. For me it is a wonderful platform, provided you know what are you doing, do not forget it is a tool you got to know how to use, just optimize your code for it and it delivers for you. Contrary to what Mr Locke's thinking Gae is THE DEFINITION of cloud computing. I know there exists an early definition of cloud computing based on just economic terms, cloud is more about technology than economy besides, yes you pay more you get more but who on earth told you that are not things on earth that money couldn't buy ? And what is 500 request/ second [if you want more ask for more?] is telling you ? and datastore 1 gigabyte free; no maximum ! etc.. etc. isn't it the definition of scalability ? who is going to wait for your page for more than 30 seconds ? All this Email talk reminds me of other conspiracy theories, could you give me a reasonable answer why Google would bother to look after to delay an Email if it is not going to a gmail recipient ? As for Thanasis ... I hope Google will not mesh up with php and all the trouble involved, and by the way... IF your coding looks like your English in your postings I am sure you will have trouble with GAE or any other platform for that matter - Prablem related to stopservice() ethod of Service class (calm down just a may be hot remark from a compatriot trying to help - no hard feelings) Happy coding ! Nick ( gaengine.blogspot.com ) ___ nickmi...@gmail.com On Jan 13, 12:40 am, Thanasis t.deleni...@gmail.com wrote: Maybe (a big maybe) if Google introduces some other lighter language (like PHP(?)), we could stand a chance. But now (the Java GAE version is what I am playing with), GAE seem so massively oversubscribed that it makes the worst web hosting industry's oversellers to look like paradise. I would expect *major* pricing increase in order to bring Java support up to a manageable level and decent performance. A $5 monthly fee (as proposed on the tracker) in order to keep a JVM is not going to happen - I don't think that this is technically possible, due to Java runtime constraints. A $30 as an absolute minimum would not surprise me at all. After all, not all of us need super scaling capabilities - which are also missing as you correctly state in your post). On Jan 12, 11:18 pm, Locke locke2...@gmail.com wrote: This isn't cloud computing unless they let our apps scale with our bank accounts. But still, I'll stick with Google in hopes that their hosting is ready for the prime-time before my app is ready for the prime-time. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.comgoogle-appengine%2bunsubscr...@googlegroups.com . For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] PersistenceManager multiple instances
I'm having some difficulty with the PersistenceManager class. When running my app locally, it appears I can have multiple PersistenceManagers open at once operating on different entity groups at the same time. However, when I try to do this in a JUnit test, I end up getting all sorts of exceptions. It stands to reason that it should be okay opening up multiple PersistenceManagers, but my unit tests beg to differ. In this case, what's wrong, my code or the unit test? Would it be better, too, if I provided a testcase as to what's going on? Thanks in advance. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] PersistenceManagerFactory
When running unit tests, should I close the PersistenceManagerFactory between tests? -- Patrick H. Twohig. Namazu Studios P.O. Box 34161 San Diego, CA 92163-4161 Office: 619.862.2890 x100 Cell: 619.453.5075 Twitter: @svm_invictvs IRC: svm_invic...@irc.freenode.net ##java, #android-dev, #iphonedev, #appengine http://www.namazustudios.com/ This communication, and any attachments, shall be considered confidential and proprietary information of Namazu Studios LLC. This message, and attachments, are intended for the listed recipients only. If you are not one of the intended recipients, please destroy all copies of this communication. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Using MySQL
Whoever was asking about HIPPA compliance, that may be a way to stay in compliance. If you put your actual sensitive records in an encrypted database on your own servers it'd probably be okay. I seem to recall when I worked for a medical company, that transmission and storage of patient names needed be encrypted with 128 bit (or stronger) encryption. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: Billing zero but shouldn't be.
Maybe it was a holiday special :) On Thu, Jan 7, 2010 at 10:26 AM, Amir Michail amich...@gmail.com wrote: On Jan 7, 12:02 pm, Prashant Gupta nextprash...@gmail.com wrote: are you sure that you are crossing free quota limit? Yes, I'm sure this happened on a few days. Amir 2010/1/7 Amir Michail amich...@gmail.com On Jan 5, 2:16 pm, Amir Michail amich...@gmail.com wrote: Hello, Why am I not billed although I should be? Could someone look into this please? Amir Amir -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com . To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.comgoogle-appengine%2bunsubscr...@googlegroups.comgoogle-appengine%2Bunsubscrib e...@googlegroups.com . For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.comgoogle-appengine%2bunsubscr...@googlegroups.com . For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Patrick H. Twohig. Namazu Studios P.O. Box 34161 San Diego, CA 92163-4161 Office: 619.862.2890 x100 Cell: 619.453.5075 Twitter: @svm_invictvs IRC: svm_invic...@irc.freenode.net ##java, #android-dev, #iphonedev, #appengine http://www.namazustudios.com/ This communication, and any attachments, shall be considered confidential and proprietary information of Namazu Studios LLC. This message, and attachments, are intended for the listed recipients only. If you are not one of the intended recipients, please destroy all copies of this communication. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] About Transactions
So, I know that transactions are restricted to a single entity group. I was curious if in the future this restriction may be relaxed. If not, what other common strategies are in place to ensure that concurrent updates to multiple entity groups are kept consistent? -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Slow query [java]
My guess is that the first time you're running it the classes required to run the query haven't been loaded yet by the classloader. On Wed, Jan 6, 2010 at 12:54 PM, Amir Michail amich...@gmail.com wrote: Hello, I'm getting slow query performance: 5848ms 8362cpu_ms 40api_cpu_ms Query: String query = select from + DemoScore.class.getName() + order by score desc; This is returning three objects. If the query is repeated, it becomes very fast. What could be wrong? Amir -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.comgoogle-appengine%2bunsubscr...@googlegroups.com . For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Patrick H. Twohig. Namazu Studios P.O. Box 34161 San Diego, CA 92163-4161 Office: 619.862.2890 x100 Cell: 619.453.5075 Twitter: @svm_invictvs IRC: svm_invic...@irc.freenode.net ##java, #android-dev, #iphonedev, #appengine http://www.namazustudios.com/ This communication, and any attachments, shall be considered confidential and proprietary information of Namazu Studios LLC. This message, and attachments, are intended for the listed recipients only. If you are not one of the intended recipients, please destroy all copies of this communication. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] About Transactions
I read that and I've put that technique into practice in my own code in a few spots. I was just curious if there were plans to relax the restriction on transactions. I'm guessing that due to the complexity of the distributed datastore, such a thin isn't possible. -- Patrick H. Twohig. Namazu Studios P.O. Box 34161 San Diego, CA 92163-4161 Office: 619.862.2890 x100 Cell: 619.453.5075 Twitter: @svm_invictvs IRC: svm_invic...@irc.freenode.net ##java, #android-dev, #iphonedev, #appengine http://www.namazustudios.com/ This communication, and any attachments, shall be considered confidential and proprietary information of Namazu Studios LLC. This message, and attachments, are intended for the listed recipients only. If you are not one of the intended recipients, please destroy all copies of this communication. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
