[google-appengine] most efficient bulk delete code?
Is this the most efficient code for bulk delete?
Is there a "SELECT __key__" technique that's faster?
This currently deletes 100 at a time, but it's slow.
public long deleteBefore(Date d, long maxRecords)
throws DaoException {
PersistenceManager pm = pmf.getPersistenceManager();
Query query = pm.newQuery(UploadedContent.class);
query.setFilter("dateCreated < dateCreatedParam");
query.declareParameters("Date dateCreatedParam");
query.declareImports("import java.util.Date;");
if (maxRecords >= 0) {
query.setRange(0, maxRecords);
}
List entities = new
ArrayList();
try {
entities = (List) query.execute(d);
pm.deletePersistentAll(entities);
} catch (Exception e) {
log.log(Level.SEVERE, e.getMessage(), e);
throw new DaoException(e);
} finally {
query.closeAll();
pm.close();
}
return entities.size();
}
--
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: Stackoverflow
I created a quick poll for tracking the community feeling about this. https://spreadsheets.google.com/spreadsheet/viewform?formkey=dEFERTJWTTJxZEZ1dU91VXFvUFpzVnc6MQ Feel free to keep the discussion going in this thread. On Tue, Aug 2, 2011 at 8:56 AM, MLTrim wrote: > StackOverflow is perfect to handle technical questions but the other > types of discussion do not fit very well and are not well accepted on > SO. > Speaking about Android, the tag on SO is pretty florid and crowded but > with some quality problem (http://meta.stackoverflow.com/questions/ > 100529/help-us-clean-up-the-android-tag) . > > michele > [systempuntoout] > > On Aug 2, 5:35 am, Robert Kluin wrote: >> I think the two medias are good for different types of questions. >> Personally, I think stackoverflow is not as good for questions needing >> or benefitting from discussion. >> >> On Mon, Aug 1, 2011 at 10:03, Vivek Puri wrote: >> > Is it possible to migrate AppEngine group to Stackoverflow or a white >> > label solution -http://stackexchange.com/. It is becoming impossible >> > to search anything meaningful and identify top threads in the current >> > form of the group. Any good solutions provided are buried quickly >> > below getting stated kind of questions or daily issue related >> > questions. Also, any important announcements made by AppEngine team >> > are hard to locate and best answers to questions are hard to validate. >> >> > -- >> > You received this message because you are subscribed to the Google Groups >> > "Google App Engine" group. >> > To post to this group, send email to google-appengine@googlegroups.com. >> > To unsubscribe from this group, send email to >> > google-appengine+unsubscr...@googlegroups.com. >> > For more options, visit this group >> > athttp://groups.google.com/group/google-appengine?hl=en. > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to google-appengine@googlegroups.com. > To unsubscribe from this group, send email to > google-appengine+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > > -- Johan Euphrosine (proppy) Developer Programs Engineer Google Developer Relations -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: Stackoverflow
+1 Objective questions have reduced usefulness here, as there's more noise and they get lost with time. SO's peer review forces people to stay on topic, thus reducing noise and making questions more "reusable". The group is better for less objective discussions, of course. -- rodrigo -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/KkBHR99piiwJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Google App Engine for commercial purposes.
Hello, Does anyone know if I can deploy an application in my Windows 2003 Server using GAE for commercial purposes? Thank you in advance, -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: Unserializing large Blobs from the Blobstore
Hi Luca, did you ever figure this out? -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/L2H3-B0iq-kJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Username/Password in Eclipse Plugin
Hi, I'm using the Eclipse plugin for GAE java development. Unfortunately, my deployment directly from the plugin fails and I'm strongly suspecting that I entered a wrong username/password at first try. Is there a possibility to change this username/password? When I click deploy, there is no username/password field anymore, I assume due to caching of the credentials? Can anybody give me a hint for this? thx & best regards, Matthias -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: Re: [google-appengine] Re: App Engine Chat Time!
Hi Brett, I have just finished a Facebook App built on Google App Engine and am currently engaged with another client to build their Facebook App. I would be happy to take a look at your requirements and discuss the project in more detail. You can contact me at [my first name] at [emotionai.com] Regards, Ian -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/z1G0y09wlOsJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Google App Engine for commercial purposes.
you can but it will not work. try typhoonae ro appscale. On Mon, Aug 1, 2011 at 11:19 PM, jhatzics wrote: > Hello, > > Does anyone know if I can deploy an application in my Windows 2003 > Server using GAE for commercial purposes? > > Thank you in advance, > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to google-appengine@googlegroups.com. > To unsubscribe from this group, send email to > google-appengine+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Username/Password in Eclipse Plugin
Hi, On 08/02/2011 10:19 AM, Matthias Luft wrote: > I'm using the Eclipse plugin for GAE java development. Unfortunately, my > deployment directly from the plugin fails and I'm strongly suspecting > that I entered a wrong username/password at first try. Is there a > possibility to change this username/password? When I click deploy, there > is no username/password field anymore, I assume due to caching of the > credentials? > > Can anybody give me a hint for this? And of course, after sending the mail I found the small icon in the bottom left corner, after searching through the preferences for a long time. thx anyway & sorry for the noise, Matthias -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: Unserializing large Blobs from the Blobstore
Hi Matt, There are some techniques for using as little memory as possible when serializing those large XML files, but you'll most likely go over the limit. Backends solve this issue. I think they need a little improvement since they require a little of sys-admin work to setup... but they're meant to fix this specific issue. Luca On Mon, Aug 1, 2011 at 9:37 PM, Matt Myers wrote: > Hi Luca, did you ever figure this out? > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To view this discussion on the web visit > https://groups.google.com/d/msg/google-appengine/-/L2H3-B0iq-kJ. > To post to this group, send email to google-appengine@googlegroups.com. > To unsubscribe from this group, send email to > google-appengine+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: Crazy number of Deadine Exceeded exceptions recently
Hi, I've got a similar issue: lots of DeadlineExceeded errors since a few weeks. I'm on the master-slave datastore too, but what I'm reporting happened again one hour ago. These errors happen in bursts, and I recently realized that it was in fact shutting down ALL instances of the application. (In the logs, I also have this warning: A serious problem was encountered with the process that handled this request, causing it to exit. This is likely to cause a new process to be used for the next request to your application. If you see this message frequently, you may be throwing exceptions during the initialization of your application. (Error code 104)) This does not happen when an instance is spinning up but after several hours. The trace I get along with the DeadlineExceeded errors show that it happens in the second phase: while the app is trying to fallback gracefully because of an other error (that does not appears in logs). Request reported processing time can be like this: ms=100878 cpu_ms=385 api_cpu_ms=58 cpm_usd=0.010945 pending_ms=9321 Here is a screenshot of the admin page, showing that all instances have been shut down about 7 minutes ago, even resident ones: http://dl.dropbox.com/u/497622/spinDown.png The app do work in batches (although not always small ones). But request processing time is usually good enough (see average latency on the screen shot). I'm trying things on my testing applications to see what can be wrong but it's still not clear for me and I'm running short of ideas... Any suggestions? On 2 août, 06:21, Robert Kluin wrote: > Hi Will, > I assume this is on the master-slave datastore? I think there were > a number of large latency spikes in both the datastore and serving > last week. > > Some things to try: > - do work in smaller batches. > - if you're doing work serially, do it in batches. > - use async interfaces to do work in batches, but in parallel using async. > > http://code.google.com/appengine/docs/python/datastore/async.html > > Robert > > > > > > > > On Fri, Jul 29, 2011 at 18:35, Will Reiher wrote: > > I'm trying to debug this issue but I keep hitting a wall. > > I keep trying new things on one of my deployments to see if I an get the > > number of errors down but nothing seems to help. It all started in the last > > week or so ago. I also have some existing deployments that I have not > > changed and are seeing these same errors while the code was never changed > > and had been stable. > > 1. This is is happening on existing code that has not changed recently > > 2. The DeadlineExceededErrors are coming up randomly and at different points > > in the code. > > 3. Latency is pretty high and app engine seems to be spawning a lot of new > > instances beyond my 3 included ones. > > > -- > > You received this message because you are subscribed to the Google Groups > > "Google App Engine" group. > > To view this discussion on the web visit > >https://groups.google.com/d/msg/google-appengine/-/g_C4iPzPeo4J. > > To post to this group, send email to google-appengine@googlegroups.com. > > To unsubscribe from this group, send email to > > google-appengine+unsubscr...@googlegroups.com. > > For more options, visit this group at > >http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Data Center (CDN) Locations
I have been searching around but have not seen information on where App Engine data centers are located and how loads are spread? So I have two questions: - Where are the App Engine data centers? (i.e. West Coast, East Coast, Europe...) - What criteria are used to serve an app from one data center over another? We have an application we are building for a customer in Japan and would like to know if there is an App Engine data center in Asia so we will have lower latency. Thanks. Ian -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/atIjL0tAAAYJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: Data Center (CDN) Locations
Hi, As far as I'm aware, the most accurate information the Google staff have divulged is that all data centres are located in North America somewhere. Cheers, Simon -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/OvHwAZFC9aYJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: Stackoverflow
I was forced to vote for Groups over Stackoverflow. My real vote is both. Stackoverflow is ok for a focused question that does not get booted Groups are better if you have a question that does not fit into the criteria for Stackoverflow or you are more having a conversation. I do a lot with Google's App Inventor groups. We looked a bit at Stackoverflow a while ago and it did not fit. we get a lot more less technical types there... and I have failed to check my Stackoverflow account in a while. It was not encouraging to me to see how much karma I would have to build in a place with limited use to me to add the pages or whatever they call them to get the App Inventor info there to where it should be to be useful. I wanted to be able to point back to the Google forums and those messages got crushed by the SO cops... and your milage may differ :-) Gary -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/r33pUmrjRIkJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] "Enforced Rate" in task queues ruining performance
I am having the same problem. Today has been the worst performance so far with the enforced rate down at 0.16/s. The application is basically non functional at this rate Could we please get some information on this. - Dan -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/6cczXBYIK6IJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. <>
[google-appengine] Connect from GAE to external PostgreSQL DB?
Hi All, For an app I need to obtain data from an external PostgreSQL database, do some calculations on it and then show the results to the clients in an MVP set-up. However, from some posts I gather that GAE cannot connect with a PostgreSQL DB over the internet out of its servlet context. Others suggest setting up an 'http proxy' for this. I'm quite a newbie on this, however, I would love to use GAE / GWT for the most part (particularly GTWP). Anyone that has a suggestion how I can get to the data on the Postrgres DB out of my GAE servlet context?? Thanks a lot! -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] "Enforced Rate" in task queues ruining performance
Can you both share your appid, so we can investigate deeper ? Also feel free to fill a production issue about it if this is directly affecting your operation: http://code.google.com/p/googleappengine/issues/entry?template=Production%20issue On Tue, Aug 2, 2011 at 3:08 PM, Dan Lines wrote: > I am having the same problem. Today has been the worst performance so far > with the enforced rate down at 0.16/s. The application is basically non > functional at this rate > > Could we please get some information on this. > > - Dan > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To view this discussion on the web visit > https://groups.google.com/d/msg/google-appengine/-/6cczXBYIK6IJ. > To post to this group, send email to google-appengine@googlegroups.com. > To unsubscribe from this group, send email to > google-appengine+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > -- Johan Euphrosine (proppy) Developer Programs Engineer Google Developer Relations -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: How do I do this query in JDO on server?
You should use Query filters, see the following page for more information:
http://code.google.com/appengine/docs/java/datastore/queries.html#Restrictions_on_Queries
Hope that helps.
On Sun, Jul 31, 2011 at 11:35 AM, SergiBC wrote:
> any help?
>
> On 26 jul, 21:11, SergiBC wrote:
>> Hello,
>>
>> I do this query in "Datastore viewer" --> select from " +
>> MyClass.class.getName() + " WHERE id >= KEY('MyClass', 415011)
>> and it works.
>>
>> But how do I do this in java?
>>
>> Thank you
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.
>
>
--
Johan Euphrosine (proppy)
Developer Programs Engineer
Google Developer Relations
--
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: Java: Precompilation fails.
Can you two share your application id, so we can investigate the issue ? Feel free to fill a production issue if this is directly affecting your operation: http://code.google.com/p/googleappengine/issues/entry?template=Production%20issue On Sun, Jul 31, 2011 at 9:45 PM, Pavel Kaplin wrote: > Experiencing same issue > > On Jul 24, 2:38 am, David wrote: >> Hello, >> >> I get the following error: >> >> Uploading 0 files. >> Initializing precompilation... >> Precompiling... 13 file(s) left. >> Precompiling... 13 file(s) left. >> Precompiling... 13 file(s) left. >> Precompiling... 13 file(s) left. >> on backend null. >> java.io.IOException: Precompilation failed with 13 file(s) remaining. >> Consider adding false >> to your appengine-web.xml and trying again. >> >> Debugging information may be found in C:\[...]\appengine- >> deploy738810777136767626.log >> >> Said debug information basically tells me that an error 503 occured >> (full log attached on bottom). On the first attempt it failed when >> only 1 file was left. Any idea what to do? Is there any way to find >> out what caused that Error 503 in the precompile-URL? Is that >> temporary or permanent? >> >> I only found little information on that error so I assume it does not >> occur too often. Any help is appreciated. >> >> Regards, >> David >> >> -- local log-file --- >> Unable to update: >> java.io.IOException: Precompilation failed with 13 file(s) remaining. >> Consider adding false >> to your appengine-web.xml and trying again. >> at >> com.google.appengine.tools.admin.AppVersionUpload.precompile(AppVersionUplo >> ad.java: >> 259) >> at >> com.google.appengine.tools.admin.AppVersionUpload.doUpload(AppVersionUpload >> .java: >> 152) >> at >> com.google.appengine.tools.admin.AppAdminImpl.doUpdate(AppAdminImpl.java: >> 284) >> at >> com.google.appengine.tools.admin.AppAdminImpl.update(AppAdminImpl.java: >> 48) >> at >> com.google.appengine.eclipse.core.proxy.AppEngineBridgeImpl.deploy(AppEngin >> eBridgeImpl.java: >> 265) >> at >> com.google.appengine.eclipse.core.deploy.DeployProjectJob.runInWorkspace(De >> ployProjectJob.java: >> 144) >> at >> org.eclipse.core.internal.resources.InternalWorkspaceJob.run(InternalWorksp >> aceJob.java: >> 38) >> at org.eclipse.core.internal.jobs.Worker.run(Worker.java:54) >> Caused by: java.io.IOException: Error posting to >> URL:https://appengine.google.com/api/appversion/precompile?app_id=[...]&version=1&; >> 503 Service Unavailable >> >> Try Again (503) >> An unexpected failure has occurred. Please try again. >> >> at >> com.google.appengine.tools.admin.AbstractServerConnection.send(AbstractServ >> erConnection.java: >> 245) >> at >> com.google.appengine.tools.admin.AbstractServerConnection.post(AbstractServ >> erConnection.java: >> 207) >> at >> com.google.appengine.tools.admin.AppVersionUpload.send(AppVersionUpload.jav >> a: >> 639) >> at >> com.google.appengine.tools.admin.AppVersionUpload.sendPrecompileRequest(App >> VersionUpload.java: >> 305) >> at >> com.google.appengine.tools.admin.AppVersionUpload.precompileChunk(AppVersio >> nUpload.java: >> 297) >> at >> com.google.appengine.tools.admin.AppVersionUpload.precompile(AppVersionUplo >> ad.java: >> 232) >> ... 7 more > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to google-appengine@googlegroups.com. > To unsubscribe from this group, send email to > google-appengine+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > > -- Johan Euphrosine (proppy) Developer Programs Engineer Google Developer Relations -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Can't download all data with bulkloader from non-default namespace
Can you share your application id, so we can investigate the issue ? Feel free to fill a production issue if this is directly affecting your operation: http://code.google.com/p/googleappengine/issues/entry?template=Production%20issue On Sat, Jul 30, 2011 at 10:19 AM, rekby wrote: > I use Python SDK 1.5.2. High-replication datastore. I see data statistic for > namespace 2 in dashboard. > appcfg.py download_data --application=s~my-optimiser > --url=https://3.my-optimiser.appspot.com/remote_api --namespace="2" > --filename=optimiser.data > Downloading data records. > [INFO ] Logging to bulkloader-log-20110730.121721 > [INFO ] Throttling transfers: > [INFO ] Bandwidth: 25 bytes/second > [INFO ] HTTP connections: 8/second > [INFO ] Entities inserted/fetched/modified: 20/second > [INFO ] Batch Size: 10 > [INFO ] Opening database: bulkloader-progress-20110730.121721.sql3 > [INFO ] Opening database: bulkloader-results-20110730.121721.sql3 > [INFO ] Connecting to 3.my-optimiser.appspot.com/remote_api > [ERROR ] Unable to download kind stats for all-kinds download. > [ERROR ] Kind stats are generated periodically by the appserver > [ERROR ] Kind stats are not available on dev_appserver. > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To view this discussion on the web visit > https://groups.google.com/d/msg/google-appengine/-/fN3ISosu8ZgJ. > To post to this group, send email to google-appengine@googlegroups.com. > To unsubscribe from this group, send email to > google-appengine+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > -- Johan Euphrosine (proppy) Developer Programs Engineer Google Developer Relations -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Can't download all data with bulkloader from non-default namespace
Yes, I can. My app ID: my-optimiser (my-optimiser.appspot.com). It isn't very need personal for me now - I have only one Entity type and I can download it with --kind option, but I think it is bug in platform or SDK. Now I try use https://my-optimiser.appspot.com instead of https://3.my-optimiser.appspot.com - it have same error. 2011/8/2 Johan Euphrosine > Can you share your application id, so we can investigate the issue ? > > Feel free to fill a production issue if this is directly affecting > your operation: > > http://code.google.com/p/googleappengine/issues/entry?template=Production%20issue > > On Sat, Jul 30, 2011 at 10:19 AM, rekby wrote: > > I use Python SDK 1.5.2. High-replication datastore. I see data statistic > for > > namespace 2 in dashboard. > > appcfg.py download_data --application=s~my-optimiser > > --url=https://3.my-optimiser.appspot.com/remote_api --namespace="2" > > --filename=optimiser.data > > Downloading data records. > > [INFO] Logging to bulkloader-log-20110730.121721 > > [INFO] Throttling transfers: > > [INFO] Bandwidth: 25 bytes/second > > [INFO] HTTP connections: 8/second > > [INFO] Entities inserted/fetched/modified: 20/second > > [INFO] Batch Size: 10 > > [INFO] Opening database: bulkloader-progress-20110730.121721.sql3 > > [INFO] Opening database: bulkloader-results-20110730.121721.sql3 > > [INFO] Connecting to 3.my-optimiser.appspot.com/remote_api > > [ERROR ] Unable to download kind stats for all-kinds download. > > [ERROR ] Kind stats are generated periodically by the appserver > > [ERROR ] Kind stats are not available on dev_appserver. > > > > -- > > You received this message because you are subscribed to the Google Groups > > "Google App Engine" group. > > To view this discussion on the web visit > > https://groups.google.com/d/msg/google-appengine/-/fN3ISosu8ZgJ. > > To post to this group, send email to google-appengine@googlegroups.com. > > To unsubscribe from this group, send email to > > google-appengine+unsubscr...@googlegroups.com. > > For more options, visit this group at > > http://groups.google.com/group/google-appengine?hl=en. > > > > > > -- > Johan Euphrosine (proppy) > Developer Programs Engineer > Google Developer Relations > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to google-appengine@googlegroups.com. > To unsubscribe from this group, send email to > google-appengine+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > > -- С уважением, Кулин Тимофей. Телефон: +7 (4852) 974793 ICQ: 114902104 email: timo...@koolin.ru Blog: http://timofey.koolin.ru -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Increase apps size restriction above 10 megabytes?
Did someone fill a doc bug for this ? Thanks in advance. On Sun, Jul 31, 2011 at 7:30 PM, Robert Kluin wrote: > I don't think that is correct, at least not according to some of the docs: > > - maximum size of an application file: 10 megabytes > - maximum size of a static file: 10 megabytes > - maximum total size of all application and static files: 150 megabytes > > But that is exactly my point, I think there are discrepancies in the docs. > > > > > On Sun, Jul 31, 2011 at 02:23, Brandon Wirtz wrote: >> 10 for app, 150 for static assets. >> >> >> >> From: google-appengine@googlegroups.com >> [mailto:google-appengine@googlegroups.com] On Behalf Of Robert Kluin >> Sent: Saturday, July 30, 2011 11:10 PM >> To: google-appengine@googlegroups.com >> Subject: Re: [google-appengine] Increase apps size restriction above 10 >> megabytes? >> >> >> >> I think there are some descrepancies in the docs about the max application >> size. In some places the max app size is listed as 10mb, others 150mb (Max >> file size of 10mb). >> >> http://code.google.com/appengine/docs/java/runtime.html#Quotas_and_Limits >> http://code.google.com/appengine/docs/python/runtime.html#Quotas_and_Limits >> >> http://code.google.com/appengine/docs/quotas.html#Deployments >> >> I'm pretty sure there are others that list 10mb too. >> >> Robert >> >> On Jul 29, 2011 1:56 PM, "Ikai Lan (Google)" wrote: >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Google App Engine" group. >> To post to this group, send email to google-appengine@googlegroups.com. >> To unsubscribe from this group, send email to >> google-appengine+unsubscr...@googlegroups.com. >> For more options, visit this group at >> http://groups.google.com/group/google-appengine?hl=en. >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Google App Engine" group. >> To post to this group, send email to google-appengine@googlegroups.com. >> To unsubscribe from this group, send email to >> google-appengine+unsubscr...@googlegroups.com. >> For more options, visit this group at >> http://groups.google.com/group/google-appengine?hl=en. >> > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to google-appengine@googlegroups.com. > To unsubscribe from this group, send email to > google-appengine+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > > -- Johan Euphrosine (proppy) Developer Programs Engineer Google Developer Relations -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: Back-ends instances are dying and not restoring
Agreed, The form is there btw: http://code.google.com/p/googleappengine/issues/entry?template=Production%20issue On Mon, Aug 1, 2011 at 3:00 PM, Tim Hoffman wrote: > If you haven't already done so I suggest you also lodge a production issue. > T > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To view this discussion on the web visit > https://groups.google.com/d/msg/google-appengine/-/F_fCE8IBJfkJ. > To post to this group, send email to google-appengine@googlegroups.com. > To unsubscribe from this group, send email to > google-appengine+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > -- Johan Euphrosine (proppy) Developer Programs Engineer Google Developer Relations -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Connect from GAE to external PostgreSQL DB?
Not exactly the answer you expect certainly but wouldn't it better to implement a Web/REST service somewhere in the same network as the Postgres DB (if you can) which would just extract from Postgres and serve the data you need from your GAE app? Like that, you don't expose directly Postgres DB to your remote GAE app which is something more secure IMHO. Pascal On Tue, Aug 2, 2011 at 3:49 PM, wouterdeg wrote: > Hi All, > > For an app I need to obtain data from an external PostgreSQL database, > do some calculations on it and then show the results to the clients in > an MVP set-up. > > However, from some posts I gather that GAE cannot connect with a > PostgreSQL DB over the internet out of its servlet context. Others > suggest setting up an 'http proxy' for this. > > I'm quite a newbie on this, however, I would love to use GAE / GWT for > the most part (particularly GTWP). Anyone that has a suggestion how I > can get to the data on the Postrgres DB out of my GAE servlet > context?? > > Thanks a lot! > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to google-appengine@googlegroups.com. > To unsubscribe from this group, send email to > google-appengine+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Problem fetching db.ListProperty data
Hi Simon,
A wild guess would be that your are not looking at the same entities
in both methods, try to check the entity key using topic.key()
You might also want to check that article about Modeling your data on
App Engine:
http://code.google.com/appengine/articles/modeling.html
Hope that helps.
On Mon, Aug 1, 2011 at 8:43 AM, Simon wrote:
> Hi,
>
> I'm new to appengine and I'm trying to write an app in python, however I
> don't think that my problem is python specific.
> If somebody else already asked the same question, I'm sorry for repeating
> it. I searched for quite a while and couldn't find anything.
>
> My app is a message board application where each user entity has a list of
> all the topics (threads) the user has participated in and each topic has a
> list of all users that have participated, like this.
>
> class User(db.Model):
> user_account = db.UserProperty()
> topic_keys = db.ListProperty(db.Key)
>
> @property
> def topics(self):
> if len(self.topic_keys) == 0:
> return []
> return db.get(self.topic_keys)
> ...
>
> class Topic(db.Model):
> name = db.StringProperty(multiline=True, required=True)
> user_keys = db.ListProperty(db.Key)
>
> @property
> def users(self):
> if len(self.user_keys) == 0:
> return []
> return db.get(self.user_keys)
>
> In the get method of a request handler I tried to retrieve the list of users
> that have participated in the topic
>
> topic_list = Topic.all().filter("name =",topic_id).fetch(limit=1)
> if len(topic_list) > 0:
> topic = topic_list[0]
> topic_users = topic.users
> ...
>
> But doing this gives me a Topic instance where the 'name' property is
> correct but topic.users is empty. However if I try to fetch the topic like
> below, the topic.users I get isn't empty
>
> for tpc in pmwuser.topics:
> if tpc.name == topic_id:
> topic = tpc
> topic_users = tpc.users
>
> What am I doing wrong in the first case? Am I modelling this wrong all
> together? Any help would be appreciated.
>
> Thanks in advance
>
> Simon
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/google-appengine/-/BxbGdTqS6pcJ.
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.
>
--
Johan Euphrosine (proppy)
Developer Programs Engineer
Google Developer Relations
--
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] JDO schema change nightmare, app going live
Hi Manish, Can you link the corresponding Stack Overflow question so we can get a bit more context. Thanks in advance. On Sun, Jul 31, 2011 at 6:47 AM, miwcoder wrote: > Hi, > We are in the process of going live with our app but struggling > with schema changes. We intentionally adopted JDO as we wanted to keep > changes minimum in case we need to port. But it is biting us now as it > is a nightmare to change the schema. We are trying to add only Object > types as suggested in one stackoverflow post but the changes still > throws tons of exceptions. > > How do we handle it? Also is Objectify the only way out? If so how > much work it is to port objectify code? > > any help is appreciated. > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to google-appengine@googlegroups.com. > To unsubscribe from this group, send email to > google-appengine+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > > -- Johan Euphrosine (proppy) Developer Programs Engineer Google Developer Relations -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: Back-ends instances are dying and not restoring
Done - issue posted. Thanks for advice. If will be resolved then will post results here. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/5v0QiIzVLjoJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Huge latency spike on one of my apps
Hi I have had a huge spike in one of my appengine instances starting 4 or so hours ago (and still ongoing). Here is a picture of the graph milliseconds/request http://twitpic.com/5zvvd8 I have only noticed this on the one app (it is a python app), the others all seem to be behaving (python and java). The last deploy before this spike was 12 days ago, so no recent changes. Anyone else seen anything like this today? Grant -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Huge latency spike on one of my apps
Can you share with us your application id ? Feel free to fill a production issue if this is directly affecting your operation: http://code.google.com/p/googleappengine/issues/entry?template=Production%20issue On Tue, Aug 2, 2011 at 5:43 PM, Grant wrote: > Hi > > I have had a huge spike in one of my appengine instances starting 4 or > so hours ago (and still ongoing). > > Here is a picture of the graph milliseconds/request > > http://twitpic.com/5zvvd8 > > I have only noticed this on the one app (it is a python app), the > others all seem to be behaving (python and java). > > The last deploy before this spike was 12 days ago, so no recent > changes. > > Anyone else seen anything like this today? > > Grant > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to google-appengine@googlegroups.com. > To unsubscribe from this group, send email to > google-appengine+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > > -- Johan Euphrosine (proppy) Developer Programs Engineer Google Developer Relations -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] JDO schema change nightmare, app going live
Sorry, just noticed this. Yeah, schema changes really don't seem to have been a design consideration for the JDO/datanucleus integration. If you want help with JDO, best to describe the exact changes you are making. If you want to port to Objectify, it shouldn't be too hard since Objectify directly models the low-level data. It will probably be just rote replacement of your JDO calls. How much work this is just depends on how much code you have. Jeff On Sat, Jul 30, 2011 at 9:47 PM, miwcoder wrote: > Hi, > We are in the process of going live with our app but struggling > with schema changes. We intentionally adopted JDO as we wanted to keep > changes minimum in case we need to port. But it is biting us now as it > is a nightmare to change the schema. We are trying to add only Object > types as suggested in one stackoverflow post but the changes still > throws tons of exceptions. > > How do we handle it? Also is Objectify the only way out? If so how > much work it is to port objectify code? > > any help is appreciated. > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to google-appengine@googlegroups.com. > To unsubscribe from this group, send email to > google-appengine+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: Back-ends instances are dying and not restoring
Seeing the same behavior. Found the url to the issue: http://code.google.com/p/googleappengine/issues/detail?id=5465 On Aug 2, 8:06 am, Kirill Zinoviev wrote: > Done - issue posted. Thanks for advice. > If will be resolved then will post results here. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: Java: Precompilation fails.
Hi Johan, Thanks for the feedback. For me, the problem solved itself. What was troubling for me was that, besides the info that precompilation failed, no info was given. So I was left clueless whether it was caused by something on my side (my app's codes and libs) or an issue with the server that handles precompilation for my app. Actually I still don't know whether my local changes fixed it, or simply the fact that time passed by. I did have some SVN-issues with my gwt-servlet.jar back then but I don't see how they could be related, especially not since it was always present in the lib-directory anyway. The fact that it failed with 1 file outstanding on the first attempt, and with a dozen outstanding in the second attempt (tried to deploy the same files, no new local compilation happened in between) makes me assume that the server had some temporary issue. If you want to check the logs from that time (if there are any), the related app-id is "colectivos-gba". Since the problem does not occur any more, I will not file an issue (unless it returns, but it hadn't so far). Regards, David Hellekalek 2011/8/2 Johan Euphrosine : > Can you two share your application id, so we can investigate the issue ? > > Feel free to fill a production issue if this is directly affecting > your operation: > http://code.google.com/p/googleappengine/issues/entry?template=Production%20issue > > On Sun, Jul 31, 2011 at 9:45 PM, Pavel Kaplin wrote: >> Experiencing same issue >> >> On Jul 24, 2:38 am, David wrote: >>> Hello, >>> >>> I get the following error: >>> >>> Uploading 0 files. >>> Initializing precompilation... >>> Precompiling... 13 file(s) left. >>> Precompiling... 13 file(s) left. >>> Precompiling... 13 file(s) left. >>> Precompiling... 13 file(s) left. >>> on backend null. >>> java.io.IOException: Precompilation failed with 13 file(s) remaining. >>> Consider adding false >>> to your appengine-web.xml and trying again. >>> >>> Debugging information may be found in C:\[...]\appengine- >>> deploy738810777136767626.log >>> >>> Said debug information basically tells me that an error 503 occured >>> (full log attached on bottom). On the first attempt it failed when >>> only 1 file was left. Any idea what to do? Is there any way to find >>> out what caused that Error 503 in the precompile-URL? Is that >>> temporary or permanent? >>> >>> I only found little information on that error so I assume it does not >>> occur too often. Any help is appreciated. >>> >>> Regards, >>> David >>> >>> -- local log-file --- >>> Unable to update: >>> java.io.IOException: Precompilation failed with 13 file(s) remaining. >>> Consider adding false >>> to your appengine-web.xml and trying again. >>> at >>> com.google.appengine.tools.admin.AppVersionUpload.precompile(AppVersionUplo >>> ad.java: >>> 259) >>> at >>> com.google.appengine.tools.admin.AppVersionUpload.doUpload(AppVersionUpload >>> .java: >>> 152) >>> at >>> com.google.appengine.tools.admin.AppAdminImpl.doUpdate(AppAdminImpl.java: >>> 284) >>> at >>> com.google.appengine.tools.admin.AppAdminImpl.update(AppAdminImpl.java: >>> 48) >>> at >>> com.google.appengine.eclipse.core.proxy.AppEngineBridgeImpl.deploy(AppEngin >>> eBridgeImpl.java: >>> 265) >>> at >>> com.google.appengine.eclipse.core.deploy.DeployProjectJob.runInWorkspace(De >>> ployProjectJob.java: >>> 144) >>> at >>> org.eclipse.core.internal.resources.InternalWorkspaceJob.run(InternalWorksp >>> aceJob.java: >>> 38) >>> at org.eclipse.core.internal.jobs.Worker.run(Worker.java:54) >>> Caused by: java.io.IOException: Error posting to >>> URL:https://appengine.google.com/api/appversion/precompile?app_id=[...]&version=1&; >>> 503 Service Unavailable >>> >>> Try Again (503) >>> An unexpected failure has occurred. Please try again. >>> >>> at >>> com.google.appengine.tools.admin.AbstractServerConnection.send(AbstractServ >>> erConnection.java: >>> 245) >>> at >>> com.google.appengine.tools.admin.AbstractServerConnection.post(AbstractServ >>> erConnection.java: >>> 207) >>> at >>> com.google.appengine.tools.admin.AppVersionUpload.send(AppVersionUpload.jav >>> a: >>> 639) >>> at >>> com.google.appengine.tools.admin.AppVersionUpload.sendPrecompileRequest(App >>> VersionUpload.java: >>> 305) >>> at >>> com.google.appengine.tools.admin.AppVersionUpload.precompileChunk(AppVersio >>> nUpload.java: >>> 297) >>> at >>> com.google.appengine.tools.admin.AppVersionUpload.precompile(AppVersionUplo >>> ad.java: >>> 232) >>> ... 7 more >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Google App Engine" group. >> To post to this group, send email to google-appengine@googlegroups.com. >> To unsubscribe from this group, send email to >> google-appengine+unsubscr...@googlegroups.com. >> For more options, visit this group at >> http://groups.google.com/group/google-
[google-appengine] Much more efficient implementation of db.ListProperty(int) - created a lib
So you want to store a "very long list of numbers" in your AppEngine Model which doesn't need to be indexed. Well, you could just do this: *class Foo(db.Model): numbers = db.ListProperty(int, indexed=False) * However, you will quickly notice that the performance and the memory consumption of this sucks if you add thousands of integers to this list. Instead you should use the native python "array.array" type as the list and store it in a BlobProperty. It's up to 30x faster! I created a library that does exactly this (free, apache 2.0 license): http://devblog.miumeet.com/2011/08/much-more-efficient-implementation-of.html Once you have the library, all you need to do is this to get a huge performance boost: *class Foo(db.Model): numbers = db.ArrayProperty()* The blogpost also has more details and an appstat comparison. Cheers & hope you like it -Andrin -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Persisting Lists
By the way, have a look at: http://devblog.miumeet.com/2011/08/much-more-efficient-implementation-of.html It's much more efficient than db.ListProperty(int) Cheers, -Andrin On Thu, Jul 28, 2011 at 6:17 PM, Pascal Voitot Dev < pascal.voitot@gmail.com> wrote: > good idea also :) > > > On Wed, Jul 27, 2011 at 7:17 PM, Ernesto Oltra wrote: > >> And for followers, you could too shard the lists. You can have several >> entities, each with, about 100 results or so (or 1000, or 2000, I prefer 100 >> for easy of serializing/deserializing). All these would have the user as >> ancestor. When listing, take only one entity, deserializing its lists (only >> 100 results) and show some of them. When listing all, you can use cursors >> and some tricks to have the job done (job = paging =) ) >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Google App Engine" group. >> To view this discussion on the web visit >> https://groups.google.com/d/msg/google-appengine/-/tRQCOATsWdUJ. >> >> To post to this group, send email to google-appengine@googlegroups.com. >> To unsubscribe from this group, send email to >> google-appengine+unsubscr...@googlegroups.com. >> For more options, visit this group at >> http://groups.google.com/group/google-appengine?hl=en. >> > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to google-appengine@googlegroups.com. > To unsubscribe from this group, send email to > google-appengine+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] most efficient bulk delete code?
Change your query to something like this:
Query query = pm.newQuery("select ek from " + UploadedContent.class
.getName());
Where "ek" is the entity key name you use in you JDO class. For example,
mine is defined like the following:
@PrimaryKey
@Persistent(valueStrategy = IdGeneratorStrategy.IDENTITY)
@Extension(vendorName="datanucleus", key="gae.encoded-pk", value="true")
private String ek;
Then, just query for the keys and do a batch delete on the keys. Right now
you're pulling in all the entity data and then deleting the entities which
is unnecessary and will slow you do a lot.
Stephen
CortexConnect (cortexconnect.appspot.com)
On Tue, Aug 2, 2011 at 12:17 AM, Carter wrote:
> Is this the most efficient code for bulk delete?
>
> Is there a "SELECT __key__" technique that's faster?
>
> This currently deletes 100 at a time, but it's slow.
>
>
>public long deleteBefore(Date d, long maxRecords)
>throws DaoException {
>
>PersistenceManager pm = pmf.getPersistenceManager();
>Query query = pm.newQuery(UploadedContent.class);
>query.setFilter("dateCreated < dateCreatedParam");
>query.declareParameters("Date dateCreatedParam");
>query.declareImports("import java.util.Date;");
>if (maxRecords >= 0) {
>query.setRange(0, maxRecords);
>}
>
>List entities = new
> ArrayList();
>try {
>entities = (List) query.execute(d);
>pm.deletePersistentAll(entities);
>} catch (Exception e) {
>log.log(Level.SEVERE, e.getMessage(), e);
>throw new DaoException(e);
>} finally {
>query.closeAll();
>pm.close();
>}
>return entities.size();
>}
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.
>
>
--
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] most efficient bulk delete code?
thanks; what delete do you use?
It looked like deletePersistentAll() accepted only entities (rather than key
values)
On Tue, Aug 2, 2011 at 9:46 AM, Stephen Johnson wrote:
> Change your query to something like this:
>
> Query query = pm.newQuery("select ek from " + UploadedContent.class
> .getName());
>
> Where "ek" is the entity key name you use in you JDO class. For example,
> mine is defined like the following:
>
> @PrimaryKey
>
> @Persistent(valueStrategy = IdGeneratorStrategy.IDENTITY)
>
> @Extension(vendorName="datanucleus", key="gae.encoded-pk", value=
> "true")
>
> private String ek;
>
> Then, just query for the keys and do a batch delete on the keys. Right now
> you're pulling in all the entity data and then deleting the entities which
> is unnecessary and will slow you do a lot.
>
> Stephen
> CortexConnect (cortexconnect.appspot.com)
>
> On Tue, Aug 2, 2011 at 12:17 AM, Carter wrote:
>
>> Is this the most efficient code for bulk delete?
>>
>> Is there a "SELECT __key__" technique that's faster?
>>
>> This currently deletes 100 at a time, but it's slow.
>>
>>
>>public long deleteBefore(Date d, long maxRecords)
>>throws DaoException {
>>
>>PersistenceManager pm = pmf.getPersistenceManager();
>>Query query = pm.newQuery(UploadedContent.class);
>>query.setFilter("dateCreated < dateCreatedParam");
>>query.declareParameters("Date dateCreatedParam");
>>query.declareImports("import java.util.Date;");
>>if (maxRecords >= 0) {
>>query.setRange(0, maxRecords);
>>}
>>
>>List entities = new
>> ArrayList();
>>try {
>>entities = (List)
>> query.execute(d);
>>pm.deletePersistentAll(entities);
>>} catch (Exception e) {
>>log.log(Level.SEVERE, e.getMessage(), e);
>>throw new DaoException(e);
>>} finally {
>>query.closeAll();
>>pm.close();
>>}
>>return entities.size();
>>}
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Google App Engine" group.
>> To post to this group, send email to google-appengine@googlegroups.com.
>> To unsubscribe from this group, send email to
>> google-appengine+unsubscr...@googlegroups.com.
>> For more options, visit this group at
>> http://groups.google.com/group/google-appengine?hl=en.
>>
>>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.
>
--
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] most efficient bulk delete code?
I use the low-level datastore to do the delete by keys.
On Tue, Aug 2, 2011 at 10:52 AM, Carter Maslan wrote:
> thanks; what delete do you use?
> It looked like deletePersistentAll() accepted only entities (rather than
> key values)
>
>
> On Tue, Aug 2, 2011 at 9:46 AM, Stephen Johnson wrote:
>
>> Change your query to something like this:
>>
>> Query query = pm.newQuery("select ek from " + UploadedContent.class
>> .getName());
>>
>> Where "ek" is the entity key name you use in you JDO class. For example,
>> mine is defined like the following:
>>
>> @PrimaryKey
>>
>> @Persistent(valueStrategy = IdGeneratorStrategy.IDENTITY)
>>
>> @Extension(vendorName="datanucleus", key="gae.encoded-pk", value=
>> "true")
>>
>> private String ek;
>>
>> Then, just query for the keys and do a batch delete on the keys. Right now
>> you're pulling in all the entity data and then deleting the entities which
>> is unnecessary and will slow you do a lot.
>>
>> Stephen
>> CortexConnect (cortexconnect.appspot.com)
>>
>> On Tue, Aug 2, 2011 at 12:17 AM, Carter wrote:
>>
>>> Is this the most efficient code for bulk delete?
>>>
>>> Is there a "SELECT __key__" technique that's faster?
>>>
>>> This currently deletes 100 at a time, but it's slow.
>>>
>>>
>>>public long deleteBefore(Date d, long maxRecords)
>>>throws DaoException {
>>>
>>>PersistenceManager pm = pmf.getPersistenceManager();
>>>Query query = pm.newQuery(UploadedContent.class);
>>>query.setFilter("dateCreated < dateCreatedParam");
>>>query.declareParameters("Date dateCreatedParam");
>>>query.declareImports("import java.util.Date;");
>>>if (maxRecords >= 0) {
>>>query.setRange(0, maxRecords);
>>>}
>>>
>>>List entities = new
>>> ArrayList();
>>>try {
>>>entities = (List)
>>> query.execute(d);
>>>pm.deletePersistentAll(entities);
>>>} catch (Exception e) {
>>>log.log(Level.SEVERE, e.getMessage(), e);
>>>throw new DaoException(e);
>>>} finally {
>>>query.closeAll();
>>>pm.close();
>>>}
>>>return entities.size();
>>>}
>>>
>>> --
>>> You received this message because you are subscribed to the Google Groups
>>> "Google App Engine" group.
>>> To post to this group, send email to google-appengine@googlegroups.com.
>>> To unsubscribe from this group, send email to
>>> google-appengine+unsubscr...@googlegroups.com.
>>> For more options, visit this group at
>>> http://groups.google.com/group/google-appengine?hl=en.
>>>
>>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Google App Engine" group.
>> To post to this group, send email to google-appengine@googlegroups.com.
>> To unsubscribe from this group, send email to
>> google-appengine+unsubscr...@googlegroups.com.
>> For more options, visit this group at
>> http://groups.google.com/group/google-appengine?hl=en.
>>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.
>
--
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: Data Center (CDN) Locations
The primary data centers serving App Engine are in North America, however, assets and items that are edge cached are served from Google edge caches which are located wherever there are Google data centers. When evaluating page speed, it's typically assets that cause page speeds to appear slow: http://stevesouders.com/hpws/rules.php Only dynamic calls to your application logic would be routed to North America. If you follow these rules, websites should still load quickly - there is a fairly large App Engine community in Japan. -- Ikai Lan Developer Programs Engineer, Google App Engine plus.ikailan.com | twitter.com/ikai On Tue, Aug 2, 2011 at 3:30 AM, Simon Knott wrote: > Hi, > > As far as I'm aware, the most accurate information the Google staff have > divulged is that all data centres are located in North America somewhere. > > Cheers, > Simon > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To view this discussion on the web visit > https://groups.google.com/d/msg/google-appengine/-/OvHwAZFC9aYJ. > > To post to this group, send email to google-appengine@googlegroups.com. > To unsubscribe from this group, send email to > google-appengine+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Website Spam posing by appid: aking-741 using IP 64.233.172.1
I did find the abuse link. It's nice but not practical for webmasters and developers to report network abuse manually. Q1: Is there a way to lookup the applications author / developer using the appid: information from the useragent? For every problem I believe a solution is just around the corner. My thought: Based on the *useragent appid: and IP network* (Google) external applications could query for Abuse / Author email address for real-time abuse alert messaging. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/ze-nIq7riCsJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Connect from GAE to external PostgreSQL DB?
Yeah, you'll need a REST proxy. I wonder if someone has built a version of DBSlayer for Postgres? http://code.nytimes.com/projects/dbslayer DBSlayer's biggest fault: no authentication. -- Ikai Lan Developer Programs Engineer, Google App Engine plus.ikailan.com | twitter.com/ikai On Tue, Aug 2, 2011 at 7:56 AM, Pascal Voitot Dev < pascal.voitot@gmail.com> wrote: > Not exactly the answer you expect certainly but wouldn't it better to > implement a Web/REST service somewhere in the same network as the Postgres > DB (if you can) which would just extract from Postgres and serve the data > you need from your GAE app? > Like that, you don't expose directly Postgres DB to your remote GAE app > which is something more secure IMHO. > > Pascal > > > On Tue, Aug 2, 2011 at 3:49 PM, wouterdeg wrote: > >> Hi All, >> >> For an app I need to obtain data from an external PostgreSQL database, >> do some calculations on it and then show the results to the clients in >> an MVP set-up. >> >> However, from some posts I gather that GAE cannot connect with a >> PostgreSQL DB over the internet out of its servlet context. Others >> suggest setting up an 'http proxy' for this. >> >> I'm quite a newbie on this, however, I would love to use GAE / GWT for >> the most part (particularly GTWP). Anyone that has a suggestion how I >> can get to the data on the Postrgres DB out of my GAE servlet >> context?? >> >> Thanks a lot! >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Google App Engine" group. >> To post to this group, send email to google-appengine@googlegroups.com. >> To unsubscribe from this group, send email to >> google-appengine+unsubscr...@googlegroups.com. >> For more options, visit this group at >> http://groups.google.com/group/google-appengine?hl=en. >> >> > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to google-appengine@googlegroups.com. > To unsubscribe from this group, send email to > google-appengine+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Much more efficient implementation of db.ListProperty(int) - created a lib
Thanks Andrin! If anyone is wondering why this is, it's because the List property is serialized and deserialized using protocol buffers. Protocol buffers are very useful for things like validation and cross language compatibility, but you can sometimes gain some performance benefits by serializing and deserializing using native methods. The protocol buffer implementation for Python is noticeably slow, even via the native extension (if you can improve this - Google wants to hire you), so I'm not surprised techniques like this work. -- Ikai Lan Developer Programs Engineer, Google App Engine plus.ikailan.com | twitter.com/ikai On Tue, Aug 2, 2011 at 9:18 AM, Andrin von Rechenberg wrote: > So you want to store a "very long list of numbers" in your AppEngine Model > which doesn't need to be indexed. > > Well, you could just do this: > > *class Foo(db.Model): > numbers = db.ListProperty(int, indexed=False) > * > However, you will quickly notice that the performance and the memory > consumption of this sucks if you add thousands of integers to this list. > > Instead you should use the native python "array.array" type as the list and > store it in a BlobProperty. It's up to 30x faster! > > I created a library that does exactly this (free, apache 2.0 license): > > http://devblog.miumeet.com/2011/08/much-more-efficient-implementation-of.html > > Once you have the library, all you need to do is this to get a huge > performance boost: > > *class Foo(db.Model): > numbers = db.ArrayProperty()* > > The blogpost also has more details and an appstat comparison. > > Cheers & hope you like it > -Andrin > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to google-appengine@googlegroups.com. > To unsubscribe from this group, send email to > google-appengine+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
RE: [google-appengine] Data Center (CDN) Locations
After doing some research it turns out that the reason GAE is so fast is that it is really a bot net running on every machine with Chrome installed. The speed comes from the fact that in many cases when you hit your own app it is very fast because it is running locally, but since very few servers are running Chrome, speed test tools often return a slower result because they have to hit some poor schmuck's computer, and if he is at Starbucks that could be slow. :-) From: google-appengine@googlegroups.com [mailto:google-appengine@googlegroups.com] On Behalf Of Ian Wilson Sent: Tuesday, August 02, 2011 3:20 AM To: google-appengine@googlegroups.com Subject: [google-appengine] Data Center (CDN) Locations I have been searching around but have not seen information on where App Engine data centers are located and how loads are spread? So I have two questions: - Where are the App Engine data centers? (i.e. West Coast, East Coast, Europe...) - What criteria are used to serve an app from one data center over another? We have an application we are building for a customer in Japan and would like to know if there is an App Engine data center in Asia so we will have lower latency. Thanks. Ian -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/atIjL0tAAAYJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: Crazy number of Deadine Exceeded exceptions recently
I'm afraid this could be related to this issue: https://groups.google.com/d/topic/google-appengine-java/u4R40A7a2X0/discussion We are using M/S too. HTH, Eduardo G. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/GMvE7EwQpIUJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] "Enforced Rate" in task queues ruining performance
Johan, I have submitted a production issue for this problem. Here is the link http://code.google.com/p/googleappengine/issues/detail?id=5471 If you have any questions or need to contact me feel free to email me directly. Thank you, - Dan -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/jJb0TuZM8j0J. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: Data Center (CDN) Locations
Ikai, Thank you for the information. I will keep in mind the optimization tips. Do you guys have local data centers as part of your road map? (I know Amazon recently set up a dedicated center for AWS in Tokyo). Ian -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/FiK61X7sH6QJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] interpreting Logs
Hi, is there anynway to analyse the user actions using the log of requests? I'm thinking in log every action a user does , in my application, like "the user have clicked in homepage". In an apache server for example, i can interprete the log and see the sessionid and the get request. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: Data Center (CDN) Locations
Our external roadmap is here: http://code.google.com/appengine/docs/roadmap.html We don't currently have regional data centers on our roadmap, but it's something we're always thinking about. -- Ikai Lan Developer Programs Engineer, Google App Engine plus.ikailan.com | twitter.com/ikai On Tue, Aug 2, 2011 at 1:38 PM, Ian Wilson wrote: > Ikai, > > Thank you for the information. I will keep in mind the optimization tips. > Do you guys have local data centers as part of your road map? (I know > Amazon recently set up a dedicated center for AWS in Tokyo). > > Ian > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To view this discussion on the web visit > https://groups.google.com/d/msg/google-appengine/-/FiK61X7sH6QJ. > > To post to this group, send email to google-appengine@googlegroups.com. > To unsubscribe from this group, send email to > google-appengine+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: Stackoverflow
So I saw this in my Google Alerts earlier, and I thought, what the heck, people ask development questions on Yahoo Answers? http://answers.yahoo.com/question/index?qid=20110726102419AAFUzgy Can someone with a Yahoo account point him here? I realized that I haven't logged into Yahoo in years and can't seem to remember my password. FWIW, I've always though Yahoo Answers was for questions like this: http://www.funnyyahooquestions.com/sons-birthday-fail/ (I'm joking, of course. I've been in some pretty idiotic Google Groups. It's the people; not the technology.) -- Ikai Lan Developer Programs Engineer, Google App Engine plus.ikailan.com | twitter.com/ikai On Tue, Aug 2, 2011 at 5:42 AM, Gary Frederick wrote: > I was forced to vote for Groups over Stackoverflow. My real vote is both. > > Stackoverflow is ok for a focused question that does not get booted > Groups are better if you have a question that does not fit into the > criteria for Stackoverflow > or you are more having a conversation. > > I do a lot with Google's App Inventor groups. We looked a bit at > Stackoverflow a while ago and it did not fit. > we get a lot more less technical types there... > > and I have failed to check my Stackoverflow account in a while. It was not > encouraging to me to see how much karma I would have to build in a place > with limited use to me to add the pages or whatever they call them to get > the App Inventor info there to where it should be to be useful. I wanted to > be able to point back to the Google forums and those messages got crushed by > the SO cops... > > and > > your milage may differ :-) > > Gary > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To view this discussion on the web visit > https://groups.google.com/d/msg/google-appengine/-/r33pUmrjRIkJ. > > To post to this group, send email to google-appengine@googlegroups.com. > To unsubscribe from this group, send email to > google-appengine+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: Stackoverflow
Had never saw funnyyahooquestions.com before. There goes another hour. As a side note, if this was a SO question we'd have to downvote your answer for being off topic. :P On Tue, Aug 2, 2011 at 17:29, Ikai Lan (Google) wrote: > So I saw this in my Google Alerts earlier, and I thought, what the heck, > people ask development questions on Yahoo Answers? > http://answers.yahoo.com/question/index?qid=20110726102419AAFUzgy > Can someone with a Yahoo account point him here? I realized that I haven't > logged into Yahoo in years and can't seem to remember my password. > FWIW, I've always though Yahoo Answers was for questions like this: > http://www.funnyyahooquestions.com/sons-birthday-fail/ > (I'm joking, of course. I've been in some pretty idiotic Google Groups. It's > the people; not the technology.) > -- > Ikai Lan > Developer Programs Engineer, Google App Engine > plus.ikailan.com | twitter.com/ikai > > > On Tue, Aug 2, 2011 at 5:42 AM, Gary Frederick wrote: >> >> I was forced to vote for Groups over Stackoverflow. My real vote is both. >> Stackoverflow is ok for a focused question that does not get booted >> Groups are better if you have a question that does not fit into the >> criteria for Stackoverflow >> or you are more having a conversation. >> I do a lot with Google's App Inventor groups. We looked a bit at >> Stackoverflow a while ago and it did not fit. >> we get a lot more less technical types there... >> and I have failed to check my Stackoverflow account in a while. It was not >> encouraging to me to see how much karma I would have to build in a place >> with limited use to me to add the pages or whatever they call them to get >> the App Inventor info there to where it should be to be useful. I wanted to >> be able to point back to the Google forums and those messages got crushed by >> the SO cops... >> and >> your milage may differ :-) >> Gary >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Google App Engine" group. >> To view this discussion on the web visit >> https://groups.google.com/d/msg/google-appengine/-/r33pUmrjRIkJ. >> To post to this group, send email to google-appengine@googlegroups.com. >> To unsubscribe from this group, send email to >> google-appengine+unsubscr...@googlegroups.com. >> For more options, visit this group at >> http://groups.google.com/group/google-appengine?hl=en. > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to google-appengine@googlegroups.com. > To unsubscribe from this group, send email to > google-appengine+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: Stackoverflow
Welcome to Knowing Me In Real Life. =) -- Ikai Lan Developer Programs Engineer, Google App Engine plus.ikailan.com | twitter.com/ikai On Tue, Aug 2, 2011 at 2:43 PM, Robert Kluin wrote: > Had never saw funnyyahooquestions.com before. There goes another hour. > > As a side note, if this was a SO question we'd have to downvote your > answer for being off topic. :P > > > > > > > On Tue, Aug 2, 2011 at 17:29, Ikai Lan (Google) wrote: > > So I saw this in my Google Alerts earlier, and I thought, what the heck, > > people ask development questions on Yahoo Answers? > > http://answers.yahoo.com/question/index?qid=20110726102419AAFUzgy > > Can someone with a Yahoo account point him here? I realized that I > haven't > > logged into Yahoo in years and can't seem to remember my password. > > FWIW, I've always though Yahoo Answers was for questions like this: > > http://www.funnyyahooquestions.com/sons-birthday-fail/ > > (I'm joking, of course. I've been in some pretty idiotic Google Groups. > It's > > the people; not the technology.) > > -- > > Ikai Lan > > Developer Programs Engineer, Google App Engine > > plus.ikailan.com | twitter.com/ikai > > > > > > On Tue, Aug 2, 2011 at 5:42 AM, Gary Frederick wrote: > >> > >> I was forced to vote for Groups over Stackoverflow. My real vote is > both. > >> Stackoverflow is ok for a focused question that does not get booted > >> Groups are better if you have a question that does not fit into the > >> criteria for Stackoverflow > >> or you are more having a conversation. > >> I do a lot with Google's App Inventor groups. We looked a bit at > >> Stackoverflow a while ago and it did not fit. > >> we get a lot more less technical types there... > >> and I have failed to check my Stackoverflow account in a while. It was > not > >> encouraging to me to see how much karma I would have to build in a place > >> with limited use to me to add the pages or whatever they call them to > get > >> the App Inventor info there to where it should be to be useful. I wanted > to > >> be able to point back to the Google forums and those messages got > crushed by > >> the SO cops... > >> and > >> your milage may differ :-) > >> Gary > >> > >> -- > >> You received this message because you are subscribed to the Google > Groups > >> "Google App Engine" group. > >> To view this discussion on the web visit > >> https://groups.google.com/d/msg/google-appengine/-/r33pUmrjRIkJ. > >> To post to this group, send email to google-appengine@googlegroups.com. > >> To unsubscribe from this group, send email to > >> google-appengine+unsubscr...@googlegroups.com. > >> For more options, visit this group at > >> http://groups.google.com/group/google-appengine?hl=en. > > > > -- > > You received this message because you are subscribed to the Google Groups > > "Google App Engine" group. > > To post to this group, send email to google-appengine@googlegroups.com. > > To unsubscribe from this group, send email to > > google-appengine+unsubscr...@googlegroups.com. > > For more options, visit this group at > > http://groups.google.com/group/google-appengine?hl=en. > > > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to google-appengine@googlegroups.com. > To unsubscribe from this group, send email to > google-appengine+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: JDO schema change nightmare, app going live
I would recommend you to scrap JDO completely and use objectify. We were in a similar situation as you, changing over to objectify was the best thing ever! And it's easier than you could ever imagine. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/8DgsKuEO1jwJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: Stackoverflow
It is done. On Tue, Aug 2, 2011 at 2:29 PM, Ikai Lan (Google) wrote: > So I saw this in my Google Alerts earlier, and I thought, what the heck, > people ask development questions on Yahoo Answers? > > http://answers.yahoo.com/question/index?qid=20110726102419AAFUzgy > > Can someone with a Yahoo account point him here? I realized that I haven't > logged into Yahoo in years and can't seem to remember my password. > > FWIW, I've always though Yahoo Answers was for questions like this: > > http://www.funnyyahooquestions.com/sons-birthday-fail/ > > (I'm joking, of course. I've been in some pretty idiotic Google Groups. > It's the people; not the technology.) > > -- > Ikai Lan > Developer Programs Engineer, Google App Engine > plus.ikailan.com | twitter.com/ikai > > > > On Tue, Aug 2, 2011 at 5:42 AM, Gary Frederick wrote: > >> I was forced to vote for Groups over Stackoverflow. My real vote is both. >> >> Stackoverflow is ok for a focused question that does not get booted >> Groups are better if you have a question that does not fit into the >> criteria for Stackoverflow >> or you are more having a conversation. >> >> I do a lot with Google's App Inventor groups. We looked a bit at >> Stackoverflow a while ago and it did not fit. >> we get a lot more less technical types there... >> >> and I have failed to check my Stackoverflow account in a while. It was not >> encouraging to me to see how much karma I would have to build in a place >> with limited use to me to add the pages or whatever they call them to get >> the App Inventor info there to where it should be to be useful. I wanted to >> be able to point back to the Google forums and those messages got crushed by >> the SO cops... >> >> and >> >> your milage may differ :-) >> >> Gary >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Google App Engine" group. >> To view this discussion on the web visit >> https://groups.google.com/d/msg/google-appengine/-/r33pUmrjRIkJ. >> >> To post to this group, send email to google-appengine@googlegroups.com. >> To unsubscribe from this group, send email to >> google-appengine+unsubscr...@googlegroups.com. >> For more options, visit this group at >> http://groups.google.com/group/google-appengine?hl=en. >> > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to google-appengine@googlegroups.com. > To unsubscribe from this group, send email to > google-appengine+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: JDO schema change nightmare, app going live
You can also give a try to Siena http://www.sienaproject.com It provides the same level of features as Objectify just above Datastore. But it's not specialized to GAE and aims at providing a bridge between NoSQL DB (and also SQL DB when it's possible). Currently we support GAE (sync/async) + MySQL/Postgres/H2. We also provide high-level features around iteration, pagination (based on cursors, forward & backward)... We also recently added GAE entity groups and entity direct embedding support. SimpleDB is being coded right now and is already quite advanced! Redis is under preparation. regards Pascal PS: i'm lead dev of Siena, don't hesitate to contact me if you have questions! On Wed, Aug 3, 2011 at 12:04 AM, nischalshetty wrote: > I would recommend you to scrap JDO completely and use objectify. We were in > a similar situation as you, changing over to objectify was the best thing > ever! And it's easier than you could ever imagine. > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To view this discussion on the web visit > https://groups.google.com/d/msg/google-appengine/-/8DgsKuEO1jwJ. > > To post to this group, send email to google-appengine@googlegroups.com. > To unsubscribe from this group, send email to > google-appengine+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: High CPU usage and how to improve performance
Issue number is 5415 It has been more than a week and I am yet to get a useful response on the ticket. On Jul 21, 4:20 am, Johan Euphrosine wrote: > > > Feel free to open a Production Issue for problem like this affecting > directly your application > operation:http://code.google.com/p/googleappengine/issues/entry?template=Produc... > > If you do so, please reply to this thread with the issue number, so we > can followup. > > Hope that helps. > > > > > > > > > > On Thu, Jul 21, 2011 at 10:24 AM,observer247 wrote: > > Is this is because of the new pricing, I would rather move out of > > Google app engine. > > > We have over 10 apps serving our iPhone apps. It will take us a lot of > > effort to port this code to php/mysql but I guess if the pricing > > changes like this, we cannot afford to pay such high prices. > > > I am not sure if there is a phone number or support chat. When I am > > paying $1000 per month for a service, I would expect to talk to a real > > person and share with him the issues I am facing. > > > On Jul 20, 1:14 am,observer247 wrote: > >> Almost the same usage is now triggering 10times the daily cost ! Not > >> sure what changed as far as billing. > > >> Checking appstats and instances. > > >> On Jul 19, 2:55 pm, Barry Hunter wrote: > > >> > Have you tried > >> > AppStats?http://code.google.com/appengine/docs/python/tools/appstats.html > > >> > On Tue, Jul 19, 2011 at 8:31 PM,observer247 wrote: > >> > > My python app engine app is somewhat popular, mainly used as via api > >> > > from my iphone app. > > >> > > Recently I am seeing high over usage charges. > > >> > > The most used call already use 15 CPU hours , with stats: > >> > > 3.7 calls/sec 26.15K requests in 12hrs 603 (0) [cpu] > > >> > > HTTP/1.1" 200 1839 - "QandA/5.4 CFNetwork/485.12.7 Darwin/10.4.0" > >> > > "may.latest.nuvusanswers.appspot.com" ms=1490 cpu_ms=467 api_cpu_ms=0 > >> > > cpm_usd=0.013200 pending_ms=1058 throttle_code=1 > >> > > instance=00c61b117c8b0de1286e8d90c22f4d5b3897c8 > > >> > > This call just reads a memcache and returns the data. Thats it. Will > >> > > really like to know are memcache reads so expensive ? Am I using a lot > >> > > more memcache that I need ? > > >> > > I have removed all other versions and have just one version running. > >> > > The number of instances now is > > >> > > 28 total 0.510 (Average QPS) 281.1 ms (Average Latency) > >> > > 59.5 MBytes > >> > > (Average Memory) > > >> > > As this is the most expensive call, I am first going after that. but I > >> > > am seeing most of my calls are now using a LOT of CPU. Will be great > >> > > if anyone can point me to things I can debug to find out the high > >> > > usage across the board. > > >> > > Thank you for reading this ! > > >> > > -- > >> > > You received this message because you are subscribed to the Google > >> > > Groups "Google App Engine" group. > >> > > To post to this group, send email to google-appengine@googlegroups.com. > >> > > To unsubscribe from this group, send email to > >> > > google-appengine+unsubscr...@googlegroups.com. > >> > > For more options, visit this group > >> > > athttp://groups.google.com/group/google-appengine?hl=en. > > > -- > > You received this message because you are subscribed to the Google Groups > > "Google App Engine" group. > > To post to this group, send email to google-appengine@googlegroups.com. > > To unsubscribe from this group, send email to > > google-appengine+unsubscr...@googlegroups.com. > > For more options, visit this group > > athttp://groups.google.com/group/google-appengine?hl=en. > > -- > Johan Euphrosine (proppy) > Developer Programs Engineer > Google Developer Relations -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: Much more efficient implementation of db.ListProperty(int) - created a lib
See also http://bit.ly/6emftF I've been using ArrayProperty for a couple of years. Cheers Greg. On Aug 3, 4:18 am, Andrin von Rechenberg wrote: > So you want to store a "very long list of numbers" in your AppEngine Model > which doesn't need to be indexed. > > Well, you could just do this: > > *class Foo(db.Model): > numbers = db.ListProperty(int, indexed=False) > * > However, you will quickly notice that the performance and the memory > consumption of this sucks if you add thousands of integers to this list. > > Instead you should use the native python "array.array" type as the list and > store it in a BlobProperty. It's up to 30x faster! > > I created a library that does exactly this (free, apache 2.0 > license):http://devblog.miumeet.com/2011/08/much-more-efficient-implementation... > > Once you have the library, all you need to do is this to get a huge > performance boost: > > *class Foo(db.Model): > numbers = db.ArrayProperty()* > > The blogpost also has more details and an appstat comparison. > > Cheers & hope you like it > -Andrin -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Data Center (CDN) Locations
I wonder how many people will think what you said here is true. On Tue, Aug 2, 2011 at 15:11, Brandon Wirtz wrote: > After doing some research it turns out that the reason GAE is so fast is > that it is really a bot net running on every machine with Chrome installed. > The speed comes from the fact that in many cases when you hit your own app > it is very fast because it is running locally, but since very few servers > are running Chrome, speed test tools often return a slower result because > they have to hit some poor schmuck’s computer, and if he is at Starbucks > that could be slow. > > > > :-) > > > > > > > > From: google-appengine@googlegroups.com > [mailto:google-appengine@googlegroups.com] On Behalf Of Ian Wilson > Sent: Tuesday, August 02, 2011 3:20 AM > To: google-appengine@googlegroups.com > Subject: [google-appengine] Data Center (CDN) Locations > > > > I have been searching around but have not seen information on where App > Engine data centers are located and how loads are spread? > > So I have two questions: > > > > - Where are the App Engine data centers? (i.e. West Coast, East Coast, > Europe...) > > - What criteria are used to serve an app from one data center over another? > > > > We have an application we are building for a customer in Japan and would > like to know if there is an App Engine data center in Asia so we will have > lower latency. > > > > Thanks. > > > > Ian > > > > > > > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To view this discussion on the web visit > https://groups.google.com/d/msg/google-appengine/-/atIjL0tAAAYJ. > To post to this group, send email to google-appengine@googlegroups.com. > To unsubscribe from this group, send email to > google-appengine+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to google-appengine@googlegroups.com. > To unsubscribe from this group, send email to > google-appengine+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: Crazy number of Deadine Exceeded exceptions recently
Interesting. I've been seen exactly the same strange behavior across several apps as well. Suddenly instances will get killed and restarted in large batches. This happens even with low request latency, small memory usage (similar to yours < 50mb), low error rates, and steady traffic. I pretty convinced this is tied to the scheduler changes they've been making over the past few weeks. As a side note, the pending_ms value (9321) indicates that the request sat there waiting to be serviced for quite a long time. That won't leave as much time to respond to the requests. Do you always see bursts of those when your instances get killed off? Are you getting big spikes in traffic when this happens or is it steady? Robert On Tue, Aug 2, 2011 at 05:24, Alexis wrote: > Hi, > > I've got a similar issue: lots of DeadlineExceeded errors since a few > weeks. I'm on the master-slave datastore too, but what I'm reporting > happened again one hour ago. > > These errors happen in bursts, and I recently realized that it was in > fact shutting down ALL instances of the application. > (In the logs, I also have this warning: A serious problem was > encountered with the process that handled this request, causing it to > exit. This is likely to cause a new process to be used for the next > request to your application. If you see this message frequently, you > may be throwing exceptions during the initialization of your > application. (Error code 104)) > This does not happen when an instance is spinning up but after several > hours. > > The trace I get along with the DeadlineExceeded errors show that it > happens in the second phase: while the app is trying to fallback > gracefully because of an other error (that does not appears in logs). > Request reported processing time can be like this: ms=100878 > cpu_ms=385 api_cpu_ms=58 cpm_usd=0.010945 pending_ms=9321 > > Here is a screenshot of the admin page, showing that all instances > have been shut down about 7 minutes ago, even resident ones: > http://dl.dropbox.com/u/497622/spinDown.png > > > The app do work in batches (although not always small ones). But > request processing time is usually good enough (see average latency on > the screen shot). > I'm trying things on my testing applications to see what can be wrong > but it's still not clear for me and I'm running short of ideas... > > Any suggestions? > > > > > On 2 août, 06:21, Robert Kluin wrote: >> Hi Will, >> I assume this is on the master-slave datastore? I think there were >> a number of large latency spikes in both the datastore and serving >> last week. >> >> Some things to try: >> - do work in smaller batches. >> - if you're doing work serially, do it in batches. >> - use async interfaces to do work in batches, but in parallel using >> async. >> >> http://code.google.com/appengine/docs/python/datastore/async.html >> >> Robert >> >> >> >> >> >> >> >> On Fri, Jul 29, 2011 at 18:35, Will Reiher wrote: >> > I'm trying to debug this issue but I keep hitting a wall. >> > I keep trying new things on one of my deployments to see if I an get the >> > number of errors down but nothing seems to help. It all started in the last >> > week or so ago. I also have some existing deployments that I have not >> > changed and are seeing these same errors while the code was never changed >> > and had been stable. >> > 1. This is is happening on existing code that has not changed recently >> > 2. The DeadlineExceededErrors are coming up randomly and at different >> > points >> > in the code. >> > 3. Latency is pretty high and app engine seems to be spawning a lot of new >> > instances beyond my 3 included ones. >> >> > -- >> > You received this message because you are subscribed to the Google Groups >> > "Google App Engine" group. >> > To view this discussion on the web visit >> >https://groups.google.com/d/msg/google-appengine/-/g_C4iPzPeo4J. >> > To post to this group, send email to google-appengine@googlegroups.com. >> > To unsubscribe from this group, send email to >> > google-appengine+unsubscr...@googlegroups.com. >> > For more options, visit this group at >> >http://groups.google.com/group/google-appengine?hl=en. > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to google-appengine@googlegroups.com. > To unsubscribe from this group, send email to > google-appengine+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Website Spam posing by appid: aking-741 using IP 64.233.172.1
On Tue, Aug 2, 2011 at 14:54, Murray W wrote: > I did find the abuse link. > It's nice but not practical for webmasters and developers to report network > abuse manually. What do you want to do, monitor a site and if some app is hitting it too frequently automatically report it to Google for abusing *your* ToS? > > Q1: Is there a way to lookup the applications author / developer using the > appid: information from the useragent? No. > > For every problem I believe a solution is just around the corner. > My thought: Based on the useragent appid: and IP network (Google) external > applications could query for Abuse / Author email address for real-time > abuse alert messaging. > > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To view this discussion on the web visit > https://groups.google.com/d/msg/google-appengine/-/ze-nIq7riCsJ. > To post to this group, send email to google-appengine@googlegroups.com. > To unsubscribe from this group, send email to > google-appengine+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Increase apps size restriction above 10 megabytes?
Yes. http://code.google.com/p/googleappengine/issues/detail?id=5472 On Tue, Aug 2, 2011 at 10:52, Johan Euphrosine wrote: > Did someone fill a doc bug for this ? > > Thanks in advance. > > On Sun, Jul 31, 2011 at 7:30 PM, Robert Kluin wrote: >> I don't think that is correct, at least not according to some of the docs: >> >> - maximum size of an application file: 10 megabytes >> - maximum size of a static file: 10 megabytes >> - maximum total size of all application and static files: 150 megabytes >> >> But that is exactly my point, I think there are discrepancies in the docs. >> >> >> >> >> On Sun, Jul 31, 2011 at 02:23, Brandon Wirtz wrote: >>> 10 for app, 150 for static assets. >>> >>> >>> >>> From: google-appengine@googlegroups.com >>> [mailto:google-appengine@googlegroups.com] On Behalf Of Robert Kluin >>> Sent: Saturday, July 30, 2011 11:10 PM >>> To: google-appengine@googlegroups.com >>> Subject: Re: [google-appengine] Increase apps size restriction above 10 >>> megabytes? >>> >>> >>> >>> I think there are some descrepancies in the docs about the max application >>> size. In some places the max app size is listed as 10mb, others 150mb (Max >>> file size of 10mb). >>> >>> http://code.google.com/appengine/docs/java/runtime.html#Quotas_and_Limits >>> http://code.google.com/appengine/docs/python/runtime.html#Quotas_and_Limits >>> >>> http://code.google.com/appengine/docs/quotas.html#Deployments >>> >>> I'm pretty sure there are others that list 10mb too. >>> >>> Robert >>> >>> On Jul 29, 2011 1:56 PM, "Ikai Lan (Google)" wrote: >>> >>> -- >>> You received this message because you are subscribed to the Google Groups >>> "Google App Engine" group. >>> To post to this group, send email to google-appengine@googlegroups.com. >>> To unsubscribe from this group, send email to >>> google-appengine+unsubscr...@googlegroups.com. >>> For more options, visit this group at >>> http://groups.google.com/group/google-appengine?hl=en. >>> >>> -- >>> You received this message because you are subscribed to the Google Groups >>> "Google App Engine" group. >>> To post to this group, send email to google-appengine@googlegroups.com. >>> To unsubscribe from this group, send email to >>> google-appengine+unsubscr...@googlegroups.com. >>> For more options, visit this group at >>> http://groups.google.com/group/google-appengine?hl=en. >>> >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Google App Engine" group. >> To post to this group, send email to google-appengine@googlegroups.com. >> To unsubscribe from this group, send email to >> google-appengine+unsubscr...@googlegroups.com. >> For more options, visit this group at >> http://groups.google.com/group/google-appengine?hl=en. >> >> > > > > -- > Johan Euphrosine (proppy) > Developer Programs Engineer > Google Developer Relations > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to google-appengine@googlegroups.com. > To unsubscribe from this group, send email to > google-appengine+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] most efficient bulk delete code?
Just a note, I've heard the built in datastore admin delete code is
running much faster recently. You might give it a shot. I have not
tried it myself to verify it is running better though (previously it
was quite poor).
Robert
On Tue, Aug 2, 2011 at 14:54, Stephen Johnson wrote:
> I use the low-level datastore to do the delete by keys.
>
> On Tue, Aug 2, 2011 at 10:52 AM, Carter Maslan wrote:
>>
>> thanks; what delete do you use?
>> It looked like deletePersistentAll() accepted only entities (rather than
>> key values)
>>
>> On Tue, Aug 2, 2011 at 9:46 AM, Stephen Johnson
>> wrote:
>>>
>>> Change your query to something like this:
>>> Query query = pm.newQuery("select ek from
>>> " + UploadedContent.class.getName());
>>> Where "ek" is the entity key name you use in you JDO class. For example,
>>> mine is defined like the following:
>>>
>>> @PrimaryKey
>>>
>>> @Persistent(valueStrategy = IdGeneratorStrategy.IDENTITY)
>>>
>>> @Extension(vendorName="datanucleus", key="gae.encoded-pk",
>>> value="true")
>>>
>>> private String ek;
>>>
>>> Then, just query for the keys and do a batch delete on the keys. Right
>>> now you're pulling in all the entity data and then deleting the entities
>>> which is unnecessary and will slow you do a lot.
>>> Stephen
>>> CortexConnect (cortexconnect.appspot.com)
>>> On Tue, Aug 2, 2011 at 12:17 AM, Carter wrote:
Is this the most efficient code for bulk delete?
Is there a "SELECT __key__" technique that's faster?
This currently deletes 100 at a time, but it's slow.
public long deleteBefore(Date d, long maxRecords)
throws DaoException {
PersistenceManager pm = pmf.getPersistenceManager();
Query query = pm.newQuery(UploadedContent.class);
query.setFilter("dateCreated < dateCreatedParam");
query.declareParameters("Date dateCreatedParam");
query.declareImports("import java.util.Date;");
if (maxRecords >= 0) {
query.setRange(0, maxRecords);
}
List entities = new
ArrayList();
try {
entities = (List)
query.execute(d);
pm.deletePersistentAll(entities);
} catch (Exception e) {
log.log(Level.SEVERE, e.getMessage(), e);
throw new DaoException(e);
} finally {
query.closeAll();
pm.close();
}
return entities.size();
}
--
You received this message because you are subscribed to the Google
Groups "Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.
>>>
>>> --
>>> You received this message because you are subscribed to the Google Groups
>>> "Google App Engine" group.
>>> To post to this group, send email to google-appengine@googlegroups.com.
>>> To unsubscribe from this group, send email to
>>> google-appengine+unsubscr...@googlegroups.com.
>>> For more options, visit this group at
>>> http://groups.google.com/group/google-appengine?hl=en.
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Google App Engine" group.
>> To post to this group, send email to google-appengine@googlegroups.com.
>> To unsubscribe from this group, send email to
>> google-appengine+unsubscr...@googlegroups.com.
>> For more options, visit this group at
>> http://groups.google.com/group/google-appengine?hl=en.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.
>
--
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] interpreting Logs
You could give analytics a try: http://www.google.com/analytics/ I suppose you could also log some additional info in each request (if needed) and periodically grab the logs if you wanted to. http://code.google.com/appengine/docs/python/tools/uploadinganapp.html#Downloading_Logs You could probably also come up with something using memcache + the datastore. I'd say it all depends on what you're trying to do with the information. Robert On Tue, Aug 2, 2011 at 16:46, Bruno Sandivilli wrote: > Hi, is there anynway to analyse the user actions using the log of requests? > I'm thinking in log every action a user does , in my application, like "the > user have clicked in homepage". In an apache server for example, i can > interprete the log and see the sessionid and the get request. > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to google-appengine@googlegroups.com. > To unsubscribe from this group, send email to > google-appengine+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Problem fetching db.ListProperty data
On Mon, Aug 1, 2011 at 02:43, Simon wrote:
> Hi,
>
> I'm new to appengine and I'm trying to write an app in python, however I
> don't think that my problem is python specific.
> If somebody else already asked the same question, I'm sorry for repeating
> it. I searched for quite a while and couldn't find anything.
>
> My app is a message board application where each user entity has a list of
> all the topics (threads) the user has participated in and each topic has a
> list of all users that have participated, like this.
>
> class User(db.Model):
> user_account = db.UserProperty()
> topic_keys = db.ListProperty(db.Key)
>
> @property
> def topics(self):
> if len(self.topic_keys) == 0:
> return []
> return db.get(self.topic_keys)
> ...
Personally, I would probably write this as:
if not self.topic_keys:
return []
return db.get(self.topic_keys)
I would also probably rather not make it a property, but a method
named something like get_topics. That way it is explicit that you'll
be making an RPC. Just a personal preference. ;)
>
> class Topic(db.Model):
> name = db.StringProperty(multiline=True, required=True)
> user_keys = db.ListProperty(db.Key)
>
> @property
> def users(self):
> if len(self.user_keys) == 0:
> return []
> return db.get(self.user_keys)
>
> In the get method of a request handler I tried to retrieve the list of users
> that have participated in the topic
>
> topic_list = Topic.all().filter("name =",topic_id).fetch(limit=1)
See Query.get:
http://code.google.com/appengine/docs/python/datastore/queryclass.html#Query_get
> if len(topic_list) > 0:
> topic = topic_list[0]
> topic_users = topic.users
> ...
After you've rewritten this using Query.get you'll have something like:
topic = Topic.all().filter("name =", topic_id).get()
if topic:
topic_users = topic.get_users()
logging.debug(topic.user_keys)
What happens if you add that logging call? Are you getting keys?
Robert
>
> But doing this gives me a Topic instance where the 'name' property is
> correct but topic.users is empty. However if I try to fetch the topic like
> below, the topic.users I get isn't empty
>
> for tpc in pmwuser.topics:
> if tpc.name == topic_id:
> topic = tpc
> topic_users = tpc.users
>
> What am I doing wrong in the first case? Am I modelling this wrong all
> together? Any help would be appreciated.
>
> Thanks in advance
>
> Simon
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/google-appengine/-/BxbGdTqS6pcJ.
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.
>
--
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: Federated Login redirecting to Google OpenID and not /_ah/login_required on Java GAE
Hi Vishal, Did you go through Wesley's example doc: http://code.google.com/appengine/articles/openid.html You might try asking on the app engine java group. Robert On Tue, Aug 2, 2011 at 00:49, Vishal wrote: > Anyone? I still can't get Federated Login to work correctly. > > Vishal > > On Jul 29, 12:05 am, Vishal wrote: >> Hello Wise AppEngineers, >> >> I need your help to figure out what I'm doing wrong. I've enabled >> Federated Login for my Java app, but whenever I visit a security- >> constraint protected url-pattern, I get redirected to a default Google >> OpenID login screen. I want my /_ah/login_required servlet handler to >> be called instead. The documentation says this is default behavior, >> but that's not happening for me. >> >> I'm aware of the potential infinite redirect issue (that's not what >> I'm facing). Here's how to reproduce the behavior I'm seeing: >> 1) Visithttp://demo.hotels-z.appspot.com/_ah/login_required. Verify >> that the servlet is installed and serves a page. >> 2) Visithttp://demo.hotels-z.appspot.com/manage. Note that I'm not >> redirected to my _ah/login_required handler but some default Google >> OpenID handler. >> >> Here are parts of my web.xml: >> >> LoginRequiredServlet >> /_ah/login_required >> >> >> >> >> /manage/* >> >> >> * >> >> >> >> What am I doing wrong? >> >> Thanks! >> >> Vishal > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to google-appengine@googlegroups.com. > To unsubscribe from this group, send email to > google-appengine+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Unserializing large Blobs from the Blobstore
On Mon, Jul 11, 2011 at 05:55, Luca Matteis wrote: > Hello, > > I store large Blobs in the datastore. These are XML files that can be > as large as 20mb. Storing a single 20mb XML file is fine, however the > issue comes when I need to unserialize it. > > There are all sorts of limitations to this: > > - I cannot read more than 1mb from Blobstore Read in 1mb chunks, then assemble and parse. > - even if I could I'm still confronted with the 5mb RAM limitation > (since I need to put all the XML into RAM before unserializing it) What 5mb RAM limit? > > How would you say I can handle this? I'm open to all sorts of > solutions, but hopefully not something that involves using another > hosting provider. Your backend solution sounds like a good way to go here. Robert > > Thanks, > Luca > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to google-appengine@googlegroups.com. > To unsubscribe from this group, send email to > google-appengine+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] full text search and/or external search services
Have you tried running whoosh (not whoosh-appengine) on a backend? Backends are expensive, but might help you out a bit. Robert On Thu, Jul 28, 2011 at 11:19, andreas schmid wrote: > hi guys, > > i really need to have a full text search on my app and i wanted to know what > kind of external services you are using on your apps. or how you implement > such a service. > i tried whoosh-appengine but first its a really old version and second it has > huge issues with 1000+ indexed entities. > > we don't know when the full text search will be released for GAE right? > > thx > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to google-appengine@googlegroups.com. > To unsubscribe from this group, send email to > google-appengine+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
