[appengine-java] Task queues seems paused.
I have many task in some queues that seems to be paused. I tried the Run now on some tasks, I tried to pause and then resume with no luck, the tasks are stuck in the queue. Hints? -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
[appengine-java] Re: 1.4.2: Can't make https call from dev server
JVM: java version 1.6.0_23 Java(TM) SE Runtime Environment (build 1.6.0_23-b05) Java HotSpot(TM) Client VM (build 19.0-b09, mixed mode, sharing) On Feb 13, 12:27 pm, Lior Harsat lior.har...@gmail.com wrote: I am having a similar problem with https: I have been testing the SDK and I found one bug which seems critical. When I call the URLFetchService.fetch API with a https URL I get a javax.net.ssl.SSLHandshakeException. for example when I fetch https://spreadsheets.google.com/feeds/ list/ I get javax.net.ssl.SSLHandshakeException: Could not verify SSL certificate for:https://spreadsheets.google.com/feeds/list/... When I switch back to an older SDK this doesn't happen. Is there a Help us test Prerelease SDK 1.4.2 thread I can post this to? Can anyone confirm this bug reproduces? Thanx, Lior On Feb 12, 11:57 am, Peter Liu tinyee...@gmail.com wrote: We are using a custom remote delegate to route datastore rpc call to live production test site on appspot. We did this because sometimes we want to develop with live data. However after upgrading to 1.4.2, https outbound call starts failing (see below). If we change to http, then it works again. It seems like some class is marked as restricted in this release. Is it intentional? What's the reason behind not allowing outbound https call? Thanks. java.lang.NoClassDefFoundError: javax.net.ssl.SSLHandshakeException is a restricted class. Please see the Google App Engine developer's guide for more details. at com.google.appengine.tools.development.agent.runtime.Runtime.reject(Runtime .java: 51) at com.google.appengine.api.urlfetch.URLFetchServiceImpl.convertApplicationExc eption(URLFetchServiceImpl.java: 114) at com.google.appengine.api.urlfetch.URLFetchServiceImpl.access $100(URLFetchServiceImpl.java:22) at com.google.appengine.api.urlfetch.URLFetchServiceImpl $1.convertException(URLFetchServiceImpl.java:86) at com.google.appengine.api.utils.FutureWrapper.get(FutureWrapper.java: 69) at com.pekca.main.delegate.RemoteDelegate$1.get(RemoteDelegate.java: 163) -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
[appengine-java] Re: 1.4.2: Can't make https call from dev server
HI All,
I found a very dirty workaround but at least it works for me.
I was not aware of the doNotValidateCertificate() API that Fred
mentioned.
I made a very small change (one line of code) to the SDK appengine-
api.jar.
I have decompiled com.google.appengine.api.urlfetch.HTTPRequest and
added the call to doNotValidateCertificate() on the FetchOptions in
the constructor:
public HTTPRequest(URL url, HTTPMethod method, FetchOptions
fetchOptions) {
payload = null;
this.url = url;
this.method = method;
this.fetchOptions = fetchOptions;
this.fetchOptions.doNotValidateCertificate(); \\ADDED BY ME
headers = new LinkedHashMap();
}
Then I repackaged the jar with my change and that solved the issue for
me - No certificate validation by default (Only on Dev server).
This way I don't have to change every piece of code that uses
URLFetchService with https and more important the production env still
makes sure the validation occurs.
Note that this solution will fix only flows where the URLFetchService
is used.
I do hope a better fix is provided by the App Engine team soon.
Thanx, Lior
On Feb 14, 11:22 pm, frew f...@google.com wrote:
Hey everybody,
In order to help me diagnose the problems, could you respond with what
JDK (Platform, JDK manufacturer (e.g. Sun, OpenJDK), and version)
you're running? As a temporary stopgap, if you're using URLFetch
service, you can use the doNotValidateCertificate() method in
FetchOptions (http://code.google.com/appengine/docs/java/javadoc/com/
google/appengine/api/urlfetch/FetchOptions.html) to disable
validation.
-Fred
On Feb 13, 5:02 am, harri mauria hmau...@googlemail.com wrote:
I am having similar problems with the recently released version
com.google.appengine.eclipse.sdkbundle.1.4.2_1.4.2.v201102111811
Problems happen with the development server, not in production. I am
performing Facebook login credentials authentication in the callback
url. Interrupt stack is below:
HTTP ERROR 500
Problem accessing /loginfacebookcallback. Reason:
Could not verify SSL certificate
for:https://graph.facebook.com/oauth/access_token?omitted
Caused by:
javax.net.ssl.SSLHandshakeException: Could not verify SSL certificate
for:https://graph.facebook.com/oauth/access_token?omitted
at
com.google.appengine.api.urlfetch.URLFetchServiceImpl.convertApplicationExc
eption(URLFetchServiceImpl.java:
114)
at
com.google.appengine.api.urlfetch.URLFetchServiceImpl.fetch(URLFetchService
Impl.java:
41)
at
com.google.apphosting.utils.security.urlfetch.URLFetchServiceStreamHandler
$Connection.fetchResponse(URLFetchServiceStreamHandler.java:418)
at
com.google.apphosting.utils.security.urlfetch.URLFetchServiceStreamHandler
$Connection.getInputStream(URLFetchServiceStreamHandler.java:297)
atwww.superpass.de.server.UrlFetcher.get(UrlFetcher.java:60)
Following message is printed to the console:
I/O exception (javax.net.ssl.SSLPeerUnverifiedException) caught when
processing request: HTTPS hostname invalid: expected
'graph.facebook.com', received '*.facebook.com'
Retrying request
Regards, Harri
On Feb 13, 11:27 am, Lior Harsat lior.har...@gmail.com wrote:
I am having a similar problem with https:
I have been testing the SDK and I found one bug which seems critical.
When I call the URLFetchService.fetch API with a https URL I get a
javax.net.ssl.SSLHandshakeException.
for example when I fetch https://spreadsheets.google.com/feeds/
list/ I get
javax.net.ssl.SSLHandshakeException: Could not verify SSL certificate
for:https://spreadsheets.google.com/feeds/list/...
When I switch back to an older SDK this doesn't happen.
Is there a Help us test Prerelease SDK 1.4.2 thread I can post this
to?
Can anyone confirm this bug reproduces?
Thanx, Lior
On Feb 12, 11:57 am, Peter Liu tinyee...@gmail.com wrote:
We are using a custom remote delegate to route datastore rpc call to
live production test site on appspot. We did this because sometimes we
want to develop with live data.
However after upgrading to 1.4.2, https outbound call starts failing
(see below). If we change to http, then it works again.
It seems like some class is marked as restricted in this release. Is
it intentional? What's the reason behind not allowing outbound https
call?
Thanks.
java.lang.NoClassDefFoundError: javax.net.ssl.SSLHandshakeException is
a restricted class. Please see the Google App Engine developer's
guide for more details.
at
com.google.appengine.tools.development.agent.runtime.Runtime.reject(Runtime
.java:
51)
at
com.google.appengine.api.urlfetch.URLFetchServiceImpl.convertApplicationExc
eption(URLFetchServiceImpl.java:
114)
at
[appengine-java] Blob upload progress
How can i show Blob upload progress to user? Thanks -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] Unable to create JSP file with google app engine in eclipse
It is quite possible you are missing some libraries. What error do you see when you open the jsp file ? On Tue, Feb 15, 2011 at 5:52 AM, Ishwarya Chandrasekaran cishwa...@gmail.com wrote: Hi All, I am a beginner to google app engine. I have a basic doubt. When ever I try to create a jsp file under war directory of my gae project, it shows a red cross across the file. I have jre installed and it is in the build path of the project. Sorry for asking this basic doubt. Please help me Thanks in advance -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en. -- Chummar Maly http://servetube.appspot.com -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
[appengine-java] Problems in deployment
Hi, I am receiving the following errors during deployment: Rolling back the update. java.io.IOException: Error posting to URL: https://appengine.google.com/api/appversion/clonefiles?app_id=stcjadeversion=1; 500 Internal Server Error Any help would be greatly appreciated. Thanks. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
[appengine-java] Re: Problems in deployment
I am getting the same error. My local logfile for this deploy has the following stack trace. This implies some sort of connection down at Google. Is there any word on this? at com.google.appengine.tools.admin.ServerConnection.send(ServerConnection.java:149) at com.google.appengine.tools.admin.ServerConnection.post(ServerConnection.java:82) at com.google.appengine.tools.admin.AppVersionUpload.send(AppVersionUpload.java:582) at com.google.appengine.tools.admin.AppVersionUpload.cloneFiles(AppVersionUpload.java:452) at com.google.appengine.tools.admin.AppVersionUpload.beginTransaction(AppVersionUpload.java:415) at com.google.appengine.tools.admin.AppVersionUpload.doUpload(AppVersionUpload.java:112) at com.google.appengine.tools.admin.AppAdminImpl.update(AppAdminImpl.java:56) at com.google.appengine.eclipse.core.proxy.AppEngineBridgeImpl.deploy(AppEngineBridgeImpl.java:271) at com.google.appengine.eclipse.core.deploy.DeployProjectJob.runInWorkspace(DeployProjectJob.java:146) at org.eclipse.core.internal.resources.InternalWorkspaceJob.run(InternalWorkspaceJob.java:38) at org.eclipse.core.internal.jobs.Worker.run(Worker.java:54) -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
[appengine-java] Re: Problems in deployment
Yep, same thing already for second day.. the problem started for me about 24 hours ago.. java.io.IOException: Error posting to URL: https://appengine.google.com/api/appversion/clonefiles?app_id=bizgymversion=dev; 500 Internal Server Error at com.google.appengine.tools.admin.ServerConnection.send(ServerConnection.java:149) at com.google.appengine.tools.admin.ServerConnection.post(ServerConnection.java:82) at com.google.appengine.tools.admin.AppVersionUpload.send(AppVersionUpload.java:579) at com.google.appengine.tools.admin.AppVersionUpload.cloneFiles(AppVersionUpload.java:449) at com.google.appengine.tools.admin.AppVersionUpload.beginTransaction(AppVersionUpload.java:412) at com.google.appengine.tools.admin.AppVersionUpload.doUpload(AppVersionUpload.java:112) at com.google.appengine.tools.admin.AppAdminImpl.update(AppAdminImpl.java:56) at com.google.appengine.tools.admin.AppCfg$UpdateAction.execute(AppCfg.java:547) at com.google.appengine.tools.admin.AppCfg.init(AppCfg.java:138) at com.google.appengine.tools.admin.AppCfg.init(AppCfg.java:61) at com.google.appengine.tools.admin.AppCfg.main(AppCfg.java:57) at com.google.appengine.tools.admin.AppAdminImpl.update(AppAdminImpl.java:62) at com.google.appengine.tools.admin.AppCfg$UpdateAction.execute(AppCfg.java:547) at com.google.appengine.tools.admin.AppCfg.init(AppCfg.java:138) at com.google.appengine.tools.admin.AppCfg.init(AppCfg.java:61) at com.google.appengine.tools.admin.AppCfg.main(AppCfg.java:57) -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
[appengine-java] Re: Problems in deployment
Same for me. This happened after updating an app thats been running peacefully for many months to 1.4.2 . Anxiously awaiting a solution. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] Re: Problems in deployment
I had the same problem last night and again tonight but not this morning (I'm on the West Coast of the US) hopefully it will work better tomorrow, but I'm losing a lot of time... On Tue, Feb 15, 2011 at 9:11 PM, bgood ben.mcgee.g...@gmail.com wrote: Same for me. This happened after updating an app thats been running peacefully for many months to 1.4.2 . Anxiously awaiting a solution. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
[appengine-java] Could not locate appengine-web.xml
Hi there, I'm developing a web app on NetBenas 6.9 and want to use the GAE appcfg.sh tool to upload the war. However, I'm running in to the following exception: bardu@bardu-laptop:~/GAE/appengine-java-sdk-1.4.2/bin$ ./appcfg.sh update /home/bardu/NetBeansProjects/enterprise/mobitec/dist Reading application configuration data... 15-Feb-2011 9:07:31 PM com.google.apphosting.utils.config.AppEngineWebXmlReader readAppEngineWebXml SEVERE: Received exception processing /home/bardu/NetBeansProjects/enterprise/mobitec/dist/WEB-INF/appengine-web.xml com.google.apphosting.utils.config.AppEngineConfigException: Could not locate /home/bardu/NetBeansProjects/enterprise/mobitec/dist/WEB-INF/appengine-web.xml at com.google.apphosting.utils.config.AppEngineWebXmlReader.getInputStream(AppEngineWebXmlReader.java:102) at com.google.apphosting.utils.config.AppEngineWebXmlReader.readAppEngineWebXml(AppEngineWebXmlReader.java:60) at com.google.appengine.tools.admin.Application.init(Application.java:115) at com.google.appengine.tools.admin.Application.readApplication(Application.java:164) at com.google.appengine.tools.admin.AppCfg.init(AppCfg.java:118) at com.google.appengine.tools.admin.AppCfg.init(AppCfg.java:63) at com.google.appengine.tools.admin.AppCfg.main(AppCfg.java:59) Caused by: java.io.FileNotFoundException: /home/bardu/NetBeansProjects/enterprise/mobitec/dist/WEB-INF/appengine-web.xml (No such file or directory) at java.io.FileInputStream.open(Native Method) at java.io.FileInputStream.init(Unknown Source) at java.io.FileInputStream.init(Unknown Source) at com.google.apphosting.utils.config.AppEngineWebXmlReader.getInputStream(AppEngineWebXmlReader.java:99) ... 6 more Bad configuration: Could not locate /home/bardu/NetBeansProjects/enterprise/mobitec/dist/WEB-INF/appengine-web.xml Caused by: /home/bardu/NetBeansProjects/enterprise/mobitec/dist/WEB-INF/appengine-web.xml (No such file or directory) Please see the logs [/tmp/appcfg2817474116007198989.log] for further information. bardu@bardu-laptop:~/GAE/appengine-java-sdk-1.4.2/bin$ What I don't understand since the appengine-war.xml is definitely in the WEB-INF/. What do I miss? Thanks. Stephan -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
[appengine-java] Re: possible bug with Blob service
Hi, I just starred it. didier On Feb 16, 1:57 am, Andrei gml...@gmail.com wrote: is now Issue 4583 On Feb 14, 11:28 pm, Didier Durand durand.did...@gmail.com wrote: Hi, This has been an issue for some times in this forum and it seems that you found the source of the problem! You definitely should open an issue athttp://code.google.com/p/googleappengine/issues/listand then announce it here back in this stream so that we can also star it for a quicker fix by Google. regards didier On Feb 15, 5:45 am, Andrei gml...@gmail.com wrote: There is example for using Blob service appengine-java-sdk/demos/mediastore There is jsp file upload.jsp If you replace input type=submit name=submit value=Submit with input type=submit name=Add Image value=Add Image you get this exception Uncaught exception from servlet java.lang.IllegalStateException: Must be called from a blob upload callback request. at com.google.appengine.api.blobstore.BlobstoreServiceImpl.getUploadedBlobs(Bl obstoreServiceImpl.java: 128) at com.google.appengine.demos.mediastore.UploadPost.doPost(UploadPost.java: 48) at javax.servlet.http.HttpServlet.service(HttpServlet.java:637) at javax.servlet.http.HttpServlet.service(HttpServlet.java:717) at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java: 511) at org.mortbay.jetty.servlet.ServletHandler $CachedChain.doFilter(ServletHandler.java:1166) at com.google.apphosting.utils.servlet.ParseBlobUploadFilter.doFilter(ParseBlo bUploadFilter.java: 95) at org.mortbay.jetty.servlet.ServletHandler $CachedChain.doFilter(ServletHandler.java:1157) at com.google.apphosting.runtime.jetty.SaveSessionFilter.doFilter(SaveSessionF ilter.java: 35) at org.mortbay.jetty.servlet.ServletHandler $CachedChain.doFilter(ServletHandler.java:1157) at com.google.apphosting.utils.servlet.TransactionCleanupFilter.doFilter(Trans actionCleanupFilter.java: 43) at org.mortbay.jetty.servlet.ServletHandler $CachedChain.doFilter(ServletHandler.java:1157) at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java: 388) at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java: 216) at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java: 182) at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java: 765) at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java: 418) at com.google.apphosting.runtime.jetty.AppVersionHandlerMap.handle(AppVersionH andlerMap.java: 238) at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java: 152) at org.mortbay.jetty.Server.handle(Server.java:326) at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java: 542) at org.mortbay.jetty.HttpConnection $RequestHandler.headerComplete(HttpConnection.java:923) at com.google.apphosting.runtime.jetty.RpcRequestParser.parseAvailable(RpcRequ estParser.java: 76) at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404) at com.google.apphosting.runtime.jetty.JettyServletEngineAdapter.serviceReques t(JettyServletEngineAdapter.java: 135) at com.google.apphosting.runtime.JavaRuntime.handleRequest(JavaRuntime.java: 261) at com.google.apphosting.base.RuntimePb$EvaluationRuntime $2.handleRequest(RuntimePb.java:8440) at com.google.net.rpc.impl.RpcUtil.runRpcInApplication(RpcUtil.java: 454) at com.google.net.rpc.impl.Server$RpcTask.runInContext(Server.java: 572) at com.google.tracing.TraceContext$TraceContextRunnable $1.run(TraceContext.java:448) at com.google.tracing.TraceContext.runInContext(TraceContext.java: 688) at com.google.tracing.TraceContext $AbstractTraceContextCallback.runInInheritedContextNoUnref(TraceContext.jav a: 326) at com.google.tracing.TraceContext $AbstractTraceContextCallback.runInInheritedContext(TraceContext.java: 318) at com.google.tracing.TraceContext $TraceContextRunnable.run(TraceContext.java:446) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java: 1110) at java.util.concurrent.ThreadPoolExecutor $Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:636) Unexpected exception from servlet: java.lang.IllegalStateException: Must be called from a blob upload callback request -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to
Re: [appengine-java] Re: Problems in deployment
Hi , I am able to deploy successfully. Which version of app engine you are using? On Wed, Feb 16, 2011 at 10:55 AM, Alain de Raynal aderay...@gmail.comwrote: I had the same problem last night and again tonight but not this morning (I'm on the West Coast of the US) hopefully it will work better tomorrow, but I'm losing a lot of time... On Tue, Feb 15, 2011 at 9:11 PM, bgood ben.mcgee.g...@gmail.com wrote: Same for me. This happened after updating an app thats been running peacefully for many months to 1.4.2 . Anxiously awaiting a solution. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] Re: Problems in deployment
The latest version 1.4.2. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] Re: Problems in deployment
Same version I'm using and I'm able to deploy it without any problem. I'm located in India. Thanks, Amit On Wed, Feb 16, 2011 at 12:19 PM, jt jooth...@stconsultancy.com wrote: The latest version 1.4.2. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
[google-appengine] Re: A difficult app engine optimisation problem - selecting distinct entities across a large table
We're returning to a web browser. Specifically as JSON. I'll look into that video! On Feb 14, 8:35 pm, Calvin calvin.r...@gmail.com wrote: Are you returning results to a web browser, or a specialized client? One of the Google I/O talks demonstrates spawning a crapload of tasks in parallel to collect results, and in that demo most of the tasks had completed by the time the web page had refreshed (it might be the one someone linked to at stackoverflow). If it's webpage responsiveness that's the issue, you could provide results using the channel api, or with ajax polling. If it's a specialized client, could it poll for results every few seconds until they're exhausted? -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: A difficult app engine optimisation problem - selecting distinct entities across a large table
Copy and pasted from StackOverflow so everyone sees the answer: It's a really good idea, but: because we're looking for the latest journey in a unknown time frame, you can't store a how many days old it is (or equivalent), because if a car stopped making journeys for a week, the last journey would have a days old or equivalent of -7. But, in a time frame between now (0) and 2 weeks ago (-14) we would need that journey (because it's the latest). In a time frame of now (0) to 3 days ago (-3) though, we must not show that journey. You end up doing a 0 and = 3 again. The alternative is to turn your singular field into N fields (say, 1 for each day/hour) for the time frame we need. So in our example, we want to look at journeys over a 2 month period. We could then have a true/false value on each field as to whether to show that journey in a given selection. So in the example above, the journey made 7 days ago would have -7, -6, -5 ... 0 all set to true and -8 onwards to false. But because of the inevitable forwardness of time maintaining that set of fields gets difficult. Add in the 100 index limit and things get difficult. On Feb 14, 11:53 pm, nickmilon nickmi...@gmail.com wrote: I have faced same kind of problem some time ago. I tried some of the solutions suggested here (in memory sort and filtering, encoding things into keys etc. and I have benchmarked those for both latency and cpu cycles using some test data around 100K entities) An other approach I have taken is encoding the date as an integer (day since start of epoch or day since start of year, same for hour of day or month depending on how much detail you need in your output) and saving this into a property. This way you turn your date query filter into an equality only filter which does not even needs to specify an index) then you can sort or filter on other properties. Benchmarking the latest solution I have found that when the filtered result set is a small fraction of the unfiltered original set, is 1+ order of magnitude faster and cpu-eficient. Worst case when no reduction of the result set due to filtering the latency and cpu usage was comparable to the previous solutions) Hope this helps, or did I missed something ? Happy coding-:) On Feb 14, 11:51 pm, Stephen Johnson onepagewo...@gmail.com wrote: Okay I think I got something that might work. Reverse the StartDate and CarId for the key from what I said above so the key would look like this: 2011:02:14:17:13:33:123 and the KEYS ONLY query then is: select __key__ where __key__ = MAKEKEY(StartDate + CarId) __key__ = MAKEKEY(EndDate + CarId) order by __key__ DESC Now, you can use the Async query to start processing. You're going to get entries that you're not interested in but you're only getting the key field back and not the whole CarJourney entry and this key/id has the Date and Car ID, so the first time you hit a Car ID for each Car then you have the ID for the latest CarJourney for that car. Now, once you've found all car ID's your looking for you can abort the query or you'll reach the end of the query results. Now, as you're looping, store the KEYs of the entries your looking for and then do a batch GET on memcache to retrieve as many Car (you've got the car id) and CarJourney (you've got the carjourney id) entries that might be stored there and then for any that you didn't get from memcache, you can do a batch GET on the datastore using the keys/ids that you have. I think that if you memcache things appropriately and use the batch gets for memcache and datastore then this might just work for you. Let me know what you think. It's an interestng problem, Stephen On Mon, Feb 14, 2011 at 2:12 PM, Stephen Johnson onepagewo...@gmail.comwrote: Or maybe it blocks on different result sets just not on getting the next fetch block?? Hmmm. Sounds like a tough problem. On Mon, Feb 14, 2011 at 2:09 PM, Stephen Johnson onepagewo...@gmail.comwrote: Are you using .asList (which I think blocks like you describe), but I thought asIterable or asIterator wasn't suppose to. (if you're using Java). On Mon, Feb 14, 2011 at 12:38 PM, Edward Hartwell Goose ed.go...@gmail.com wrote: Hi Calvin Stephen, Thanks for the ideas. Calvin: We can't do the filtering in memory. We potentially have a car making a journey (the car analogy isn't so good...) making a journey every 3 seconds, and we could have up to 2,000 cars. We need to be able to look back up to 2 months, so it could be up to 1.8 billion rows in this table. Stephen: That's an interesting idea. However the Asynchronous api actually fires the requests synchronously, it just doesn't block. (Or at least, that's my experience). So, at the moment we fire off 1 query (which Google turns into 2) for each site. And although the method call returns instantly, it still takes ~5 seconds in total with basic
[google-appengine] Re: A difficult app engine optimisation problem - selecting distinct entities across a large table
The query window is between now and 2 months ago, with granularity of a minute. So 15/02/2011 09:28:59 is the same as 15/02/2011 09:28:01. The number of cars is likely to be very slow increasing, probably won't see big jumps at all. The majority will start at the beginning of our beta period. Once a car starts reporting journeys, it's likely to continue reporting journeys at the same rate (which could be somewhere between every 3 seconds and every 24 hours) indefinitely. Both Stephen and your ideas sound like really good steps. I'll discuss them with my team later and try and get one implemented. Fingers crossed! And of course - thanks for the help! On Feb 15, 3:42 am, Robert Kluin robert.kl...@gmail.com wrote: Hi Ed, I think Stephen's most recent idea (key name as date + car id) might work well. I have a suggestion that is somewhat similar to Nick's idea. How wide and variable is the query window? How variable are the occurrences of car journey data? Once a car starts having journeys, will it continue having journeys for a (relatively) long time? You said there would be up to 2,000 cars, are new cars frequently introduced and old cars removed? Depending on those questions, perhaps you could build your own index. Think about an entity that stores a (non-indexed) list of cars / car journeys. If you make the key_name for that entity lower resolution than the start time, you store the most recent journey for each car. If you construct these entities properly, then you could easily build a list of the keys needed to completely enclose a particular query window by fetching a small number of these lists. If you also create higher-level groups, you can also efficiently query over very large periods of time. The key names might look something like: 201102151430: 15 Feb 2011 14:30: Journeys that occurred between 14:25 and 14:30. 2011021514: 15 Feb 2011 14: Journeys that occurred from 13:01 to 14:00 20110215: 15 Feb 2011: Journeys that occurred on 15 Feb 2011 etc... You might use a list of tuples on the entity that stores the car id, journey id, and start time. That will allow you to efficiently fetch the cars and the journeys and correctly combine and filter the results in memory when you need more than one index entity. You would need to construct the key names based on your query windows. With some planning, the technique work very well even for 'continuous' query windows that are highly variable in size. I've used this technique for reporting and am very pleased with it. It is also relatively easy to implement. You just need a background process that maintains your indexes, it can be (almost) real-time or an offline background process. Robert On Mon, Feb 14, 2011 at 18:53, nickmilon nickmi...@gmail.com wrote: I have faced same kind of problem some time ago. I tried some of the solutions suggested here (in memory sort and filtering, encoding things into keys etc. and I have benchmarked those for both latency and cpu cycles using some test data around 100K entities) An other approach I have taken is encoding the date as an integer (day since start of epoch or day since start of year, same for hour of day or month depending on how much detail you need in your output) and saving this into a property. This way you turn your date query filter into an equality only filter which does not even needs to specify an index) then you can sort or filter on other properties. Benchmarking the latest solution I have found that when the filtered result set is a small fraction of the unfiltered original set, is 1+ order of magnitude faster and cpu-eficient. Worst case when no reduction of the result set due to filtering the latency and cpu usage was comparable to the previous solutions) Hope this helps, or did I missed something ? Happy coding-:) On Feb 14, 11:51 pm, Stephen Johnson onepagewo...@gmail.com wrote: Okay I think I got something that might work. Reverse the StartDate and CarId for the key from what I said above so the key would look like this: 2011:02:14:17:13:33:123 and the KEYS ONLY query then is: select __key__ where __key__ = MAKEKEY(StartDate + CarId) __key__ = MAKEKEY(EndDate + CarId) order by __key__ DESC Now, you can use the Async query to start processing. You're going to get entries that you're not interested in but you're only getting the key field back and not the whole CarJourney entry and this key/id has the Date and Car ID, so the first time you hit a Car ID for each Car then you have the ID for the latest CarJourney for that car. Now, once you've found all car ID's your looking for you can abort the query or you'll reach the end of the query results. Now, as you're looping, store the KEYs of the entries your looking for and then do a batch GET on memcache to retrieve as many Car (you've got the car id) and CarJourney (you've got
Re: [google-appengine] Re: Sudden redirect to Turing anti-bot test during remote api session
I'm getting the same issue again today. It's happening more often recently, which is worrying. I put my other server in maintenance more so it doesn't call App Engine hoping that after a few minutes GAE would stop blocking my IP, but so far that's not helping either. I'll try waiting a longer time. Tim, Thanks for the tip, but in my case my calls don't require authentication. I'm calling public APIs that I host on the App Engine over http and without any kind of authentication. So I guess that issue in the thread you referred to doesn't apply. Any way to white-list my server's IP address so it doesn't get blocked like this? Regards, Waleed On Thu, Jan 27, 2011 at 6:07 AM, Tim Hoffman zutes...@gmail.com wrote: Hi You might want to have a look at this thread hhttps://groups.google.com/forum/?hl=en#!searchin/google-appengine/Tim$20Hoffman$20remote_api/google-appengine/YrkJqnpCoL0/f1droDRP-DEJ *https://groups.google.com/d/topic/google-appengine/YrkJqnpCoL0/discussion * I ran into this a long time ago and I found it occurred if I wasn't reusing the remote_api handle. I had been recreating the remote_api handle on each transaction, which is inefficient and after 200 transactions I would be redirected to a recaptcha service so I could prove I wasn't a bot ;-) Rgds T -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: Sudden redirect to Turing anti-bot test during remote api session
Hi Maybe you should require auth, we never get the turing bot now that I fixed the connection T -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: Sudden redirect to Turing anti-bot test during remote api session
Good point. I'll give that a shot. Regarding the current issue, I put my other server on hold for a while and tried again, and GAE kept rejecting my calls. I tried that a few times, waiting for more than 10 minutes between tests and it still failed. Finally, I tried uploading my app to GAE again, and that fixed it. I'm guessing uploading a new version restarts the counter for whatever system was falsely considering me like a bot. Regards, Waleed On Tue, Feb 15, 2011 at 2:19 AM, Tim Hoffman zutes...@gmail.com wrote: Hi Maybe you should require auth, we never get the turing bot now that I fixed the connection T -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: Could not verify SSL certificate
Hi, Frew, I'm using JDK 1.6.0 (MacOS X). Mauro, as far as I know this is not development SDK, I installed it from Eclipse yesterday. I didn't installed the previous releases of 1.4.2. Also when I changed the SDK to 1.4.0 from project settings it works. Thanks. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: Could not verify SSL certificate
I was using 1.2.1 version of google api client library and I updated it to 1.2.2. Its working well with 1.4.0 but now I'm getting a different error for 1.4.2. :) -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Python Admin pages no longer recognised
I have an admin page in my app. In the app.yaml, it is configured to be accessible only to project admins by use of the login: admin line. But when I visit the url that this is set up for, I see this (in Chrome): Error: Not FoundThe requested URL /_ah/login_required?continue=http://nat-abbotts.appspot.com/post was not found on this server. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: Sudden redirect to Turing anti-bot test during remote api session
Hi Waleed, Approximately how many requests/minute was your other server making to your GAE app before it was blocked? Would be interesting to know what the cut-off point is. Nick On 15 February 2011 21:24, Waleed Abdulla wal...@ninua.com wrote: Good point. I'll give that a shot. Regarding the current issue, I put my other server on hold for a while and tried again, and GAE kept rejecting my calls. I tried that a few times, waiting for more than 10 minutes between tests and it still failed. Finally, I tried uploading my app to GAE again, and that fixed it. I'm guessing uploading a new version restarts the counter for whatever system was falsely considering me like a bot. Regards, Waleed On Tue, Feb 15, 2011 at 2:19 AM, Tim Hoffman zutes...@gmail.com wrote: Hi Maybe you should require auth, we never get the turing bot now that I fixed the connection T -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: Python Admin pages no longer recognised
Thought some additional details would be helpful: All other pages on the app respond. The logs have no errors displayed regarding this, but one warning appears: /_ah/login_required?continue=http://nat-abbotts.appspot.com/post 1. No handlers matched this URL. Changing the version had no effect. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: Could not verify SSL certificate
To be more clear here are the versions I currently use which I get the error; Java(TM) SE Runtime Environment (build 1.6.0_22-b04-307-10M3261) App Engine API 1.4.2 Google API Client 1.2.2 alpha With the following versions I'm not getting the same error. Java(TM) SE Runtime Environment (build 1.6.0_22-b04-307-10M3261) App Engine API 1.4.0 Google API Client 1.2.2 alpha If I'm getting the error because of JDK I'll see the same error when using GAE 1.4.0 too? -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] App Engine SDK 1.4.2 is out!
@Sandeep I am encountering the same problem. Did you find a solution? Thanks -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] App Engine SDK 1.4.2 is out!
Sandeep and Norvo, I was bitten by this bug as well, and I was able to
figure out a solution. It seems that with Django 1.2,
the register_template_library call no longer takes a simple file path.
Instead, it expects a module.class dot-notation. To fix it, I put the file
that contains my custom tag definitions (MVCTags.py in my case) in a
directory called 'tags' and added an empty __init__.py file. That makes
'tags' a Python module. Then, I changed the call to
register_template_library to this: register_template_library('tags.MVCTags')
and it works just fine.
--
You received this message because you are subscribed to the Google Groups
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] App Engine SDK 1.4.2 is out!
Ya it worked for the same way, added a folder with init file and the actual
file inside it.
app_path, taglib = taglib_module.rsplit('.',1)
This is the bug here , trying to split with the last word in the module
path.
for the single word input its raising this kind errors. but this is helpful,
can place the filter file in any path of the app...
On Tue, Feb 15, 2011 at 8:04 PM, Adam adam.crossl...@gmail.com wrote:
Sandeep and Norvo, I was bitten by this bug as well, and I was able to
figure out a solution. It seems that with Django 1.2,
the register_template_library call no longer takes a simple file path.
Instead, it expects a module.class dot-notation. To fix it, I put the file
that contains my custom tag definitions (MVCTags.py in my case) in a
directory called 'tags' and added an empty __init__.py file. That makes
'tags' a Python module. Then, I changed the call to
register_template_library to this: register_template_library('tags.MVCTags')
and it works just fine.
--
You received this message because you are subscribed to the Google Groups
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.
--
Regards
Sandeep Koduri
Phone: +91- 99 666 02 456
Gtalk: sandeep.koduri | Skype: sandeep.koduri
--
You received this message because you are subscribed to the Google Groups
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: My server can't reach app engine for being in a forbidden country (it's in Atlanta)
Google fixed this issue last night. Thanks to the Google engineers who addressed the problem. I'm not sure this would have been addressed if we hadn't had a friend internal to Google who looked into the issue for us. I'm still frustrated that the moderators of this board never replied. I suppose it's difficult to separate real issues (i.e., something's broken) from spurious complaints (i.e., I can't get my code to work). Max On Feb 11, 4:40 pm, Max Harper maxhar...@gmail.com wrote: Our VPS server (in Atlanta, GA, USA) appears to be blacklisted by app engine (403 errors, forbidden country), causing our app to be functionally down. Help? Details: We stream data to our app engine app from a Linode server, hosted in Atlanta, GA. Our ip address is 74.207.230.***. As of 19 hours ago, we're getting 403s when trying to access pages on our app engine app. Lynx shows: Your client does not have permission to get URL / from this server. (Client IP address: 74.207.230.***) You are accessing this page from a forbidden country. Log messages (403, or otherwise) do not make it to the app engine app console. wget also shows 403s for any of my app engine apps. For instance: max@nea:~/tmp$ wgethttp://maxharp3r-sandbox.appspot.com --2011-02-11 16:32:18-- http://maxharp3r-sandbox.appspot.com/ Resolving maxharp3r-sandbox.appspot.com... 74.125.67.141 Connecting to maxharp3r-sandbox.appspot.com|74.125.67.141|:80... connected. HTTP request sent, awaiting response... 403 Forbidden 2011-02-11 16:32:18 ERROR 403: Forbidden. These urls all work when I'm accessing from my local development machine (in Minneapolis, MN, USA). I'm not quite sure where to turn, since app engine has essentially no support contact. Thanks for your suggestions and help. Max -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] App Engine SDK 1.4.2 is out!
Fantastic! That solved it for me indeed as well. Thanks a million -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: LIFO key name strategy?
Julian, that is _exactlty_ what I was looking for. Counting up to a future time to create descending values... It seems so obvious now! It's a much better solution than the de-accumulator handler that I wrote. Thanks! On Feb 14, 11:57 pm, Julian Namaro namarojul...@gmail.com wrote: I am not sure about the mathematics of it, but intuitively there is no perfect algorithm for constructing timestamps in a reverse lexicographical ordering, because adding a character to a string will always make it lexicographically superior. But I noticed the mapreduce library just pick a ridiculously future time and go in reverse order from there:http://code.google.com/p/appengine-mapreduce/source/browse/trunk/pyth... The library also add a random string to reduce the chance of duplicates, maybe that can be replaced by an UUID if you're really concerned by uniqueness. On Feb 14, 5:57 am, Joseph Letness joe.letn...@gmail.com wrote: Hi Calvin and Robert, thanks for your replies. I should have been more clear about what I am doing, here is some more info: Calvin, thanks for the link to Ikai's blog post, I haven't seen that one and it was very interesting. Robert, here are specific answers to your questions: Why do you say: I can't use a composite index since it would explode with my use case? I'm using Brett Slatkin's Relation Index method of building and querying set memberships (Google I/O 2009 - Building Scalable, Complex Apps on App Engine). According to Brett, using a composite index on this kind would cause explosion, so any ordering of results will need to be done in-memory during the request. If the sort order is immutable, sorted key names can be used to order results based on the their lexicographical position. Since a creation timestamp is immutable data, I figured that using lexicographic key names would be the way to go. What would be fine if you could handle your entire result set in one request? Ordering the result set in-memory. What are you trying to do? The app is a digital-asset manager. Users need to be able to query a set (using the relation index method) and have the results return the most recent additions first. The result set could easily be a few thousand, so I want to use cursor-pagination to display the results which would preclude any in-memory ordering. (I'm actually refactoring my existing app that I use to manage/deliver graphic assets to my clients so that they can add their own data.) Is there a single global LIFO stack, or are there multiple stacks? The entities are all of the same kind, however, LIFO behavior is localized to individual user groups. How are new items added to the stack(s)?, What is the addition rate? Just one item per user request. User groups would be just a few individual users probably less than twenty. The rate per group would be so low that chances of contention on any sort of accumulator would be almost nonexistent. Is there a requirement that the items are precisely ordered or are some (or small) mis-orderings acceptable? Precision is NOT critical. Close approximation of chronology is just fine. --The auto-generated ids are not strictly increasing I did not know that. Thanks! --Using the current time may also be problematic since the machines will have slight variations, and in some cases significant variations. I was aware of that, but since absolute precision is not necessary I could still use the timestamp as an accumulator if there is some thing as an inverse-timestamp algorithm!?!? So... After spending some more time thinking about this, here is what I plan to do: Create a counter model kind that is created with an IntegerProperty starting value of ten billion (I'd like to see somebody reach the bottom of that!). Give each user group it's own counter and de-count the values in a transaction (or not, it might be simpler to dismiss contention and write a handler that ensures uniqueness of the key name but maintains approximate lexicographic position). When the counter value is read, convert the value to a padded string and concatenate it with the user group name and a leading lowercase letter (k999836/ usergroupname) and use that as the key name for the new asset. Furthermore, it occurred to me that as a result set is reduced to a manageable in-memory size, I could test for the length of results and offer the user the ability to custom order their results (asset name alphanumeric or asset kind, for example). Just a thought. Thanks again for the replies, If anyone thinks there is a better approach please let me know, I kind of make this stuff up as I go along.. --Joe On Feb 12, 10:52 pm, Robert Kluin robert.kl...@gmail.com wrote: Hi Joe, What are you actually trying to do? Is there a single global LIFO stack, or are there multiple stacks? How are new items
Re: [google-appengine] Re: A difficult app engine optimisation problem - selecting distinct entities across a large table
Hi Ed, One issue you'll have is where you have a large time frame to query over and some cars that make infrequent journeys and cars that make very frequent journeys so you'll have to scan a lot of key entries to find all cars, so as a further optimization, if you are able to somehow classify a car by it's general frequency of journal entries, you could assign slot numbers to the cars. It wouldn't matter if sometimes they deviate from this general frequency. For example, create five slots, let's call them A,B,C,D,E and let's say car 123 infrequently makes journeys then they go to slot A and car 789 makes frequent journeys so they go to slot E. Then you can add that slot number to the Car entity information and add the slot number to the beginning of the CarJournal keys so it now looks like: SLOT:DATE:CAR-ID A:2011:02:14:17:13:33:123 So, now when doing your query for cars, you'll get the cars your interested in and determine the slot they are in and group the cars according to slot and run the following query per slot for the cars in that slot: select __key__ where __key__ = MAKEKEY(Slot + StartDate + CarId) __key__ = MAKEKEY(Slot + EndDate + CarId) order by __key__ DESC Now, you'll have more queries to run (up to five or maybe 3 if you choose three slots, etc.), but if you're classification of cars is even mildly accurate over a long period you'll probably save yourself a lot of scanning. Let us know how your testing goes and what you end up doing, Stephen On Tue, Feb 15, 2011 at 2:32 AM, Edward Hartwell Goose ed.go...@gmail.comwrote: The query window is between now and 2 months ago, with granularity of a minute. So 15/02/2011 09:28:59 is the same as 15/02/2011 09:28:01. The number of cars is likely to be very slow increasing, probably won't see big jumps at all. The majority will start at the beginning of our beta period. Once a car starts reporting journeys, it's likely to continue reporting journeys at the same rate (which could be somewhere between every 3 seconds and every 24 hours) indefinitely. Both Stephen and your ideas sound like really good steps. I'll discuss them with my team later and try and get one implemented. Fingers crossed! And of course - thanks for the help! On Feb 15, 3:42 am, Robert Kluin robert.kl...@gmail.com wrote: Hi Ed, I think Stephen's most recent idea (key name as date + car id) might work well. I have a suggestion that is somewhat similar to Nick's idea. How wide and variable is the query window? How variable are the occurrences of car journey data? Once a car starts having journeys, will it continue having journeys for a (relatively) long time? You said there would be up to 2,000 cars, are new cars frequently introduced and old cars removed? Depending on those questions, perhaps you could build your own index. Think about an entity that stores a (non-indexed) list of cars / car journeys. If you make the key_name for that entity lower resolution than the start time, you store the most recent journey for each car. If you construct these entities properly, then you could easily build a list of the keys needed to completely enclose a particular query window by fetching a small number of these lists. If you also create higher-level groups, you can also efficiently query over very large periods of time. The key names might look something like: 201102151430: 15 Feb 2011 14:30: Journeys that occurred between 14:25 and 14:30. 2011021514: 15 Feb 2011 14: Journeys that occurred from 13:01 to 14:00 20110215: 15 Feb 2011: Journeys that occurred on 15 Feb 2011 etc... You might use a list of tuples on the entity that stores the car id, journey id, and start time. That will allow you to efficiently fetch the cars and the journeys and correctly combine and filter the results in memory when you need more than one index entity. You would need to construct the key names based on your query windows. With some planning, the technique work very well even for 'continuous' query windows that are highly variable in size. I've used this technique for reporting and am very pleased with it. It is also relatively easy to implement. You just need a background process that maintains your indexes, it can be (almost) real-time or an offline background process. Robert On Mon, Feb 14, 2011 at 18:53, nickmilon nickmi...@gmail.com wrote: I have faced same kind of problem some time ago. I tried some of the solutions suggested here (in memory sort and filtering, encoding things into keys etc. and I have benchmarked those for both latency and cpu cycles using some test data around 100K entities) An other approach I have taken is encoding the date as an integer (day since start of epoch or day since start of year, same for hour of day or month depending on how much detail you need in your output) and saving
Re: [google-appengine] Re: Python Admin pages no longer recognised
hi Nathanael, Looks like you've got your app set to use federated login (openid). You need to write a handler for that url (/_ah/login_required) to setup the login handler. http://code.google.com/appengine/articles/openid.html Robert On Tue, Feb 15, 2011 at 06:51, Nathanael Abbotts nat.abbo...@gmail.com wrote: Thought some additional details would be helpful: All other pages on the app respond. The logs have no errors displayed regarding this, but one warning appears: /_ah/login_required?continue=http://nat-abbotts.appspot.com/post No handlers matched this URL. Changing the version had no effect. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: Python Admin pages no longer recognised
Ah, Thanks! I'll just switch back to google account authentication. That's fixed the problem. Thank you :) -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] startsWith multiple times
In our company we feel obstructed by the restriction that a query may only contain one inequality filter like startsWith. Currently we are thinking about performing two independent startsWith queries and combining the result inside the service method. Do you think this is a good idea? Does anybody have experience with such a strategy or even know a better solution? Thank you very much Jan -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Multiple Google App for Business customers, single cellphone number verification
Hi, We are thinking of using Google App Engine as our platform to build custom applications for our clients. As part of our new service offer, we plan to migrate our clients to Google Apps for Business. The single cellphone number per Google App Engine account looks like a problematic limitation for the use case described above. We want our customers to be directly billable for their custom applications usage, as they already are for Google App for Business, while not having to «verify» their account through some random cellphone within their organization. In other words, is there a streamlined way for us to register one Google App Engine account per domain registered with Google App for Business? Thanks. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] A difficult app engine optimisation problem - selecting distinct entities across a large table
Hi Edward, (I'm by no means expert, so feel free to discount the following) In my somewhat limited experience, I've identified only three ways of making queries faster: 1. denormalizing data (pushing more work into the writes) 2. deferring work (typically via task queues) 3. admitting inconsistency (typically reconciling with cron jobs) and I try to apply these techniques in that order. Disregarding (3) (under the assumption that users need accurate results), you've already explained that the data goes back to the client on the same request, so (2) is only an option if the client is able to page the results of queries over smaller lists of Cars, or poll for the completion of a task. Perhaps this is an option. As I see it, the crux of your problem is that you are trying to retrieve the CarJourneys based on a list of Keys. Since App Engine doesn't provide an equivalent to the SQL IN operator (one that won't just repeat the query N times on your behalf), no amount of denormalization is going to help you here, because you're always going to be stuck making separate queries against each Car key OR plucking out the small set of Cars you want from all the results. IF there is something that identifies the Cars in your list (eg. that they are all associated with the same Company say) then it might be possible to denormalize that (eg. adding the Company key to every CarJourney entity would allow you to narrow down the dataset returned even if some application level filtering was still necessary). Without that, I can't see a direct way of significantly improving the query. Tom. On 14 February 2011 14:59, Edward Hartwell Goose ed.go...@gmail.com wrote: Hi, I was wondering if anyone can help me with this problem. We have an idea we'd like to implement, and we're currently unable to do this efficiently. I've anonymised the data as best as possible, but the structure is the same. We have two entities, Car and CarJourney. Each Car has 0 to many CarJourney's. Each Car Journey has (amongst other properties) a date associated with it - the date the journey was started. I wish to query by time over car journeys. I'll have two times, a start date and an end date, where start date = endDate, and I want to receive the most recently started journey in that period. So, if I had a particular car in mind, say car 123, I'd write a query that limits by Car.key and Car.startDate, where Car.key == 123 and Journey.startDate = startDate and Journey.startDate = endDate with an ordering on Journey.startDate descending and a limit of 1. e.g. Car A has 3 journeys, taken on 1st, 2nd and the 3rd of the month. The query start date is 1st and the query end date is the 2nd. The result of this query would be one Car journey, the 2nd. Once the result of that query is returned, a very small amount of processing is done to return a result to the user. That's the easy bit. But, instead of over 1 Car, I want a list of cars, where the list contains N keys to cars. So, I want to run the above query N times, once for every car. And I want the latest journey for each car. Because the time range is flexible (and thus can't be known beforehand) we can't implement a isMostRecent flag, because while it might be the most recent for now, it might not be the most recent for the specified date parameters. We also need to ensure that this returns promptly (current queries are around the 3-5 second mark for a small set of data) as this goes straight back to the user. This means that we can't use task queues, and because the specified dates are arbitrary we can't implement mass indexing of isWithinDate fields. We tried using an async query, but because the amount of processing is negligible the bottleneck is still the queries on the datastore (because the async api still sends the requests synchronously, it just doesn't block). Ideally, we'd implement this as a select on car journeys ordered by startDate where the Car.key is distinct, but we can't seem to pull this off in GAE. There are numerous small optimisations we can make (for example, some MemCaching of repeated queries) but none have made a significant dent in our query time. And MemCaching can only help for a maximum of 1-2 minutes (due to the inevitable forward march of time!) Any ideas are most welcome and highly appreciated. Thanks, Ed -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to
[google-appengine] Re: Having a problem with 2 Legged OAuth
The gdata java lib is a tricky one. Lots of things can go wrong, resulting in the same error message. Just a guess: are you adding the xoauth_requestor_id parameter to the feed URL? It's value should be the emailaddress of an administrator of the apps domain. On Feb 14, 4:43 pm, Cherny D. C. Berbesí I. cherny.berb...@gmail.com wrote: Hi guys, I getting Authorization required when I try to do: client = gdata.contacts.service.ContactsService(source=APPLICATION_SOURCE) client.auth_token = gdata.gauth.TwoLeggedOAuthHmacToken(CONSUMER_KEY, CONSUMER_SECRET, requestor_id) I'm doing this on a task queue, I'm not sure why I get this error, I'm pretty sure that the parameters are right. How can I make another test or debug of this? Thanks in advance -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] 301 Too many redirects
servlet servlet-nameDirectory/servlet-name servlet-classcom.x.leaders.Directory/servlet-class /servlet servlet-mapping servlet-nameDirectory/servlet-name url-pattern/directory/url-pattern /servlet-mapping welcome-file-list welcome-filedirectory.jsp/welcome-file /welcome-file-list The very first time I look at this app each day I receive a 301 Error Too Many Redirects. After I refresh it will load fine every time afterward. The jsp is is the root of the war file so I am not sure why I am getting that redirect error. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] DeadlineExceededError blobstore.FetchData()
Suddenly my app started showing this error (after only around 5 seconds of processing) when attempting to pull 500k PDF files from the blobstore, to serve through a custom handler: *DeadlineExceededError: The API call blobstore.FetchData() took too long to respond and was cancelled. * Occasionally the PDF gets delivered, but 80% of the time it appears to throw the error. I suppose trapping the error is one course of action, but I'm puzzled why the deadline is being exceeded so quickly. Is this expected behavior? The files aren't that big, and 5 seconds isn't that long in my opinion. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] _AE_DatastoreAdmin_Operation Entities active_jobs
There is nothing failing ... I guess the good news is the jobs showing as active are not really running I tried to delete the entire datastore to duplicate what i can do on the GoogleAppEngineLauncher via info (clear datastore on launch) the datastore appears to be deleted but _AE_DatastoreAdmin_Operation Entities continues to show active jobs On Mon, Feb 14, 2011 at 9:31 PM, Robert Kluin robert.kl...@gmail.comwrote: Hi James, Check your logs and task-queue. Are there tasks that are failing? Are you actively adding data to the kinds you're trying to delete? If you're done running jobs, deleting the _AE_DatastoreAdmin_Operation entities should not hurt anything. Robert On Mon, Feb 14, 2011 at 11:34, James Gilliam jimgill...@gmail.com wrote: I used the datastore admin feature to delete all of my entities ... it fires off one job for each entity ... some of the jobs end quickly and others never end ... days later, it still shows active_jobs outstanding ... not sure what to do about it ... ? also, the datastore shows the table _AE_DatastoreAdmin_Operation as an entity with rows and was wondering if i should delete those as well ... ? -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: DeadlineExceededError blobstore.FetchData()
I've been getting random deadline exceed errors too. Don't know what the deal is... would be nice if there were better ways of debugging them. On Feb 15, 2:08 pm, Carlos carlos.p...@gmail.com wrote: Suddenly my app started showing this error (after only around 5 seconds of processing) when attempting to pull 500k PDF files from the blobstore, to serve through a custom handler: *DeadlineExceededError: The API call blobstore.FetchData() took too long to respond and was cancelled. * Occasionally the PDF gets delivered, but 80% of the time it appears to throw the error. I suppose trapping the error is one course of action, but I'm puzzled why the deadline is being exceeded so quickly. Is this expected behavior? The files aren't that big, and 5 seconds isn't that long in my opinion. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: DeadlineExceededError blobstore.FetchData()
Are yours related to blobstore.FetchData() too? Or a different function? -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] startsWith multiple times
Hi Jan, Based on the information you provide its hard to say if that's a good idea or not; perhaps for your problem it is the best way, maybe it is not. If you are going to run multiple queries, I would try to run them asynchronously to reduce the latency. The idea is not necessarily without merit, Google for zig-zag merge join. You can watch the Building scalable, complex apps on App Engine IO talk for a nice description of it. You need to make sure you take into account how the results will be ordered when they're returned to you. You should also consider the sizes of the result sets. If you want alternate suggestions you should provide a description of the (specific) _problem_ not only the solution. ;) Robert On Tue, Feb 15, 2011 at 05:28, Jan jan.morl...@googlemail.com wrote: In our company we feel obstructed by the restriction that a query may only contain one inequality filter like startsWith. Currently we are thinking about performing two independent startsWith queries and combining the result inside the service method. Do you think this is a good idea? Does anybody have experience with such a strategy or even know a better solution? Thank you very much Jan -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Tasks and rate- throttled urls
I have generated some tasks which call the same URL with different parameters. Unfortunately the URL is rate throttled - I can only make one call every 10 secs - so I want to configure queue.yaml to make one call every 10 secs. I started by setting the rate parameter at 6/m, but all the tasks were executed immediately. On closer reading of the queue.yaml docs, I can see there's nothing to stop app engine executing 6 tasks per minute in a single burst (rather than every 10 secs). The best I can do at the minute is to use a rate of 1/m, so a task is executed every 60 secs; anything more than 1/m triggers the URL throttle. Is there some combination of queue.yaml params that can get the interval down to 10 secs ? -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Tasks and rate- throttled urls
Hi Justin, Have you tried 0.1/s as a rate? -Nick On Wed, Feb 16, 2011 at 10:40 AM, Justin justin.worr...@gmail.com wrote: I have generated some tasks which call the same URL with different parameters. Unfortunately the URL is rate throttled - I can only make one call every 10 secs - so I want to configure queue.yaml to make one call every 10 secs. I started by setting the rate parameter at 6/m, but all the tasks were executed immediately. On closer reading of the queue.yaml docs, I can see there's nothing to stop app engine executing 6 tasks per minute in a single burst (rather than every 10 secs). The best I can do at the minute is to use a rate of 1/m, so a task is executed every 60 secs; anything more than 1/m triggers the URL throttle. Is there some combination of queue.yaml params that can get the interval down to 10 secs ? -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Nick Johnson, Developer Programs Engineer, App Engine Google Ireland Ltd. :: Registered in Dublin, Ireland, Registration Number: 368047 -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Cleanup All for GAE mapreduce
I have adapted the source code for mapreduce to add a Cleanup All function to the UI. I only modified the code for the python part of mapreduce. You can find the code and a blog post at http://djidjadji.appspot.com/2011-02-15-Cleanup-All-for-GAE-mapreduce -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] App Engine SDK 1.4.2 is out!
just for posterity, can you guys confirm that register_template_library() worked without any problems in 1.0 1.1 (show original snippet?), and only broke with 1.2? (i was under the impression that it has always taken a module name [rather than a path].) thanks! -- wesley - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Core Python Programming, Prentice Hall, (c)2007,2001 Python Fundamentals, Prentice Hall, (c)2009 http://corepython.com wesley.chun : wesc+api at google.com : @wescpy developer relations :: google cloud products -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: LIFO key name strategy?
Why not just use regular timestamps, and sort descending? -Nick On Wed, Feb 16, 2011 at 1:42 AM, Joseph Letness joe.letn...@gmail.comwrote: Julian, that is _exactlty_ what I was looking for. Counting up to a future time to create descending values... It seems so obvious now! It's a much better solution than the de-accumulator handler that I wrote. Thanks! On Feb 14, 11:57 pm, Julian Namaro namarojul...@gmail.com wrote: I am not sure about the mathematics of it, but intuitively there is no perfect algorithm for constructing timestamps in a reverse lexicographical ordering, because adding a character to a string will always make it lexicographically superior. But I noticed the mapreduce library just pick a ridiculously future time and go in reverse order from there: http://code.google.com/p/appengine-mapreduce/source/browse/trunk/pyth... The library also add a random string to reduce the chance of duplicates, maybe that can be replaced by an UUID if you're really concerned by uniqueness. On Feb 14, 5:57 am, Joseph Letness joe.letn...@gmail.com wrote: Hi Calvin and Robert, thanks for your replies. I should have been more clear about what I am doing, here is some more info: Calvin, thanks for the link to Ikai's blog post, I haven't seen that one and it was very interesting. Robert, here are specific answers to your questions: Why do you say: I can't use a composite index since it would explode with my use case? I'm using Brett Slatkin's Relation Index method of building and querying set memberships (Google I/O 2009 - Building Scalable, Complex Apps on App Engine). According to Brett, using a composite index on this kind would cause explosion, so any ordering of results will need to be done in-memory during the request. If the sort order is immutable, sorted key names can be used to order results based on the their lexicographical position. Since a creation timestamp is immutable data, I figured that using lexicographic key names would be the way to go. What would be fine if you could handle your entire result set in one request? Ordering the result set in-memory. What are you trying to do? The app is a digital-asset manager. Users need to be able to query a set (using the relation index method) and have the results return the most recent additions first. The result set could easily be a few thousand, so I want to use cursor-pagination to display the results which would preclude any in-memory ordering. (I'm actually refactoring my existing app that I use to manage/deliver graphic assets to my clients so that they can add their own data.) Is there a single global LIFO stack, or are there multiple stacks? The entities are all of the same kind, however, LIFO behavior is localized to individual user groups. How are new items added to the stack(s)?, What is the addition rate? Just one item per user request. User groups would be just a few individual users probably less than twenty. The rate per group would be so low that chances of contention on any sort of accumulator would be almost nonexistent. Is there a requirement that the items are precisely ordered or are some (or small) mis-orderings acceptable? Precision is NOT critical. Close approximation of chronology is just fine. --The auto-generated ids are not strictly increasing I did not know that. Thanks! --Using the current time may also be problematic since the machines will have slight variations, and in some cases significant variations. I was aware of that, but since absolute precision is not necessary I could still use the timestamp as an accumulator if there is some thing as an inverse-timestamp algorithm!?!? So... After spending some more time thinking about this, here is what I plan to do: Create a counter model kind that is created with an IntegerProperty starting value of ten billion (I'd like to see somebody reach the bottom of that!). Give each user group it's own counter and de-count the values in a transaction (or not, it might be simpler to dismiss contention and write a handler that ensures uniqueness of the key name but maintains approximate lexicographic position). When the counter value is read, convert the value to a padded string and concatenate it with the user group name and a leading lowercase letter (k999836/ usergroupname) and use that as the key name for the new asset. Furthermore, it occurred to me that as a result set is reduced to a manageable in-memory size, I could test for the length of results and offer the user the ability to custom order their results (asset name alphanumeric or asset kind, for example). Just a thought. Thanks again for the replies, If anyone thinks there is a better approach please let me know, I kind of make this
Re: [google-appengine] Inviting another developer
Thanks! I think the problem was that the e-mail I was using is the primary admin on my Google Apps account. I tried a different e-mail address and it worked fine. Thanks. Michael -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: Delivering SVG dynamically (through XFORMS)
WORKAROUND found! Set the mimetype for the SVG static files explicitly by including * mime_type: image/svg+xml *in the .yaml specifications -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: everything is down
There were really way too much error thrown from the datastore for the past few days. My app is having 8% errors among all requests for the past 18 hours. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] My app is completely down, log shows Deadline Exceeded Errors on datastore operations
This started at 2011-02-15 18:16:42.285 PST Anybody else experiencing a similar issue right now? Just filed case #4584 Dashboard shows that everything is fine with the datastore Thanks -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: My app is completely down, log shows Deadline Exceeded Errors on datastore operations
+1, just got a sudden burst of over 70 errors in the past 20 minutes. Normally I just get 1 or 2 every other hour... On Feb 15, 9:38 pm, PK p...@gae123.com wrote: This started at 2011-02-15 18:16:42.285 PST Anybody else experiencing a similar issue right now? Just filed case #4584 Dashboard shows that everything is fine with the datastore Thanks -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: My app is completely down, log shows Deadline Exceeded Errors on datastore operations
Yep, and now a steady stream of DeadlineExceededExceptions... right on schedule GAE! Guess I should have expected this; only been a week or so since this last happened... I better get to checking my support email, probably some frustrated customers by now. You think they'd be used to it by this point :D On Feb 16, 2:47 am, coltsith conla...@gmail.com wrote: +1, just got a sudden burst of over 70 errors in the past 20 minutes. Normally I just get 1 or 2 every other hour... On Feb 15, 9:38 pm, PK p...@gae123.com wrote: This started at 2011-02-15 18:16:42.285 PST Anybody else experiencing a similar issue right now? Just filed case #4584 Dashboard shows that everything is fine with the datastore Thanks -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: My app is completely down, log shows Deadline Exceeded Errors on datastore operations
The problem is now gone. My monitoring logs show that the outage took place for about 15 minutes from 2011-02-15 18:16:42 to 2011-02-15 18:31:09. In general, I have been having a very good experience with GAE and this was an unpleasant surprise. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: App Engine SDK 1.4.2 is out!
Hi Colin, We currently do not have plans to allow users to disable the mechanism we use to lower the latency of tasks. On 14 February 2011 23:50, Colin Hawkett hawk...@gmail.com wrote: Thanks Greg - would it be a future possibility to configure the queue algorithm not to do this (on a per-queue basis)? Obviously the low latency heuristic is a good one in most situations, but an app developer might be prepared to wear a slight variation in latency to get guaranteed serialisation. I'm guessing that there are other edge cases which make guaranteeing serialised execution tricky (beyond the task selection algorithm). -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Greg Darke -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: everything is down
@David: +1: My exact thoughts! I find that at least once every 2 - 3 weeks GAE will go down like this for hours at a time. @Everyone: Unless the GAE team improves general uptime, avoid app engine like the plague! We're only maintaining our last project since it uses app engine (unfortunate that we're also maintaining a few thousand paid customers) but our next endeavor is already being developed using Amazon Web Services... On Feb 16, 2:31 am, Raymond C. windz...@gmail.com wrote: There were really way too much error thrown from the datastore for the past few days. My app is having 8% errors among all requests for the past 18 hours. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Tasks and rate- throttled urls
Hi Justin, If you change your queue's bucket size to be 1 it will give you the semantics you are after. On 16 February 2011 10:40, Justin justin.worr...@gmail.com wrote: I have generated some tasks which call the same URL with different parameters. Unfortunately the URL is rate throttled - I can only make one call every 10 secs - so I want to configure queue.yaml to make one call every 10 secs. I started by setting the rate parameter at 6/m, but all the tasks were executed immediately. On closer reading of the queue.yaml docs, I can see there's nothing to stop app engine executing 6 tasks per minute in a single burst (rather than every 10 secs). The best I can do at the minute is to use a rate of 1/m, so a task is executed every 60 secs; anything more than 1/m triggers the URL throttle. Is there some combination of queue.yaml params that can get the interval down to 10 secs ? -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Greg Darke -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: everything is down
Taking a decision of leaving is actually a hard call. There are many things involved ( + Business decisions) I guess everything falls at what you expect from your service to end users: 99.99% uptime? That's _expensive_ everywhere (or eats everyones life at the sysadmin level resetting load balancers, memcached pools, dealing with transactions deadlocks, etc) If appengine team comes up with the SLA things will change drastically, because most of the people wanting this SLA will have to: - redesign the app so it meets best practices ( avoid relational mindset) - move to the HR so you can assure the datastore latency is consistent and you plan around it - have better visibility of your internal app: what's so time consuming, error rate per entity kind, etc. There are not many PaaS that satisfy all my needs, google at least has a roadmap and you can expect the enterprise jump soon (if you really want a 99.9% at least) For now, we try to make sure our app is not the one making all those stability issues :) On 15 February 2011 21:07, coltsith conla...@gmail.com wrote: @David: +1: My exact thoughts! I find that at least once every 2 - 3 weeks GAE will go down like this for hours at a time. @Everyone: Unless the GAE team improves general uptime, avoid app engine like the plague! We're only maintaining our last project since it uses app engine (unfortunate that we're also maintaining a few thousand paid customers) but our next endeavor is already being developed using Amazon Web Services... On Feb 16, 2:31 am, Raymond C. windz...@gmail.com wrote: There were really way too much error thrown from the datastore for the past few days. My app is having 8% errors among all requests for the past 18 hours. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- http://about.me/david.mora -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] App Engine SDK 1.4.2 is out!
Hello Mr. wesley,
webapp.template.register_template_library('customfilters')
This is the only way we could import register_template_library,
customfilters.py file on the root level of the app, access from other paths
is not available.
We used the default package django 0.96 didnt try other version
now for 1.2 we altered the file structure to customfilters\customfilters.py
and added a init file..
webapp.template.register_template_library('customfilters.customfilters')
using this way making it work fine for 1.2
the previous mechanism didnt work for 1.2, and is raising errors.
On Wed, Feb 16, 2011 at 5:20 AM, Wesley C (Google) wesc+...@google.comwrote:
just for posterity, can you guys confirm that
register_template_library() worked without any problems in 1.0 1.1
(show original snippet?), and only broke with 1.2? (i was under the
impression that it has always taken a module name [rather than a
path].)
thanks!
-- wesley
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Core Python Programming, Prentice Hall, (c)2007,2001
Python Fundamentals, Prentice Hall, (c)2009
http://corepython.com
wesley.chun : wesc+api at google.com : @wescpy
developer relations :: google cloud products
--
You received this message because you are subscribed to the Google Groups
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.
--
Regards
Sandeep Koduri
Phone: +91- 99 666 02 456
Gtalk: sandeep.koduri | Skype: sandeep.koduri
--
You received this message because you are subscribed to the Google Groups
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] A difficult app engine optimisation problem - selecting distinct entities across a large table
I am going to post two replies as I have a small idea that might help a little bit, and then a further post on the whole problem in general. First, Ed, what you have said is consistent and the problem of identifying the most recent Journey within a constrained time window does appear to imply looking at all Journeys. So the question comes up, is there a way to exclude some of the Journeys? Is there perhaps some relationship between Journeys that has not been mentioned yet that could be used to make the data set smaller? The only thing I came up with is the relationship to the search fidelity/granularity. As stated above, there is no differentiation _within_ one minute of time, and further, the boundary of that minute is well defined. So, for any GIVEN minute, the system can at least determine, prior to search, that some Journeys are not the latest in that minute. Therefore, every time a new Journey starts, the system can (asynchronously of course) get all other Journeys that started in that same minute, and mark them as possibly_latest = false. By default, all Journeys are possibly_latest = true. And your search now includes this extra term, to narrow the results. [and one would use the obvious optimization of storing the minute on the Journey as an integer to make this search faster] And of course, the system should also have a cron job that runs at some appropriate frequency to mark all Journeys that are more than 2 months old as possibly_latest = false. This is given the other constraint that the oldest searched Journeys start should be no more than 2 months ago. This is no magic solution. But depending on the data, this could significantly reduce the number of Journeys that are searched (through index). -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] A difficult app engine optimisation problem - selecting distinct entities across a large table
I also wanted to add this comment which is probably not helpful, but speaks more to the root of the problem. :) And that is this, you can't get there from here. The fundamental problem does not appear capable of being reduced beyond looking at a bunch of Journeys. I'm going to summarize a couple of things here, but in the end it just comes down to look at a bunch of data ... unless you change something else. A quick summary of some points: 1. You can de-normalize Car fields to Journey as it relates to this search. So the problem does reduce, slightly, to looking for Journeys. The two lookup aspect of it should not be much of a problem. I know this was mentioned above. 2. You can try to squash aspects of your search into the Key as Stephen suggested. This is a good idea but it does not change the nature of the problem - looking at a whole lot of entities, if only the keys of those entities. These other possibilities are excluded from being viable because of the problem description. 3. Make the processing more parallel by turning it into a counting problem and using a map reduce approach or similar. This is the same as changing the requirements from query to report in practical terms. There is also the sticky issue of having a reduce implementation on app engine right now. 4. Changing the search criteria to be more restricted in some way that introduces a further relationship between Journeys. This additional information could probably be used to make the search perform better. In fact, one would engineer it in such a way, there would probably be no need to make a change in the way this search worked unless it resulted in this benefit. 5. There is also the break the fourth wall kind of solution. Take this part of the problem off of app engine. You would pay a lot more for this. Both in terms of complexity and cost (and maybe some other things too). It may not be realistic, but how realistic is the alternative? I hope you will report back at some stage and tell us. (this might allow for a truly parallel query for example ... come on Datastore Plus!) What I am getting at is that the system probably needs a higher level change to achieve the desired goal given the constraints that exist at this time. That is the you can't get there from here. I think this has been virtually proven at this point though I would be happy to be proved wrong. It comes to trade-offs like everything. You can often trade space for speed, etc. etc. But it is often a trade. So you may need to look at trades between the existing use cases / user expectation and speed. Thanks Ed for sharing with the group. This has been a good discussion and interesting to think about. -- Jay -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: everything is down
99.99% uptime? That's _expensive_ everywhere I'd settle for even 90% right now. Today I'm nearing a 25% error rate in the past 22 hours. That's pretty pathetic for a service. If it was just one day I'd understand, but lately it's been nearly every day where something has gone haywire (usually stemming from datastore latency) without me changing any code. David -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: everything is down
well, i also agree to that. I mean, my position is really clear. Im with you about the performance just after the maintenance period (like right now im starting to reply to all the complains for the last 4 hours today) - has been the worst in the last 4 months. was just being clear with a wider open general view :) On 16 February 2011 00:19, David s2kd...@gmail.com wrote: 99.99% uptime? That's _expensive_ everywhere I'd settle for even 90% right now. Today I'm nearing a 25% error rate in the past 22 hours. That's pretty pathetic for a service. If it was just one day I'd understand, but lately it's been nearly every day where something has gone haywire (usually stemming from datastore latency) without me changing any code. David -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- http://about.me/david.mora -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: My server can't reach app engine for being in a forbidden country (it's in Atlanta)
In my opinion, Google, Facebook etc, they don't really care On Feb 15, 4:29 pm, Max Harper maxhar...@gmail.com wrote: Google fixed this issue last night. Thanks to the Google engineers who addressed the problem. I'm not sure this would have been addressed if we hadn't had a friend internal to Google who looked into the issue for us. I'm still frustrated that the moderators of this board never replied. I suppose it's difficult to separate real issues (i.e., something's broken) from spurious complaints (i.e., I can't get my code to work). Max On Feb 11, 4:40 pm, Max Harper maxhar...@gmail.com wrote: Our VPS server (in Atlanta, GA, USA) appears to be blacklisted by app engine (403 errors, forbidden country), causing our app to be functionally down. Help? Details: We stream data to our app engine app from a Linode server, hosted in Atlanta, GA. Our ip address is 74.207.230.***. As of 19 hours ago, we're getting 403s when trying to access pages on our app engine app. Lynx shows: Your client does not have permission to get URL / from this server. (Client IP address: 74.207.230.***) You are accessing this page from a forbidden country. Log messages (403, or otherwise) do not make it to the app engine app console. wget also shows 403s for any of my app engine apps. For instance: max@nea:~/tmp$ wgethttp://maxharp3r-sandbox.appspot.com --2011-02-11 16:32:18-- http://maxharp3r-sandbox.appspot.com/ Resolving maxharp3r-sandbox.appspot.com... 74.125.67.141 Connecting to maxharp3r-sandbox.appspot.com|74.125.67.141|:80... connected. HTTP request sent, awaiting response... 403 Forbidden 2011-02-11 16:32:18 ERROR 403: Forbidden. These urls all work when I'm accessing from my local development machine (in Minneapolis, MN, USA). I'm not quite sure where to turn, since app engine has essentially no support contact. Thanks for your suggestions and help. Max -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
