[appengine-java] Re: [google-appengine] Re: CMS for GAE

[Michel Schroeder]

Thanks for your explications.

PS
I don't want to start a discussion but my excerpt is clear, with the
package:
package com.claymus.site.module.page;

and the definition of the class:

@PersistenceCapable(detachable = true)
public class Page extends PersistentCapable {

This page is clearly a Data Access Object (not a servlet) and one off
it's method 'serve' output html.


On Jul 22, 8:18 pm, Prashant antsh...@gmail.com wrote:
 Hello Michel,

 Thanks for checking out the code and the query :)

 On Thu, Jul 21, 2011 at 9:49 PM, Michel Schroeder michels...@gmail.comwrote:

  Hello,

  Thanks for your contributions for a GAE-Java CMS.

  I had a look at the code of claymus. It seems that you use the
  combination of gae and gwt, this is a very interesting point.

  But on the client side, some architectural questions come to my
  mind...

  I don't understand your separation of code responsability?

 Code separation is done considering mainly, modularity and pluggability of
 modules. All the classes related to a module are placed in a package (and
 sub-packages) meant for that module. Modules are designed such that it won't
 affect the functioning of App or other modules if a new module is added or
 an old one (which is not a core module) is removed from the App.

 Up to now, I don't see something like MVC or MVP or UI Builder ... It

  seems that your write the html output directly from java (and that in
  the same file that the DAO definition if I am not wrong)...

 Yes, I didn't try to follow MVC or MVP strictly because I don't think that
 is a strict requirement. I din't use UI Builder because none of the GWT
 modules have very big and/or complex UI. If you are worried about supporting
 mobile devices, that can supported just by creating a mobile theme (which
 will include layout and look  feel) and few tweeks in Content and Block
 types.

 I think you are thinking of following methods as Data-Access methods. They
 are not Data-Access methods, they are just converting Data Object to Data
 Transfer Object. PageData class, in the same package, is the DAO for page
 module.

 public PageDTO getDTO() {
 PageDTO pageDTO = new PageDTO();
  pageDTO.setUri(getUri());
 pageDTO.setTitle(getTitle());
 pageDTO.setLayout(getLayout().getClass().getSimpleName());
  return pageDTO;

 }

 public void update(PageDTO pageDTO) {
  setTitle(pageDTO.getTitle());
 setLayout(pageDTO.getLayout());

 }

 And, I don't see any thing wrong in outputting html from a java servlet.

 Could you explain a bit the advantages of your strategy?



 I am not following popular design architectures, like MVP and MVC, that may
 have some disadvantages that I am not aware about now.

 I am trying to support maximum modularity and plaggability like any other
 CMS. e.g. if you have to add a new Content or Block type, you just need to
 write 2 small classes, one for backend and one for front-end (see
 com/claymus/site/module/block/types/RichText.java and
 com/claymus/site/module/block/types/gwt/RichText.java) and drop them in to
 Content/Block Types package. Same for a theme.







  A lot of people could think that a bad separation of responsibilities
  will lead to a lot of difficulties in code maintenance and
  evolution...

  But it must be subtilities in your approach that I don't catch up to
  now...?

  Thanks in advance for your explanations,

  Michel

  PS:
  I look for example in:
  package com.claymus.site.module.page;
  (...)
  @PersistenceCapable(detachable = true)
  public class Page extends PersistentCapable {
  (...)
         public void serve(ListListContent contents, ListListBlock
  blocks, Theme theme) throws ServletException, IOException {

                 PrintWriter out = ClaymusMain.getResponse().getWriter();

                 // DOC TYPE
                 out.print(getDocType());

                 out.print(html);
  (...)

  On Jul 18, 5:50 pm, Prashant antsh...@gmail.com wrote:
   Yes, you are right. Picking an opensource project and making it work for
  you
   is very easy, but making something from scratch has its own fun :) . I
   stared building Claymus because I felt need of Java based CMS which is
  made
   for GAE, considering what GAE is, how it is different from other hosting
   services, services/apis GAE is providing to developers, etc. etc. etc.

   Claymus is designed to give you maximum flexibility and support to build
   your app on top of Claymus. Along with all the GAE features you can take
   advantage of Servlet Level caching to minimize delay and cpu usage,
  plugable
   modules and themes, etc ... (complete list will be put on updated Claymus
   Website :) ).

   Regards,
   Prashant

   On Mon, Jul 18, 2011 at 7:28 PM, Niklas Rosencrantz nikla...@gmail.com
  wrote:

Having something like a CMS eg like wordpress, joomla or mediawiki
compatible then we won't have to start every project from scratch and
project would become more configuration than programming which
  naturally is
an 

[appengine-java] java.lang.NoClassDefFoundError: for sample project ( CodeLabEx2.zip.)

[bgapps]

Hi,

I am using sdk 1.5.1 to run one sample google application/project
CodeLabEx2.zip.

WARNING: Error for /product
java.lang.NoClassDefFoundError: com/google/appengine/api/datastore/
EntityNotFoundException
at com.google.appengine.codelab.Product.getAllProducts(Product.java:
59)
at
com.google.appengine.codelab.ProductServlet.doGet(ProductServlet.java:
54)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:617)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:
511)
at org.mortbay.jetty.servlet.ServletHandler
$CachedChain.doFilter(ServletHandler.java:1166)
at
com.google.appengine.tools.development.HeaderVerificationFilter.doFilter(HeaderVerificationFilter.java:
35)
at org.mortbay.jetty.servlet.ServletHandler
$CachedChain.doFilter(ServletHandler.java:1157)
at
com.google.appengine.api.blobstore.dev.ServeBlobFilter.doFilter(ServeBlobFilter.java:
58)
at org.mortbay.jetty.servlet.ServletHandler
$CachedChain.doFilter(ServletHandler.java:1157)
at
com.google.apphosting.utils.servlet.TransactionCleanupFilter.doFilter(TransactionCleanupFilter.java:
43)
at org.mortbay.jetty.servlet.ServletHandler
$CachedChain.doFilter(ServletHandler.java:1157)
at
com.google.appengine.tools.development.StaticFileFilter.doFilter(StaticFileFilter.java:
122)
at org.mortbay.jetty.servlet.ServletHandler
$CachedChain.doFilter(ServletHandler.java:1157)
at
com.google.appengine.tools.development.BackendServersFilter.doFilter(BackendServersFilter.java:
97)
at org.mortbay.jetty.servlet.ServletHandler
$CachedChain.doFilter(ServletHandler.java:1157)
at
org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:
388)
at
org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:
216)
at
org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:
182)
at
org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:
765)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:
418)
at
com.google.apphosting.utils.jetty.DevAppEngineWebAppContext.handle(DevAppEngineWebAppContext.java:
70)
at
org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:
152)
at com.google.appengine.tools.development.JettyContainerService
$ApiProxyHandler.handle(JettyContainerService.java:351)
at
org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:
152)
at org.mortbay.jetty.Server.handle(Server.java:326)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:
542)
at org.mortbay.jetty.HttpConnection
$RequestHandler.headerComplete(HttpConnection.java:923)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:547)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
at
org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:
409)
at org.mortbay.thread.QueuedThreadPool
$PoolThread.run(QueuedThreadPool.java:582)
Caused by: java.lang.ClassNotFoundException:
com.google.appengine.api.datastore.EntityNotFoundException
at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
at
com.google.appengine.tools.development.IsolatedAppClassLoader.loadClass(IsolatedAppClassLoader.java:
176)
at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
... 33 more
Jul 25, 2011 2:43:11 PM
com.google.appengine.tools.development.LocalResourceFileServlet doGet
WARNING: No file found for: /favicon.ico


Could you please let me know for same.

Thanks

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine for Java group.
To post to this group, send email to google-appengine-java@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine-java+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine-java?hl=en.

[appengine-java] Re: Development server deletes automatically-generated datastore indices

[peter]

I've noticed the same problem today. I hope the SDK will be updated to
make life easier for developers.

On Jul 1, 9:29 am, Ian Marshall ianmarshall...@gmail.com wrote:
 Yes, the essence of what you describe below is what I plan to do -
 what a pain, though.

 It's a pity that the dev app server doesn't look at the automatically-
 generated indices at launch-time, but I'll live with it.

 On Jul 1, 8:22 am, Cyrille Vincey cvin...@qunb.com wrote:







  Yes, I have noticed the same behaviour.
  My workaround: once a index has been automatically generated, I copy
  its xml configuration from datastore-indexes-auto.xml to datastore-
  indexes.xml.
  Thus I consider  datastore-indexes-auto.xml as automatic
  propositions.

  On Jun 30, 12:14 pm, Ian Marshall ianmarshall...@gmail.com wrote:

   I have my datastore-indexes.xml configuration file set as

     datastore-indexes autoGenerate=true
     /datastore-indexes

   I make successive runs of the development server. Each time before
   stopping it, I wait a couple of minutes for updates to be flushed to
   local_db.bin and datastore-indexes-auto.xml.

   I have found that successive runs of the development server can cause
   previously-existing automatically-generated datastore indices to be
   erased (even though the datastore data in local_db.bin is preserved).

   Has anyone else found this? Is this expected behaviour or a glitch?

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine for Java group.
To post to this group, send email to google-appengine-java@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine-java+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine-java?hl=en.

[appengine-java] Running UI tests against AppServer in a CI environment

[Kyle Baley]

Has anyone set this up? We're running UI tests against our app using 
Cucumber and Capybara and it works great locally. But I'd like to get it 
done in a continuous integration environment. The major stumbling block as 
that I can't seem to start the server process (DevAppServerMain) and have it 
outlive Ant. I'm using a modified runserver target because of issues with 
a hanging java.exe process (outlined here: 
http://code.google.com/p/googleappengine/issues/detail?id=2093).

My target is based on one from that thread:

import file=${appengine.folder}\config\user\ant-macros.xml /
target name=runserver description=Starts the development server.
java 
classname=com.google.appengine.tools.development.DevAppServerMain
  classpath=${appengine.tools.classpath}
  fork=true failonerror=true
  jvmarg 
value=-javaagent:${appengine.folder}/lib/agent/appengine-agent.jar/
  arg value=--address=localhost/
  arg value=--port=/
  arg value=war/
/java
/target

This runs but it blocks waiting for you to cancel it manually. If I add 
spawn=true to the java task, I get an error that it's not allowed.

Note that I'm doing something similar with a GWT/GAE project successfully. 
In that project, I launch hosted mode with spawn=true and it runs fine. This 
project is AppEngine only so I'm not using GWT or hosted mode.

Any ideas?

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine for Java group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine-java/-/q4Z78GjokQIJ.
To post to this group, send email to google-appengine-java@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine-java+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine-java?hl=en.

[appengine-java] How to handle SNMP traps on appengine?

[Kesava Neeli]

Hi,

I just wanted to know what's the best process to handle SNMP traps from the 
application. Our app will be doing some critical third party backend hits 
and we need to alerts/notifications on any errors. One option is sto end 
SNMP traps to some SNMP sever outside appengine but I don't want to have 
SNMP server installed in EC2 or any other cloud just for that. I could send 
email notification on error conditions but that is lame. Any suggestions? 

Thanks
Neeli 

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine for Java group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine-java/-/Gf_BGTf3opkJ.
To post to this group, send email to google-appengine-java@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine-java+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine-java?hl=en.

[appengine-java] deployment to production stalling

[Nichole]

Hello,

   Anyone having troubles deploying to appengine production?

   My deployments are currently stalling out to production
since Friday July 21 though they still succeed to development.
I didn't change my java jdk binaries, didn't change the appengine
sdks that I was using, and had previously successfully
deployed the app using the same configuration and builds,
even after the July 14th maintenance.

   To troubleshoot, I tried installing and building from variations
of java jdks, variations of appengine sdks (fresh downloads and
existing that I already had) and had the same stalls during
attempts to upload to production.

  I ran the deployment with java verbose options to print the
classes loaded and then created aspects and used load time
weaving to see the methods where the deployment is stalling.
I can print them out here if it's helpful, but it looks to be a bug
in the appengine-tools-api.jar

  Has anyone else had the same problem recently?  If so, how
did you solve it?

Thanks,
Nichole

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine for Java group.
To post to this group, send email to google-appengine-java@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine-java+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine-java?hl=en.

Re: [appengine-java] Task queue locking up

[Ikai Lan (Google)]

John, yeah, it's a deadlock, possibly in userspace code. Can you look into
this?

On our end we'll see if we can surface and more usable error message.

Ikai Lan
Developer Programs Engineer, Google App Engine
Blog: http://googleappengine.blogspot.com
Twitter: http://twitter.com/app_engine
Reddit: http://www.reddit.com/r/appengine



On Thu, Jul 21, 2011 at 11:31 PM, John Patterson jdpatter...@gmail.comwrote:

 In case others are facing these time-outs, by setting threadsafe to
 false, all errors stopped and response time does not
 seem significantly different.

 --
 You received this message because you are subscribed to the Google Groups
 Google App Engine for Java group.
 To view this discussion on the web visit
 https://groups.google.com/d/msg/google-appengine-java/-/ofJmJRTpR2oJ.

 To post to this group, send email to
 google-appengine-java@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine-java+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine-java?hl=en.


-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine for Java group.
To post to this group, send email to google-appengine-java@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine-java+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine-java?hl=en.

[appengine-java] GAE Mail Sending Error-

[sats]

Dear All-

Here is a log trace that I get for an piece of code that is trying to send 
out an email.

java.lang.NoClassDefFoundError: com/sun/mail/util/TraceOutputStream
at 
com.google.appengine.runtime.Request.process-1ef8200785964f49(Request.java)
at java.lang.Class.getDeclaredConstructors0(Native Method)
at java.lang.Class.privateGetDeclaredConstructors(Class.java:2406)
at java.lang.Class.getConstructor0(Class.java:2716)
at java.lang.Class.getConstructor(Class.java:266)
at javax.mail.Session.getService(Session.java:483)
at javax.mail.Session.getTransport(Session.java:379)
at javax.mail.Session.getTransport(Session.java:339)
at javax.mail.Session.getTransport(Session.java:368)
at javax.mail.Transport.send(Transport.java:67)
at javax.mail.Transport.send(Transport.java:48)
at 
com.company.trisunkworld.emails.EmailFacade.sendEmail(EmailFacade.java:28)
at 
com.company.trisunkworld.controllers.EmailController.sendEmailHandler(EmailController.java:116)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:43)
at 
org.springframework.web.bind.annotation.support.HandlerMethodInvoker.doInvokeMethod(HandlerMethodInvoker.java:710)
at 
org.springframework.web.bind.annotation.support.HandlerMethodInvoker.invokeHandlerMethod(HandlerMethodInvoker.java:167)

at 
org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter.invokeHandlerMethod(AnnotationMethodHandlerAdapter.java:414)


Any thoughts of I have different in my Classpath or information? I have a 
whole stack of code that is associated, If you could let me know what 
specific code will help you debug this faster, kindly let me know.

-Thanks 

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine for Java group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine-java/-/kyJCWEDpDcMJ.
To post to this group, send email to google-appengine-java@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine-java+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine-java?hl=en.

[appengine-java] Mining logs for performance data

[Kesava Neeli]

Hi,

My application writes timing information for the backend hits in the log 
using standard logger. I can see them in the logs using GAE dashboard 'Logs' 
tab. I would like to run a cron job to get these metrics and have a report. 
How can I mine the application logs? Any suggestions on how to handle this?

Thanks
Neeli 

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine for Java group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine-java/-/z82OQfwZHGUJ.
To post to this group, send email to google-appengine-java@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine-java+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine-java?hl=en.

Re: [appengine-java] Task queue locking up

[John Patterson]

On Tuesday, 26 July 2011 04:03:44 UTC+7, Ikai L (Google) wrote:

 John, yeah, it's a deadlock, possibly in userspace code. Can you look into 
 this?


The only pattern I could detect was the dead-lock with concurrent file 
access - I had my web framework in dev mode so each request was loading 
templates.  I do a lot of reflective reads and writes (Method.get 
Method.set) and annotation reading, which makes me suspect the GAE security 
manager.

I've run a load test on the dev server and not seen the dead-locks locally.


On our end we'll see if we can surface and more usable error message.


Nice one, a stack trace would really help figure this one out.  I'll turn on 
threadsafe mode again once its possible to get some more details on the 
problem.

I think that running single threaded increases the variance of response 
times.  I often now see a request take e.g. 600ms but my code competes in 
50ms.

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine for Java group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine-java/-/YfJhHVZZo70J.
To post to this group, send email to google-appengine-java@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine-java+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine-java?hl=en.

[appengine-java] Re: Mining logs for performance data

[John Patterson]

I've seen a couple of apps (iPad app) that can log into your account and 
monitor various things.

I took a very low-tech approach to average the response time from logs: I 
filter logs by label (e.g. path:/place/.*) and set the limit to 200 (the 
maximum logs you can show in one page) and then copy and past the results 
into a file and run this to average the times:


public class GetAverageTimeFromLogPage
{
public static void main(String[] args) throws MalformedURLException, 
IOException
{
FileReader reader = new FileReader(args[0]);
String text = CharStreams.toString(reader);
long total = 0;
int count = 0;
 Pattern pattern = Pattern.compile( 200 (\\d+)ms);
Matcher matcher = pattern.matcher(text);
while (matcher.find())
{
String time = matcher.group(1);
long value = Long.parseLong(time);
 // remove restarts
if (value = 1000)
{
count++;
total += value;
}
}
 System.out.println(total / count +  from  + count);
}
}

I hope someone has a nicer method!!

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine for Java group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine-java/-/lwLRH0p6XYkJ.
To post to this group, send email to google-appengine-java@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine-java+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine-java?hl=en.

[appengine-java] Re: java.lang.NoClassDefFoundError: for sample project ( CodeLabEx2.zip.)

[John Patterson]

make sure you have all the app engine libs in /WEB-INF/lib

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine for Java group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine-java/-/QTTGn-_VOmAJ.
To post to this group, send email to google-appengine-java@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine-java+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine-java?hl=en.

Re: [google-appengine] Problem using Google Apps Viewer to view a blob

[Nick Johnson (Google)]

On Thu, Jul 21, 2011 at 2:08 AM, Joshua Smith joshuaesm...@charter.netwrote:

 Excellent question. No, I hadn't.

 I just added this:

 self.response.headers['Content-Type'] =
 str(mimetypes.guess_type(document.title)[0]) or 'application/octet-stream'

 and the headers now say:

 HTTP/1.1 200 OK
 Cache-Control: no-cache
 Expires: Fri, 01 Jan 1990 00:00:00 GMT
 Content-Type:
 application/vnd.openxmlformats-officedocument.wordprocessingml.document
 Date: Wed, 20 Jul 2011 16:06:14 GMT
 Server: Google Frontend
 Content-Length: 10762

 But there is no change.  .doc files still work, and .docx files do not.


Determine the actual mimetype that docx files should be served as, and make
sure you're serving as that. Don't expect the mimetype module to know.

-Nick



 Any other ideas?

 -Joshua

 On Jul 19, 2011, at 11:59 PM, Nick Johnson (Google) wrote:

 Have you checked what Mime-Type you're serving the documents as, and if
 it's valid for docx documents?

 -Nick Johnson

 On Wed, Jul 20, 2011 at 9:55 AM, Joshua Smith joshuaesm...@charter.netwrote:

 If you look at this URL:

 http://www.mytowngovernment.org/meeting/1234040

 and click the link for one of the .doc files, they work in google's
 viewer.  But if you click on one of the .docx files, they don't.

 When you click on the link to view, it does this:

   dlurl=re.sub('viewer','download',self.request.uri)
   self.redirect(http://docs.google.com/viewer?url=%s; %
 urllib.quote(dlurl))

 and the download handler is simply:

   documentKey = self.request.get(document)
   document = DocumentModel.get(documentKey)
   self.send_blob(document.blob.key())

 Everything I've read says that google's docs viewer should be able to view
 .docx files, so I'm guessing the problem is in the way the blob is served
 from GAE to Google Apps.

 Any ideas?

 -Joshua

 --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.




 --
 Nick Johnson, Developer Programs Engineer, App Engine



 --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.


  --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.




-- 
Nick Johnson, Developer Programs Engineer, App Engine

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: HR database, ancestor queries and data consistency: docs not 100% clear

[Pol]

On Jul 25, 7:30 am, Nick Johnson (Google) nick.john...@google.com
wrote:

 The first part specifically refers to ancestor queries. Ancestor queries
 always operate on the latest copy of the data, as they're restricted to an
 entity group; regular queries don't have that guarantee.

I understand that, but my problem is with the indexes: even in the
scope of HR + transactions, the doc is confusing regarding wether or
not there can be a race-condition on indexes update.

For instance:
- transaction A modifies entity X and commits
- then transaction B happens almost concurrently:
  - at this point, using db.get() X will return the latest X no matter
what
  - however, using db.GqlQuery() with a predicate that matches X *may
not* return X because the indexes may not be done updating yet (the A -
 B thing in the doc)

So is this race-condition possible or not? No matter what, you might
wanna rewrite a couple sentences in this article :)

  My understanding of this article is therefore the following (assuming
  an HR datastore and that we are inside a transaction):

  1) Using db.get() / db.put() on an entity always executes on its
  latest version: no race conditions possible, period.
  - Observations in my app appears to confirm that behavior.

 Correct, with the usual caveat about transactionality (if you're not doing
 this inside a transaction, you have a potential race condition).

Yes, I was assuming HR datastore + transaction (see above).

  2) Using db.GqlQuery() / db.put() on entities in the same entity group
  might still be subject to race-conditions: for instance, a previously
  *committed* transaction modified the entities that would be returned
  by the query in the later transactions, but the datastore indexes are
  not up-to-date yet (or something like that), so the query doesn't see
  the updated entities - even if calling db.get() directly on them would
  return the versions that match the query.
  - I've reviewed my code carefully and I think I'm seeing such race
  conditions in my app.

 Correct - but again, any update outside a transaction always has possible
 race conditions.

Same here: I was also assuming HR datastore + transaction + ancestor
queries (by definition since inside a transaction).

So is my code wrong or am I observing some very rare but expected race
conditions?

Thanks

- Pol

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Datanucleus Enhancer problem

[Jamil Rzayev]

I am developing GWT-based application with JPA support. I have a
problem with datanucleus enhancer. I accessed repository and installed
eclipse-based datanucleus enhancer plugin but i doesn't work.
It gives the following problem:

java.lang.NoClassDefFoundError: org/datanucleus/enhancer/
DataNucleusEnhancer
Caused by: java.lang.ClassNotFoundException:
org.datanucleus.enhancer.DataNucleusEnhancer
at java.net.URLClassLoader$1.run(Unknown Source)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
at sun.misc.Launcher$AppClassLoader.loadClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
Exception in thread main

How do I solve the problem,?

Is there any other method to work with JPA? Or the only way is to use
enhancer?
I know that we can enhance classes manually but it will take lots of
time.

Any help would be most appreciated

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] TransactionFailedError: Is the transaction committed or not?

[Pol]

Hi,

The exception says: TransactionFailedError: The transaction could not
be committed. Please try again.

The doc at 
http://code.google.com/appengine/docs/python/datastore/transactions.html
says: You can receive Timeout, TransactionFailedError, or
InternalError exceptions in cases where transactions have been
committed and eventually will be applied successfully.

So if you get TransactionFailedError, to you need to execute the
transaction again or will it automatically be applied later?

- Pol

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Fork-Join queue implementation question regarding overlapping joins for same index/batch

[Bert]

Hi,

Based on Brett's talk
http://www.google.com/url?sa=Dq=http://www.google.com/events/io/2010/sessions/high-throughput-data-pipelines-appengine.html,
is it possible to have a scenario where you get 2 join tasks happening
closer than 1 second(or given batch time duration) apart?

A scenario where I think it may happen is as follows:
Batch time 1 second
Stall timeout is 30 seconds
The first insert is added at hh:mm:29.9 (multiple inserts follow)
Task_name is e.g. sumName-1-1237847
The task will only execute at about hh:mm:30.9

At hh:mm:30.0 a new task is inserted but the index remains the same
(the task sumName-1-1237847 hasn't incremented index yet).
The task_name is sumName-2-1237847 and it will execute at hh:mm:31.0
(only 0.1 seconds after the previous task for the same index).

In most cases this may be OK (because 2 updates in a second for an
entity group is probably OK), but what happens if there are many
records for index 1237847 (more than task can finish in 0.1 seconds)?
Potentially task sumName-2-1237847 could start executing before task
sumName-1-1237847 has had time to finish executing, in which case it
will also retrieve the work items for index 1237847 and may sum the
same values twice.

Here's the code (from the slides):

now = time.time()
stallTimeout =30
try:
  taskqueue.add(
  name='%s-%d-%d' % (
  sum_name, int(now / stallTimeout ), index),
  url='/work',
  eta=datetime.datetime.utcfromtimestamp(now) +
  datetime.timedelta(seconds=1))
except taskqueue.TaskAlreadyExistsError:
  pass


def join(sum_name, index):
  # force new writers to use the next index
  memcache.incr('index-' + sum_name)
  lock = '%s-lock-%d' % (sum_name, index)
  memcache.decr(lock, 2**15) # You missed the boat
  # busy wait for writers
  for i in xrange(20): # timeout after 5s
counter = memcache.get(lock)
if counter is None or int(counter) = 2**15:
  break
time.sleep(0.250)
  #get work for index
  #sum or whatever you want
  #delete work

If what I've said is valid, is a workaround to check the lock at join
time to see if its already been decremented by 2**15 before starting
the join as that would indicate it's already been run?

def join(sum_name,index):
  lock = '%s-lock-%d' % (sum_name, index)
  counter =memcache.get(lock)
  if int(counter)  2**16:
#the task for index has already started running, return to prevent
overlap
return False

   #rest of code here...

Thanks
Rob

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: do we need pay for the unnecesssary instances which should be created?

[Francois Masurel]

Hi Tapir,

Check this post 
: https://groups.google.com/d/topic/google-appengine/uvQ5puBz9to/discussion

Still no clear answer from Google about the problem.

Francois

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/oAdzVxWeqrwJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Datastore Admin - Delete didn't work, burned $7.85

[GregR]

I have a Entity with 2,356,292 in it.  I tried using the datastore Admin 
Delete Entities to purge it.  It ran for a while, got a bunch of errors, 
burned $7.85 (mostly in CPU), and didn't actually delete anything.

I've included some of the errors below.

Any idea what's up? I can go back to the old way of deleting, but it would 
take a long time (and CPU) with this # of entities.

Thanks,
-Greg

·  0.1.0.2 - - [24/Jul/2011:13:20:37 -0700] POST 
/_ah/mapreduce/worker_callback HTTP/1.1 500 2394 
http://1-dot-latest-dot-locationplay.appspot.com/_ah/mapreduce/worker_callback;
 
AppEngine-Google; (+http://code.google.com/appengine) 
1-dot-latest-dot-locationplay.appspot.com ms=29696 cpu_ms=306889 
api_cpu_ms=305046 cpm_usd=8.524742 queue_name=default 
task_name=appengine-mrshard-15868331507823D8715B7-11-8 pending_ms=5160 
instance=00c61b117c29bef02c280156561831181c935a2e

*·**  E 2011-07-24 13:20:37.069 *

The API call datastore_v3.Delete() took too long to respond and was 
cancelled.

Traceback (most recent call last):

  File 
/base/python_runtime/python_lib/versions/1/google/appengine/ext/webapp/__init__.py,
 
line 702, in __call__

handler.post(*groups)

  File 
/base/python_runtime/python_lib/versions/1/google/appengine/ext/mapreduce/base_handler.py,
 
line 81, in post

self.handle()

  File 
/base/python_runtime/python_lib/versions/1/google/appengine/ext/mapreduce/handlers.py,
 
line 184, in handle

entity, input_reader, ctx, tstate)

  File 
/base/python_runtime/python_lib/versions/1/google/appengine/ext/mapreduce/handlers.py,
 
line 254, in process_data

output(ctx)

  File 
/base/python_runtime/python_lib/versions/1/google/appengine/ext/mapreduce/operation/db.py,
 
line 87, in __call__

context.mutation_pool.delete(self.entity)

  File 
/base/python_runtime/python_lib/versions/1/google/appengine/ext/mapreduce/context.py,
 
line 171, in delete

self.__flush_deletes()

  File 
/base/python_runtime/python_lib/versions/1/google/appengine/ext/mapreduce/context.py,
 
line 189, in __flush_deletes

datastore.Delete(self.deletes.items, rpc=self.__create_rpc())

  File 
/base/python_runtime/python_lib/versions/1/google/appengine/api/datastore.py, 
line 550, in Delete

return DeleteAsync(keys, **kwargs).get_result()

  File 
/base/python_runtime/python_lib/versions/1/google/appengine/api/apiproxy_stub_map.py,
 
line 592, in get_result

return self.__get_result_hook(self)

  File 
/base/python_runtime/python_lib/versions/1/google/appengine/datastore/datastore_rpc.py,
 
line 1464, in __delete_hook

self.check_rpc_success(rpc)

  File 
/base/python_runtime/python_lib/versions/1/google/appengine/datastore/datastore_rpc.py,
 
line 1072, in check_rpc_success

rpc.check_success()

  File 
/base/python_runtime/python_lib/versions/1/google/appengine/api/apiproxy_stub_map.py,
 
line 558, in check_success

self.__rpc.CheckSuccess()

  File 
/base/python_runtime/python_lib/versions/1/google/appengine/api/apiproxy_rpc.py,
 
line 133, in CheckSuccess

raise self.exception

DeadlineExceededError: The API call datastore_v3.Delete() took too long to 
respond and was cancelled.

 

   1. 
  1.  2011-07-24 13:20:37.282 /_ah/mapreduce/worker_callback 500 24793ms 
879cpu_ms 
  529api_cpu_ms  1kb AppEngine-Google; 
  (+http://code.google.com/appengine) 
  
  0.1.0.2 - - [24/Jul/2011:13:20:37 -0700] POST 
/_ah/mapreduce/worker_callback HTTP/1.1 500 1996 
http://1-dot-latest-dot-locationplay.appspot.com/_ah/mapreduce/worker_callback;
 AppEngine-Google; (+http://code.google.com/appengine) 
1-dot-latest-dot-locationplay.appspot.com ms=24793 cpu_ms=879 api_cpu_ms=529 
cpm_usd=0.024815 queue_name=default 
task_name=appengine-mrshard-15868331507823D8715B7-28-8 pending_ms=9561 
instance=00c61b117cbc93cdc72672caefcebc6c52198bc4
  
  2.  E 2011-07-24 13:20:37.274  
  
  Traceback (most recent call last):
File 
/base/python_runtime/python_lib/versions/1/google/appengine/ext/webapp/__init__.py,
 line 702, in __call__
  handler.post(*groups)
File 
/base/python_runtime/python_lib/versions/1/google/appengine/ext/mapreduce/base_handler.py,
 line 81, in post
  self.handle()
File 
/base/python_runtime/python_lib/versions/1/google/appengine/ext/mapreduce/handlers.py,
 line 184, in handle
  entity, input_reader, ctx, tstate)
File 
/base/python_runtime/python_lib/versions/1/google/appengine/ext/mapreduce/handlers.py,
 line 254, in process_data
  output(ctx)
File 
/base/python_runtime/python_lib/versions/1/google/appengine/ext/mapreduce/operation/db.py,
 line 87, in __call__
  context.mutation_pool.delete(self.entity)
File 
/base/python_runtime/python_lib/versions/1/google/appengine/ext/mapreduce/context.py,
 line 171, in delete
  self.__flush_deletes()
File 
/base/python_runtime/python_lib/versions/1/google/appengine/ext/mapreduce/context.py,
 line 189, in 

[google-appengine] Blobstore return 404 on appengine, but works on SDK

[Karamfil]

Hello,

I'm trying to upload files from a CSV with a python command line tool
I wrote.

The process for every image is:
- get the blobstore url
- send a multipart form-data post request to the url
- proccess additional data and save it to the datastore

Everything works on the SDK but on appengine I receive 404 error when
I send the request to the blobstore url.
Are there any logs for the blobstore or anything that can help me see
whats wrong.

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] When MailService throws IOException - will the mail service retry sending?

[Paul Bridger]

Got this exception on a well-formed MailService.send call:

javax.mail.SendFailedException: MailService IO failed
(java.io.IOException: Internal error)
at
com.google.appengine.api.mail.stdimpl.GMTransport.sendMessage(GMTransport.java:
253)
at javax.mail.Transport.send(Transport.java:95)
at javax.mail.Transport.send(Transport.java:48)

I know that send returns immediately and the mail system will
asynchronously send the mail, but what is the behaviour when send
throws IOException (Internal Error)?

I'm assuming this call has failed, and I need to retry at a later
stage myself.

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: GAE starting unnecessary instances

[Johan Euphrosine]

On Fri, Jul 22, 2011 at 8:57 PM, Galoch galoch...@gmail.com wrote:
 Hi Johan,

 Thanks for the explanation. I have couple of questions on that.

Thanks for showing interest in GAE internals, I'd be happy to answer
those questions directly if I can, or forward them to someone who can
answer them better.

 1. 1 Hours ago while all your Always On instance were busy and you
 had a burst of incoming requests
 While this may be true when my Always On instances were busy running
 some stuff but what about when 2 Always On instances show only 1
 request served which is the Warmup request itself. Does this mean
 Warmup requests are considered as traffic? If that is the case then
 Always On instances seem rather useless since they will never ever get
 called in this scenario.

On the admin console capture you included in your previous mail, I
didn't see Always On instances showing only 1 request served but
rather:
Resident Instance 1:   Requests: 49 Age: 1Hr
Resident Instance 2:   Requests: 6  Age: 1Hr
Resident Instance 3:   Requests: 2  Age: 1Hr

Let me know if I missed something.

 2. As Tom mentioned, what qualifies busy. When threadsafe option was
 implemented in GAE these 3 Always On instances were able to do most of
 the heavy lifting with occasional spinning of dynamic instances.
 Nothing has changed on our side that should alter this behavior. With
 all these changes happening within GAE I am trying to figure out what
 changed and what we can do to contain this burst of traffic within 3
 (or more ) Always On instances with less frequent spinning of Dynamic
 instances.

There are two scheduler knobs that could help you to affect the way
Dynamic instance are spawned.
 Minimum Pending Latency and Max Idle Instances as described here:
http://code.google.com/appengine/docs/adminconsole/performancesettings.html

 3. - 2 Minutes ago all your instances Always On + Dynamic were busy
 again and the scheduler spawned a new Dynamic instance that handle 7
 incoming requests. 
 Again what constitutes busy as I do not see any request being served
 by Always On instances 2 and 3 in last 1 hour. Note that number of
 requests served by Always On 2/3 are unchanged since they were
 created ...
 Here's my reading in this scenario:
 a. It kills Dynamic Instance 1 within 2 minutes of serving a request
 b. When traffic comes in it looks only for Dynamic Instances if they
 are busy and completely ignores Always On instances at this point
 c. It recreates Dynamic Instance 1

 In other words, what rule is applied in this case?

Sorry, those were mostly specification of mine, I didn't know that the
request served by Always On 2/3 were unchanged according to the
information you provided.
I can investigate deeper into the specific behaviour of your
application, if you open a Production Issue with your application id.

 Also I fail to understand rule 4 as both Rob and Luca mentioned. That
 completely undermines having Always On instances under threadsafe
 mode.

 4. I like Rob's suggestion of better load balancing techniques but
 again with a caveat that an instance needs to be able to serve
 multiple threads before reaching a set capacity (80% or so)

 5. Luca's suggestion also makes sense but again with the same
 caveat ... it should be able to process multiple threads before
 queuing

Thanks a lot for your feedback, I will make sure to forward those
suggestions to the engineering team.


 6. I looked at the new sliders in the Admin console and with those the
 situation is even worse. I set the Max Idle Instances to 3 (that's the
 minimum I could choose) and Min Pending Latency to 15 secs ... Guess
 what our CPU usage has gone up to 15 in 12 hrs because of constant
 creation and killing of 3 dynamic instances. Bare minimum traffic and
 few light weight crons.
 But the good side is now I see requests coming in on the 3 Always On
 instances. Is that enough load they are serving ... I don't know yet
 but something to observe.

Maybe you can open a feature request for having a smaller min for 'Max
Idle Instance' when Always On is activated or having Always On
instances count in Max Idle Instance.

 Two things I suggest would be really helpful for us:
 A. The overall key here is to know the thread handling capacity of an
 instance. Better yet if it can be configured similar to Backends but
 dynamic in nature (and of course Backends pricing is outrageous ...
 but that's another topic)

Are you looking for max-concurrent-requests support for Servlet ? If
so I would recommend to open a Feature request.

 B. Able to add more Always On instances but again with a dependency
 explained in point A.

Again, opening a feature request make sense to track this separately.

 On Jul 22, 7:57 am, Johan Euphrosine pro...@google.com wrote:
 HI Galoch,

 Thanks for the followup,

 I think you are experiencing a combinaison fo the two following rules
 I was pointing to in my previous email:
 ( reads as has priority for handling the incoming 

Re: [google-appengine] Re: GAE starting unnecessary instances

[Johan Euphrosine]

Hi,

I will ask to the engineer team and get back to you in thread.

Hope that helps.

On Fri, Jul 22, 2011 at 5:33 PM, Tom Phillips tphill0...@gmail.com wrote:
 When are threaded instances considered busy?

 If it is while they are serving only a one request it would explain
 why Always on instances for threaded Java are now severely under-
 utilized.

 /Tom



 On Jul 22, 10:57 am, Johan Euphrosine pro...@google.com wrote:
 HI Galoch,

 Thanks for the followup,

 I think you are experiencing a combinaison fo the two following rules
 I was pointing to in my previous email:
 ( reads as has priority for handling the incoming request)
 2/ Spawning a new Dynamic instance  Busy Always On instance
 4/ Idle Dynamic instance  Idle Always On instance

 Applied to your example it could means that:
 Resident Instance 1:   Requests: 49     Age: 1Hr
 Resident Instance 2:   Requests: 6      Age: 1Hr
 Resident Instance 3:   Requests: 2      Age: 1Hr
 Dynamic Instance 1:   Requests: 7      Age: 2min
 Dynamic Instance 2:   Requests: 291  Age: 1Hr
 Dynamic Instance 3:   Requests: 322  Age: 1Hr

 - 1 Hours ago while all your Always On instance were busy and you had
 a burst of incoming requests and the scheduler spawned new Dynamic
 instances as per rule 2/ highlighted above.
 - After the burst and back to normal traffic the new Dynamic Instances
 were handing incoming requests in priority as per rule 4/ highlighted
 above.
 - 2 Minutes ago all your instances Always On + Dynamic were busy again
 and the scheduler spawned a new Dynamic instance that handle 7
 incoming requests.

 Hope that make more sense for you and Francois, but as I said earlier
 we are open to suggestion and I will make sure someone working on the
 scheduler team monitor this thread for your input.



 On Fri, Jul 22, 2011 at 9:09 AM, Galoch galoch...@gmail.com wrote:
  @Johan,
  The issue is not about Always On instance being busy. Its actually the
  other way ... the Always On instance is never busy ... at least that
  is what we observed in last 3-4 days. Your explanation may be partly
  true since this behavior keeps on changing.

  For e.g. I have a snapshot of instances from July 19th and here's the
  details (for some reason I can't see a link to attach the snapshot
  images here):
  Resident Instance 1:   Requests: 49     Age: 1Hr
  Resident Instance 2:   Requests: 6      Age: 1Hr
  Resident Instance 3:   Requests: 2      Age: 1Hr
  Dynamic Instance 1:   Requests: 7      Age: 2min
  Dynamic Instance 2:   Requests: 291  Age: 1Hr
  Dynamic Instance 3:   Requests: 322  Age: 1Hr

  This is under no load with only very light weight cron jobs running.
  This gets much much worse during the day under peak load with requests
  for dynamic instances reaching 1000+ in matter of minutes and resident
  instances have only 1 request served.

  As you see above Resident Instance 2 and 3 are hardly hit so I don't
  think they are busy at all. On the other hand, Dynamic Instance 2 and
  3 get most of the hits.

  Dynamic Instance 1 is what is killing us. It keeps getting killed and
  reborn within that 5 minute window!!

  We use Spring framework and it is really very expensive for us when a
  new instance starts up.

  Just to give you a background, we had gone through a real roller
  coaster ride to make this to work on GAE by breaking the loading of
  framework into many different chunks. But still spinning was out of
  control. Then we found java threads to our rescue. We worked through
  the hack to load JDO to avoid UnsupportedOperationException. We
  finally got it to work where most of our requests were served by
  Always On instances with occasional spinning of Dynamic instances. It
  was quite impressive.

  Unfortunately, this was short lived when we hit this new behavior with
  GAE. The very last thing we want GAE to do is create a new instance
  every few minutes as it could easily reach 30 second deadline during
  the day and throw critical error.

  I am not sure when the new billing will come into effect but we really
  need this thing fixed as it literally brings down our app to a
  grinding halt. So I am open to any suggestions you guys think can help
  us.

  Another thought about new scheduler is to have a configurable
  schedule. For e.g. our users are mostly business users who work during
  normal business hours. We want to be able to spin more Always On
  instances during those hours and bring the number down during nights
  and weekends. Dynamic instances won't work for us due to reason
  explained above.

  Thanks,
  galoch

  On Jul 21, 5:56 pm, Johan Euphrosine pro...@google.com wrote:
  After speaking with Engs, I think I can explain what is going on:

  Here are the current scheduling rules: ( reads as has priority for
  handling the incoming request)

  1/ Idle Always On instance  Spawning a new Dynamic instance
  2/ Spawning a new Dynamic instance  Busy Always On instance
  3/ Idle Dynamic instance  Busy Always On instance
  4/ 

[google-appengine] Re: the copied data show me Stats not available

[saintthor]

now, it is ok.

thanks. but the time is so long.

On Jul 25, 1:14 pm, Robert Kluin robert.kl...@gmail.com wrote:
 Have you given it some time to update the stats?







 On Sun, Jul 24, 2011 at 21:54, saintthor saintt...@gmail.com wrote:
  i copied some data to another app.

  in target app, datastore admin, i see the copied entity kinds. but the
  columns show Stats not available.

  how can i know how many entitys were successfully copied?

  --
  You received this message because you are subscribed to the Google Groups 
  Google App Engine group.
  To post to this group, send email to google-appengine@googlegroups.com.
  To unsubscribe from this group, send email to 
  google-appengine+unsubscr...@googlegroups.com.
  For more options, visit this group 
  athttp://groups.google.com/group/google-appengine?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: GAE starting unnecessary instances

[Johan Euphrosine]

On Fri, Jul 22, 2011 at 7:09 PM, Luca luca.de.alf...@gmail.com wrote:
 Johan,
 it seems to me the problem is rule 4/ below.
 Shouldn't the rule be:
 Idle Always On instance  Idle Dynamic instance
 instead?
 In this way, if you have a dynamic instance on, it will not handle traffic
 unless the always-on ones are busy.  So when the traffic decreases, and the
 dynamic instances are no longer needed, they will be correctly shut off.

I confirm the current rules is  Idle Dynamic instance  Always On instance
While I agree your proposition make sense in some cases, it sounds
also good to take advantage of the Dynamic instances that have just be
spawned especially if the startup cost is high.

Like Gregd highlighted in his pricing faq, Always On will became a
settings to control the number of idle instances you would like to
have running.

Q: How will Always On work under the new model?
A: When App Engine leaves preview all Paid Apps and Apps in Premier
Accounts will be able to set the number of idle instances they would
like to have running.  Always On was designed to allow an app to
always have idle instances running to save on instance start-up
latency.  For many Apps a single idle instance should be enough
(especially when using concurrent requests).  This means that for many
customers, setting an App to be paid will mean a $9/month minimum
spend, you can then use the 24 free IH/day to keep an instance running
all the time by setting Min Idle Instances to be 1.

 I also think an auxiliary problem is with rule 2/
 If you spawn a new instance just because the others are all busy, you may
 tend to have too many instances.
 The fact of spawning new instances should be configurable, depending on how
 many requests are already queued for the instances you have already active
 -- 0, 1, more, etc.
 More in general, if you allowed users to specify how much it costs to them
 to delay serving a request, it would not be difficult to synthesize for each
 app an optimal decision policy to decide whether to switch another instance
 on or off.  This can be done using tools from dynamic optimization / control
 theory.  I would be glad to help if the people there need guidance on this
 (I used to be in the research group there till a month ago).

It sounds to me that the scheduler knobs are there precisely to address this:
http://code.google.com/appengine/docs/adminconsole/performancesettings.html

Let me know if I overlooked it.

 Luca
 On Thursday, July 21, 2011 5:56:42 PM UTC-7, Johan Euphrosine (Google)
 wrote:

 After speaking with Engs, I think I can explain what is going on:

 Here are the current scheduling rules: ( reads as has priority for
 handling the incoming request)

 1/ Idle Always On instance  Spawning a new Dynamic instance
 2/ Spawning a new Dynamic instance  Busy Always On instance
 3/ Idle Dynamic instance  Busy Always On instance
 4/ Idle Dynamic instance  Idle Always On instance

 --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To view this discussion on the web visit
 https://groups.google.com/d/msg/google-appengine/-/X8gH6jjIK0MJ.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.




-- 
Johan Euphrosine (proppy)
Developer Programs Engineer
Google Developer Relations

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: GAE starting unnecessary instances

[Johan Euphrosine]

I think Ggegd covered this in his pricing FAQ:

Q: How will Always On work under the new model?
A: When App Engine leaves preview all Paid Apps and Apps in Premier
Accounts will be able to set the number of idle instances they would
like to have running.  Always On was designed to allow an app to
always have idle instances running to save on instance start-up
latency.  For many Apps a single idle instance should be enough
(especially when using concurrent requests).  This means that for many
customers, setting an App to be paid will mean a $9/month minimum
spend, you can then use the 24 free IH/day to keep an instance running
all the time by setting Min Idle Instances to be 1.

Let me know if you need more information.

On Fri, Jul 22, 2011 at 9:46 PM, Tom Phillips tphill0...@gmail.com wrote:
 The current behavior makes me suspect we are being prepared for Always-
 on being replaced completely by the new scheduler knobs. Being able to
 turn up the number of idle instances does make always-on somewhat
 redundant, as long as the idle instances stick around for a while.

 Also, if always-on instances WERE being properly utilized right now,
 it would require artificial load to ascertain the effects of the new
 scheduler on dynamic instances (assuming the three AO instances were
 sufficient for your app previously). Many devs wouldn't learn about
 the new scheduler attributes until after their app (suddenly) becomes
 popular - not the best time for surprises. And Google wouldn't get as
 much feedback on the features/behavior of the new scheduler.

 Is Always-on going to be kept under the new model?

 /Tom


 On Jul 22, 10:57 am, Johan Euphrosine pro...@google.com wrote:
 HI Galoch,

 Thanks for the followup,

 I think you are experiencing a combinaison fo the two following rules
 I was pointing to in my previous email:
 ( reads as has priority for handling the incoming request)
 2/ Spawning a new Dynamic instance  Busy Always On instance
 4/ Idle Dynamic instance  Idle Always On instance

 Applied to your example it could means that:
 Resident Instance 1:   Requests: 49     Age: 1Hr
 Resident Instance 2:   Requests: 6      Age: 1Hr
 Resident Instance 3:   Requests: 2      Age: 1Hr
 Dynamic Instance 1:   Requests: 7      Age: 2min
 Dynamic Instance 2:   Requests: 291  Age: 1Hr
 Dynamic Instance 3:   Requests: 322  Age: 1Hr

 - 1 Hours ago while all your Always On instance were busy and you had
 a burst of incoming requests and the scheduler spawned new Dynamic
 instances as per rule 2/ highlighted above.
 - After the burst and back to normal traffic the new Dynamic Instances
 were handing incoming requests in priority as per rule 4/ highlighted
 above.
 - 2 Minutes ago all your instances Always On + Dynamic were busy again
 and the scheduler spawned a new Dynamic instance that handle 7
 incoming requests.

 Hope that make more sense for you and Francois, but as I said earlier
 we are open to suggestion and I will make sure someone working on the
 scheduler team monitor this thread for your input.



 On Fri, Jul 22, 2011 at 9:09 AM, Galoch galoch...@gmail.com wrote:
  @Johan,
  The issue is not about Always On instance being busy. Its actually the
  other way ... the Always On instance is never busy ... at least that
  is what we observed in last 3-4 days. Your explanation may be partly
  true since this behavior keeps on changing.

  For e.g. I have a snapshot of instances from July 19th and here's the
  details (for some reason I can't see a link to attach the snapshot
  images here):
  Resident Instance 1:   Requests: 49     Age: 1Hr
  Resident Instance 2:   Requests: 6      Age: 1Hr
  Resident Instance 3:   Requests: 2      Age: 1Hr
  Dynamic Instance 1:   Requests: 7      Age: 2min
  Dynamic Instance 2:   Requests: 291  Age: 1Hr
  Dynamic Instance 3:   Requests: 322  Age: 1Hr

  This is under no load with only very light weight cron jobs running.
  This gets much much worse during the day under peak load with requests
  for dynamic instances reaching 1000+ in matter of minutes and resident
  instances have only 1 request served.

  As you see above Resident Instance 2 and 3 are hardly hit so I don't
  think they are busy at all. On the other hand, Dynamic Instance 2 and
  3 get most of the hits.

  Dynamic Instance 1 is what is killing us. It keeps getting killed and
  reborn within that 5 minute window!!

  We use Spring framework and it is really very expensive for us when a
  new instance starts up.

  Just to give you a background, we had gone through a real roller
  coaster ride to make this to work on GAE by breaking the loading of
  framework into many different chunks. But still spinning was out of
  control. Then we found java threads to our rescue. We worked through
  the hack to load JDO to avoid UnsupportedOperationException. We
  finally got it to work where most of our requests were served by
  Always On instances with occasional spinning of Dynamic instances. It
  was quite impressive.

  

Re: [google-appengine] Re: GAE starting unnecessary instances

[Johan Euphrosine]

What is your setting for 'Max Idle Instances' ?

Feel free to open a Production issue with you appid if you want me to
track this specifically.

On Sat, Jul 23, 2011 at 11:01 AM, Francois Masurel f.masu...@gmail.com wrote:
 Geez, now even some Dynamic instances are not used too and stay idle for
 hours (check attachment).
 With the new pricing model this will get really really expensive.
 I hope there is definitely something going wrong.
 Francois

 --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To view this discussion on the web visit
 https://groups.google.com/d/msg/google-appengine/-/Qczx6foLplUJ.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.




-- 
Johan Euphrosine (proppy)
Developer Programs Engineer
Google Developer Relations

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: GAE starting unnecessary instances

[Francois Masurel]

Hi Johan,

Thanx for your answer.

Setting MaxIdleInstances to 3 did force the resident instances to be used 
instead of the dynamic ones.

As we will soon pay for active instances, I would haved like to have the 3 
resident ones being used instead of extra dynamic ones being started.

Thanx again for your help.

Francois

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/2E_xthK5Xo4J.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: Error 403: --- begin server output ---You do not have permission to modify this app

[Jonathan]

Thank you Robert

However:
1.  I'm using Ubuntu platform and could not find any appcfg_cookies
file. Any idea where it should be?
2.  The
 Error 403: --- begin server output ---
 You do not have permission to modify this app
 is showing up before I'm requested to provide the username and
password

Any idea how to proceed?


On Jul 25, 8:36 am, Robert Kluin robert.kl...@gmail.com wrote:
 I suggest you try deleting your ~/.appcfg_cookies file and being extra
 careful when entering your account info.  Maybe you're making a small
 typo / using the wrong account to push?

 Robert







 On Sun, Jul 24, 2011 at 14:47, Jonathan jonathan.na...@gmail.com wrote:
  Is there any chance that after providing a wrong password I was
  permanently locked out?

  On Jul 24, 10:30 pm, Jonathan jonathan.na...@gmail.com wrote:
  After two years of successful uploads of all my five applications I've
  changed to Django 1.2

  Now it says: You do not have permission to modify this app

  Details:

  Application: reimbu-germany; version: 570
  Host: appengine.google.com

  Starting update of app: reimbu-germany, version: 570
  Scanning files on local disk.
  Error 403: --- begin server output ---
  You do not have permission to modify this app (app_id=u'reimbu-
  germany').
  --- end server output ---

  It's the same with all my apps.
  I've checked that the app name is the same on appspot.com and in my
  app.yaml

  Any idea how to resolve?

  --
  You received this message because you are subscribed to the Google Groups 
  Google App Engine group.
  To post to this group, send email to google-appengine@googlegroups.com.
  To unsubscribe from this group, send email to 
  google-appengine+unsubscr...@googlegroups.com.
  For more options, visit this group 
  athttp://groups.google.com/group/google-appengine?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: GAE starting unnecessary instances

[Francois Masurel]

Something strange :

GAE keeps starting new dynamic instances though I have set the Min Pending 
Latency to 15s via the Application Settings page.

My requests are served in a few ms on average and I have multithreading
 enabled.

These instances are immediately destroyed as I have set the Max Idle 
Instances to 3 (corresponding to my always on instances).

But still lots of warmup requests in my logs showing these dynamic instances 
start every few minutes.

Will I have to pay for all these short lived dynamic instances ?

My app ID is : vncts1

Thanx for your help.

Francois

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/TVr3Ko1-baIJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Digest for google-appengine@googlegroups.com - 20 Messages in 7 Topics

[Kemuel Enders]

Hi all, I'm a novice. Can anyone show me, If there is a way to run Multiple
apps on gae with linux?


On Sat, Jul 23, 2011 at 7:11 PM,
google-appengine+nore...@googlegroups.comwrote:

   Today's Topic Summary

 Group: http://groups.google.com/group/google-appengine/topics

- SDK Upgrade 1.5.2: --datastore_path 
 ignored?#13158691f992ddfb_group_thread_0[9 Updates]
- 1.5.2 SDK Prerelease #13158691f992ddfb_group_thread_1 [1 Update]
- Transactions and ancestor in a different 
 namespace#13158691f992ddfb_group_thread_2[1 Update]
- For deployment only does the sdk version 
 matter?#13158691f992ddfb_group_thread_3[1 Update]
- GAE starting unnecessary instances #13158691f992ddfb_group_thread_4[4 
 Updates]
- path/wildcards on GAE Boto 
 get_bucket()?#13158691f992ddfb_group_thread_5[1 Update]
- App Engine Weekly Community Update 
 #1#13158691f992ddfb_group_thread_6[3 Updates]

   Topic: SDK Upgrade 1.5.2: --datastore_path 
 ignored?http://groups.google.com/group/google-appengine/t/90918f593e1f860c

c h how...@umich.edu Jul 22 12:17PM -0700 ^#13158691f992ddfb_digest_top

hi all,

i *think* that it is honoring your datastore location (though the log
message is incorrect), but the change to rename your application to
dev~your appname in development has just rendered all of our test
data
useless.

after re-importing my test data it does look like it is stored where i
ask
it to be, but under the new application name.

if you are lucky enough to be using sqlite you can connect to the db
and
rename some tables to get it to work:

sqlite3 local_appname_dev_sqlite.datastore
sqlite .tables
Apps
IdSeq
Namespaces
appname!!Entities
appname!!EntitiesByProperty
appname!namespace!Entities
appname!namespace!EntitiesByProperty
sqlite alter table `appname!!Entities` rename to
`dev~appname!!Entities`;
sqlite alter table `appname!!EntitiesByProperty` rename to
`dev~appname!!EntitiesByProperty`;
sqlite alter table `appname!namespace!Entities` rename to
`dev~appname!namespace!Entities`;
sqlite alter table `appname!namespace!EntitiesByProperty` rename to
`dev~appname!namespace!EntitiesByProperty`;


where you substitute 'appname' for your application's name, and
'namespace'
for your datanamespace.

cfh




Matthew Blain matthew.bl...@google.com Jul 22 12:39PM -0700 
 ^#13158691f992ddfb_digest_top

That's a clever way to update the appid. I do not know if it works for
all cases (e.g. it may not work for all reference properties (stored
keys)) but is a neat trick.

Another way to deal with it is to use the --default_partition= flag
rather than using an older version of the sdk.

--Matthew





Chris Copeland ch...@cope360.com Jul 22 04:57PM -0500 
 ^#13158691f992ddfb_digest_top

Thanks, Matthew.

I was able to update to 1.5.2 and use my existing datastore by adding
that
flag.

It would have been useful if the release notes had mentioned that this
would
be necessary.

-Chrsi





c h how...@umich.edu Jul 22 06:38PM -0700 ^#13158691f992ddfb_digest_top

yup the sqlite updates don't quite seem to work.

i second that the release notes should be updated, the flag does seem
to
work for me

cfh




Joseph Letness joe.letn...@gmail.com Jul 22 06:49PM -0700 
 ^#13158691f992ddfb_digest_top

I tried the --default_partition= flag but now validation.py is
throwing an exception:

Traceback (most recent call last):
File /Applications/GoogleAppEngineLauncher.app/Contents/Resources/
GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/
google/appengine/tools/dev_appserver.py, line 4099, in _HandleRequest
default_partition)
File /Applications/GoogleAppEngineLauncher.app/Contents/Resources/
GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/
google/appengine/api/validation.py, line 360, in __setattr__
value = self.GetValidator(key)(value, key)
File /Applications/GoogleAppEngineLauncher.app/Contents/Resources/
GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/
google/appengine/api/validation.py, line 598, in __call__
return self.Validate(value, key)
File /Applications/GoogleAppEngineLauncher.app/Contents/Resources/
GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/
google/appengine/api/validation.py, line 923, in Validate
'\'%s\'' % (value, key, self.re.pattern))
ValidationError: Value '~tagassetspro' for application does not
match expression '^(?:[a-z\d\-]{1,100}\~)?(?:(?!\-)[a-z\d\-\.]
{1,100}:)?(?!-)[a-z\d\-]{1,100}$'

This has got to be a bug. It's not like using stored test data with
the dev server is some kind of oddball edge case ;-)

--Joe







Cat katz...@gmail.com Jul 23 12:25AM -0700 
 ^#13158691f992ddfb_digest_top

One thing 

[google-appengine] Disabling the Google Frontend's Reverse Caching Proxy Behavior

[Ludvig Ericson]

Hello,

my case is that I have data that is fine if it's stale for some
people, but I would of course not want everybody to have equally stale
data.

This becomes a problem however, for of course I use the appointed HTTP
caching headers per RFC 2616, section 13. To cut it briefly, I always
set the expiry to tomorrow 00:00 because this is an acceptable trade-
off for our purposes.

When we rolled this out, everybody would be getting stale data
however! It seemed the Google Frontend servers simply did not
revalidate, and why should they? After all, the headers are saying
it's OK until tomorrow.

So basically I *do want* this behavior, but I *do not want* it for the
scope that a reverse proxy provides - Google's got me by the balls
here, pretty much. How do I tell the GFE not to be a reverse caching
proxy?

Sincerely
-Ludvig Ericson, http://sendapatch.se/

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Disabling the Google Frontend's Reverse Caching Proxy Behavior

[Barry Hunter]

Use

Cache-Control: private

http://jigsaw.w3.org/HTTP/cc-private.html

The Google Proxy (and any other proxy along the line for that matter!)
shouldnt cache it.


(But it can still be cached in the users 'private' cache. Which I
persume is why you want to set an expiry at all)


On Mon, Jul 25, 2011 at 4:52 PM, Ludvig Ericson
ludvig.eric...@gmail.com wrote:
 Hello,

 my case is that I have data that is fine if it's stale for some
 people, but I would of course not want everybody to have equally stale
 data.

 This becomes a problem however, for of course I use the appointed HTTP
 caching headers per RFC 2616, section 13. To cut it briefly, I always
 set the expiry to tomorrow 00:00 because this is an acceptable trade-
 off for our purposes.

 When we rolled this out, everybody would be getting stale data
 however! It seemed the Google Frontend servers simply did not
 revalidate, and why should they? After all, the headers are saying
 it's OK until tomorrow.

 So basically I *do want* this behavior, but I *do not want* it for the
 scope that a reverse proxy provides - Google's got me by the balls
 here, pretty much. How do I tell the GFE not to be a reverse caching
 proxy?

 Sincerely
 -Ludvig Ericson, http://sendapatch.se/

 --
 You received this message because you are subscribed to the Google Groups 
 Google App Engine group.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to 
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at 
 http://groups.google.com/group/google-appengine?hl=en.



-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: Digest for google-appengine@googlegroups.com - 20 Messages in 7 Topics

[Geoffrey Spear]

On Jul 25, 11:22 am, Kemuel Enders hit...@gmail.com wrote:
 Hi all, I'm a novice. Can anyone show me, If there is a way to run Multiple
 apps on gae with linux?

Sure, just specify a different port for each to run on. Which runtime
are you using?

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Disabling the Google Frontend's Reverse Caching Proxy Behavior

[Johan Euphrosine]

Thanks for answering Barry.

On a side note, I confirm that App Engine caching should behave like
any regular HTTP/1.1 proxy if you mark your requests with the
appropriate headers.

On Mon, Jul 25, 2011 at 5:57 PM, Barry Hunter barrybhun...@gmail.com wrote:
 Use

 Cache-Control: private

 http://jigsaw.w3.org/HTTP/cc-private.html

 The Google Proxy (and any other proxy along the line for that matter!)
 shouldnt cache it.


 (But it can still be cached in the users 'private' cache. Which I
 persume is why you want to set an expiry at all)


 On Mon, Jul 25, 2011 at 4:52 PM, Ludvig Ericson
 ludvig.eric...@gmail.com wrote:
 Hello,

 my case is that I have data that is fine if it's stale for some
 people, but I would of course not want everybody to have equally stale
 data.

 This becomes a problem however, for of course I use the appointed HTTP
 caching headers per RFC 2616, section 13. To cut it briefly, I always
 set the expiry to tomorrow 00:00 because this is an acceptable trade-
 off for our purposes.

 When we rolled this out, everybody would be getting stale data
 however! It seemed the Google Frontend servers simply did not
 revalidate, and why should they? After all, the headers are saying
 it's OK until tomorrow.

 So basically I *do want* this behavior, but I *do not want* it for the
 scope that a reverse proxy provides - Google's got me by the balls
 here, pretty much. How do I tell the GFE not to be a reverse caching
 proxy?

 Sincerely
 -Ludvig Ericson, http://sendapatch.se/

 --
 You received this message because you are subscribed to the Google Groups 
 Google App Engine group.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to 
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at 
 http://groups.google.com/group/google-appengine?hl=en.



 --
 You received this message because you are subscribed to the Google Groups 
 Google App Engine group.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to 
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at 
 http://groups.google.com/group/google-appengine?hl=en.





-- 
Johan Euphrosine (proppy)
Developer Programs Engineer
Google Developer Relations

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: GAE starting unnecessary instances

[Johan Euphrosine]

Hi Francois,

I think the help text of the 'Idle Instances' settings is pretty
self-explanatory:

You will not be charged for instances over the specified maximum.


I can investigate on why these instances are created even thought you
setup a high min pending latency, and a a Max idle instances
corresponding to the number of Always On instance.

Feel free to open a Production issue, with your application id if this
is affection your operation.

Thanks in advance.

On Mon, Jul 25, 2011 at 4:58 PM, Francois Masurel f.masu...@gmail.com wrote:
 Something strange :
 GAE keeps starting new dynamic instances though I have set the Min Pending
 Latency to 15s via the Application Settings page.
 My requests are served in a few ms on average and I have
 multithreading enabled.
 These instances are immediately destroyed as I have set the Max Idle
 Instances to 3 (corresponding to my always on instances).

 But still lots of warmup requests in my logs showing these dynamic instances
 start every few minutes.
 Will I have to pay for all these short lived dynamic instances ?
 My app ID is : vncts1
 Thanx for your help.
 Francois

 --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To view this discussion on the web visit
 https://groups.google.com/d/msg/google-appengine/-/TVr3Ko1-baIJ.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.




-- 
Johan Euphrosine (proppy)
Developer Programs Engineer
Google Developer Relations

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Disabling the Google Frontend's Reverse Caching Proxy Behavior

[Barry Hunter]

 Setting the cache control to private will also mean intranets, ISPs
 and suchlike won't be caching anymore, which is a situation I would rather
 avoid.

So you saying you dont mind all customers of that ISP getting the same
stale data, just dont want it for all you users.
?


You could just add a changing cache-busting fragment to the URL.
Perhaps hashed off the remote IP address. Then you will have say 10
differnt versions stored in Googles cache, and different people
(depending on their IP) will get a different version.

By the laws of probablity, each version will (or should) get cached at
different times on the frontend, so each will update on a different
cycle.

(but you will also have to be more specific with your expirely time -
not just using 00:00. eg using 24 hours from the request time. Or use
say 25 hours, so over time the exact refresh time of each 'shard'
shifts around)

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Disabling the Google Frontend's Reverse Caching Proxy Behavior

[Ludvig Ericson]

Hi Barry, and thank you for your quick reply.

Yes, I found Cache-Control: private too now that I looked a little more 
closely, but I guess I want to avoid a workaround: the HTTP 1.1 RFC says 
private should be used for non-shared data, while this really is shared 
data! Setting the cache control to private will also mean intranets, ISPs 
and suchlike won't be caching anymore, which is a situation I would rather 
avoid.

Is there really no way to disable the reverse caching behavior?

-L

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/ZITOUGJ7OhAJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: 回复：Re: [google-appengine] Total Stored Data is larger than the actual use.

[Stephen Johnson]

The thing to note is that doing this won't affect any of the current data.
You'll need to either delete the current data and re-upload with those
properties set to unindexed or you need to create your model with the
properties set to unindexed and then iterate over the current data and
retrieve each entity and then re-put the entity. With the composite indexes,
you can delete/vacuum them and re-create them sort of like you can create
and drop indexes in a normal relational database, but the individual
property indexes can't be manipulated as a whole unit. I hope that makes
sense. Also, if you're using Java/JDO you use the following annotation:

   @Extension(vendorName = datanucleus, key = gae.unindexed, value=
true)


Other frameworks (Objectify, etc.) use different mechanisms.


Some other useful tips for keeping datastore usage to a minimum:

1.) keep your property names short (these names are duplicated for each
entity as metadata and also in the indexes, etc.) - I use 3 character names

2.) keep your entity names short (these are duplicated inside the keys,
etc.) - I use 2 character names

3.) keep your namespace names short (again duplicated inside of keys)

4.) keep your app-id short (this one is difficult because we all want
distinctive names)


Why the big deal about keys? Because all these keys get copied for each
index entry and for single property indexes you don't get just one index,
you get two by default (an ascending and a descending index). So, if you've
got 20 indexed properties on an entity, you've got 41 indexes (1 for the
primary key and 40 for the properties). All that key overhead adds up quick
as you've discovered.


As an example, I have 391 MB of data and my datastore usage is 400 MB right
now in the stats and dashboard. My case isn't probably typical because I've
got more blob usage than most, but I offer it as an example.

Hope this helps,
Stephen
CortexConnect (cortexconnect.appspot.com) http://cortexconnect.appspot.com

On Sun, Jul 24, 2011 at 10:38 PM, Robert Kluin robert.kl...@gmail.comwrote:

 In Python it looks like this:

 class Example(db.Model):
  prop = db.StringProperty(indexed=False)

 The Java syntax is more complex, and I don't recall it from memory. ;)


 Robert






 On Mon, Jul 25, 2011 at 00:36, lezizi w...@lezizi.org wrote:
  I've stared it.
  By the way, can you show me how to disable indexes?
 
  --
  You received this message because you are subscribed to the Google Groups
  Google App Engine group.
  To view this discussion on the web visit
  https://groups.google.com/d/msg/google-appengine/-/DZfwwUK0QrgJ.
  To post to this group, send email to google-appengine@googlegroups.com.
  To unsubscribe from this group, send email to
  google-appengine+unsubscr...@googlegroups.com.
  For more options, visit this group at
  http://groups.google.com/group/google-appengine?hl=en.
 

 --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.



-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: get_original_metadata working?

[Ed Estes]

I too am interested in *get_original_metadata *calls to pull metadata from 
photos.  I haven't seen any updates to the docs to reflect what came out 
with GAE 1.5.2 SDK.

Any confirmation/assistance with this would be greatly appreciated.

Thanks!

--Ed

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/xgtDUMIzDTQJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Disabling the Google Frontend's Reverse Caching Proxy Behavior

[Ludvig Ericson]

On Monday, July 25, 2011 6:24:19 PM UTC+2, barryhunter wrote:

  Setting the cache control to private will also mean intranets, ISPs
  and suchlike won't be caching anymore, which is a situation I would 
 rather
  avoid.

 So you saying you dont mind all customers of that ISP getting the same
 stale data, just dont want it for all you users?

Exactly! 

 You could just add a changing cache-busting fragment to the URL.
 Perhaps hashed off the remote IP address. Then you will have say 10
 differnt versions stored in Googles cache, and different people
 (depending on their IP) will get a different version.

 By the laws of probablity, each version will (or should) get cached at
 different times on the frontend, so each will update on a different
 cycle.

Ingenious, I might go for this to be honest. That or just marking it private 
and letting some extraneous requests through, I mean how common are 
organization-wide Web proxies these days anyway?
-L

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/aiXEB0rcqW4J.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Disabling the Google Frontend's Reverse Caching Proxy Behavior

[Stephen]

On Mon, Jul 25, 2011 at 5:13 PM, Ludvig Ericson
ludvig.eric...@gmail.com wrote:
 Hi Barry, and thank you for your quick reply.
 Yes, I found Cache-Control: private too now that I looked a little more
 closely, but I guess I want to avoid a workaround: the HTTP 1.1 RFC says
 private should be used for non-shared data, while this really is shared
 data! Setting the cache control to private will also mean intranets, ISPs
 and suchlike won't be caching anymore, which is a situation I would rather
 avoid.
 Is there really no way to disable the reverse caching behavior?


Use s-maxage to set a shared cache time of 1 hour, overriding the
private cache time of 24 hours.

  http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.9.3

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: GAE starting unnecessary instances

[Galoch]

Hi Johan,

Thanks for following up on this discussion.

I have new findings to share with you since GAE now works a bit
differently with all these recent changes. Before I do that I will
share the changes I made to our app which has helped us mitigate some
performance issues.

1. We moved all our cron jobs to the Backends with 1 B1 RI. Amazingly
it is able to handle all the load single handed compared to 6
instances (3 RI + 3 DI) in the regular app. We have our eye brows
raised! But we are playing it with caution by setting max-concurrent-
requests to 1. Will gradually raise the limits until it starts
complaining.

2. We set Max Idle Instances to 6 and Min Pending Latency to 15s. I
recommend keeping Max Idle Instances to at least 4. Because if you
keep it at 3 GAE kills and creates DI very aggressively ... in some
cases 10-20 seconds.

3. In our regular app we are running dummy cron jobs to keep 3 DI up
at all times. Again, amazingly with many concurrent users they are
able to handle ALL the load. On the contrary RI are almost untouched.
Here's the latest statistics:

QPS*Latency*RequestsErrors  Age Memory
Availability
0.000   0.0 ms  25  0  12:43:22 82.3 MBytes 
Resident
Icon Resident
0.000   0.0 ms  10  0  12:44:45 80.6 MBytes 
Resident
Icon Resident
0.000   0.0 ms  5   0  12:44:46 81.9 MBytes 
Resident
Icon Resident
0.350   19.1 ms 11762   0   9:45:11 118.5 MBytes
Dynamic Icon
Dynamic
0.317   17.5 ms 38900   3:11:04 97.7 MBytes 
Dynamic Icon
Dynamic
0.333   67.5 ms 15604   0   12:43:41104.8 MBytes
Dynamic Icon
Dynamic

But this is just a sample set of users. I don't know yet if we can
scale on this principle but it seems to work on light load. But this
is definitely short term fix since this is going to be expensive under
new billing.


After spending fairly good amount of time collecting these statistics
I can boil down to a few things that I want to share, that can help us
tune our app to GAE.
A. Know the current usage of an instance (RI or DI) in terms of
current CPU utilization in percentage. This will help us correlate CPU
utilization shown in each request in the logs and help us tune our
queries. Right now there is no way to determine what effect the tuning
had on an individual instance.

B. Know the number of current threads running within an instance
(maybe in a span of 15 seconds or so). This will help us identify
ratio of thread handling capacity vs CPU consumed (point A). Higher
the number means better throughput. Lower the number means we need to
closely examine our requests.

C. Able to override the 30 second deadline for the warm up requests so
that we have enough time to load our frameworks.

D. Able to specify how long an instance can live before reclaimed by
GAE. Something similar to session timeouts. This will help avoid
unnecessary warm up requests and will improve overall performance.

E. Ability to specify rules within GAE console that will adjust the
Max Idle Instances during different times of a day and week.

F. Able to specify the size of  DI (something similar to Backends B1,
B2 ... so on) instead of standard DI. Should be able to configure
along with each rule mentioned in point E.


I understand that C, D, E, F may involve some additional costs to us
but these short / long term enhancements will hugely help us run our
business applications on GAE. The idea is to give us more time and
capacity with the DI while keeping a control on the cost.

Let me know if I missed something that is already there or I am going
off tangent here.




On Jul 25, 9:18 am, Johan Euphrosine pro...@google.com wrote:
 Hi Francois,

 I think the help text of the 'Idle Instances' settings is pretty
 self-explanatory:
 
 You will not be charged for instances over the specified maximum.
 

 I can investigate on why these instances are created even thought you
 setup a high min pending latency, and a a Max idle instances
 corresponding to the number of Always On instance.

 Feel free to open a Production issue, with your application id if this
 is affection your operation.

 Thanks in advance.









 On Mon, Jul 25, 2011 at 4:58 PM, Francois Masurel f.masu...@gmail.com wrote:
  Something strange :
  GAE keeps starting new dynamic instances though I have set the Min Pending
  Latency to 15s via the Application Settings page.
  My requests are served in a few ms on average and I have
  multithreading enabled.
  These instances are immediately destroyed as I have set the Max Idle
  Instances to 3 (corresponding to my always on instances).

  But still lots of warmup requests in my logs showing these dynamic instances
  start every few minutes.
  Will I have to pay for all these short lived dynamic instances ?
  My app ID is : vncts1
  Thanx for your help.
  Francois

 

Re: [google-appengine] get_original_metadata working?

[Martin Ceperley]

Hi Ed,

I looked at the source and it is just an instance method of the images.Image 
class, called with no parameters, i.e. image.get_original_metadata()

I'm stilling getting None from the method, I have tried it both by 
instantiating an Image from a blob_key and directly with image data, in both 
cases get_original_metadata returned None so it appears not to be working at 
all.

-Martin

On Jul 25, 2011, at 1:02 PM, Ed Estes wrote:

 I too am interested in get_original_metadata calls to pull metadata from 
 photos.  I haven't seen any updates to the docs to reflect what came out with 
 GAE 1.5.2 SDK.
 
 Any confirmation/assistance with this would be greatly appreciated.
 
 Thanks!
 
 --Ed
 
 -- 
 You received this message because you are subscribed to the Google Groups 
 Google App Engine group.
 To view this discussion on the web visit 
 https://groups.google.com/d/msg/google-appengine/-/xgtDUMIzDTQJ.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to 
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at 
 http://groups.google.com/group/google-appengine?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: Need help understanding Appengine billing

[Kwame]

Can you elaborate a little more on your costs using Appengine? It will
be very useful information for us. What we need to know is what we
should expect as our site scales up in users. Approximately how many
requests are you getting and how much is it costing you? Thanks.

On Jul 24, 3:48 pm, Kaan Soral kaanso...@gmail.com wrote:
 Generally, with the new billing scheme if you don't have a good
 monetization model you shouldn't use AppEngine

 I am always building products that are ad-monetized and for example
 today my ECPM dropped from 0.9 to 0.04

 The income is still greater than AppEngine costs but I am sure I will
 be on the edge or over the edge with the new scheme

 On Jul 24, 9:35 am, Robert Kluin robert.kl...@gmail.com wrote:







  Hi,
    I've seen apps that are doing maybe a couple million requests per day cost
  $50 a day, I've seen apps doing tens of millions of requests a day cost $50
  a day, and I've seen apps do over one hundred million requests in a day cost
  $500 a day. It completely depends on how intensive your app is to run.

  Robert

  On Jul 22, 2011 9:50 AM, Kwame iweg...@gmail.com wrote:

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: 1.5.2 SDK Prerelease

[Nevin Freeman]

For developers facing this issue locally, it may be useful to know that I 
only was able to access the old local datastore by adding the 
flag --default_partition= (without any quotes).

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/RAuu4omiwwUJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: Error 403: --- begin server output ---You do not have permission to modify this app

[Ernesto Oltra]

You should read Robert response more closely:

 ~/.appcfg_cookies

Means if you're using Ubuntu, you should have a hidden file called 
.appcfg_cookies in your home directory (/home/[username]). Notice the 
point before the filename.

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/ZIu0Qzh8dcAJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] get_original_metadata working?

[Ed Estes]

Thanks for confirming this.  I was calling it the same way (similar to the 
*histogram 
*method) but since I wasn't seeing any data (it was also returning *None*), 
I thought I was not calling *get_original_metadata* properly.

I was using some of the sample photos found over on 
exif.orghttp://exif.org/samples.html
.

Hopefully someone will clarify things shortly.

--Ed

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/GoKS967umKIJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] GAE Testbed Error: No api proxy found for service file

[Dan Dubois]

Thanks both of you. It is Python I am using so I have starred the issue.

Best wishes,
Dan

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/AhTsggs7aaEJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: HR database, ancestor queries and data consistency: docs not 100% clear

[Nick Johnson (Google)]

On Mon, Jul 25, 2011 at 4:30 PM, Pol i...@pol-online.net wrote:

 On Jul 25, 7:30 am, Nick Johnson (Google) nick.john...@google.com
 wrote:

  The first part specifically refers to ancestor queries. Ancestor queries
  always operate on the latest copy of the data, as they're restricted to
 an
  entity group; regular queries don't have that guarantee.

 I understand that, but my problem is with the indexes: even in the
 scope of HR + transactions, the doc is confusing regarding wether or
 not there can be a race-condition on indexes update.

 For instance:
 - transaction A modifies entity X and commits
 - then transaction B happens almost concurrently:
  - at this point, using db.get() X will return the latest X no matter
 what
  - however, using db.GqlQuery() with a predicate that matches X *may
 not* return X because the indexes may not be done updating yet (the A -
  B thing in the doc)

 So is this race-condition possible or not? No matter what, you might
 wanna rewrite a couple sentences in this article :)


Again, this depends on if you're using an ancestor query. Queries that
include ancestor filters are strongly consistent, queries that don't aren't.



   My understanding of this article is therefore the following (assuming
   an HR datastore and that we are inside a transaction):
 
   1) Using db.get() / db.put() on an entity always executes on its
   latest version: no race conditions possible, period.
   - Observations in my app appears to confirm that behavior.
 
  Correct, with the usual caveat about transactionality (if you're not
 doing
  this inside a transaction, you have a potential race condition).

 Yes, I was assuming HR datastore + transaction (see above).

   2) Using db.GqlQuery() / db.put() on entities in the same entity group
   might still be subject to race-conditions: for instance, a previously
   *committed* transaction modified the entities that would be returned
   by the query in the later transactions, but the datastore indexes are
   not up-to-date yet (or something like that), so the query doesn't see
   the updated entities - even if calling db.get() directly on them would
   return the versions that match the query.
   - I've reviewed my code carefully and I think I'm seeing such race
   conditions in my app.
 
  Correct - but again, any update outside a transaction always has possible
  race conditions.

 Same here: I was also assuming HR datastore + transaction + ancestor
 queries (by definition since inside a transaction).

 So is my code wrong or am I observing some very rare but expected race
 conditions?

 Thanks

 - Pol

 --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.




-- 
Nick Johnson, Developer Programs Engineer, App Engine

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Datastore Admin - Delete didn't work, burned $7.85

[Robert Kluin]

You're getting datastore timeouts.  If you're on master-slave, then
you might try running it again.  Perhaps you were just unlucky and ran
durring a latency spike.

Also, does that entity have a lot of indexes or something?


Robert






On Sun, Jul 24, 2011 at 16:09, GregR rata...@gmail.com wrote:
 I have a Entity with 2,356,292 in it.  I tried using the datastore Admin
 Delete Entities to purge it.  It ran for a while, got a bunch of errors,
 burned $7.85 (mostly in CPU), and didn't actually delete anything.

 I've included some of the errors below.

 Any idea what's up? I can go back to the old way of deleting, but it would
 take a long time (and CPU) with this # of entities.

 Thanks,
 -Greg

 ·  0.1.0.2 - - [24/Jul/2011:13:20:37 -0700] POST
 /_ah/mapreduce/worker_callback HTTP/1.1 500 2394
 http://1-dot-latest-dot-locationplay.appspot.com/_ah/mapreduce/worker_callback;
 AppEngine-Google; (+http://code.google.com/appengine)
 1-dot-latest-dot-locationplay.appspot.com ms=29696 cpu_ms=306889
 api_cpu_ms=305046 cpm_usd=8.524742 queue_name=default
 task_name=appengine-mrshard-15868331507823D8715B7-11-8 pending_ms=5160
 instance=00c61b117c29bef02c280156561831181c935a2e

 ·  E 2011-07-24 13:20:37.069

 The API call datastore_v3.Delete() took too long to respond and was
 cancelled.

 Traceback (most recent call last):

   File
 /base/python_runtime/python_lib/versions/1/google/appengine/ext/webapp/__init__.py,
 line 702, in __call__

     handler.post(*groups)

   File
 /base/python_runtime/python_lib/versions/1/google/appengine/ext/mapreduce/base_handler.py,
 line 81, in post

     self.handle()

   File
 /base/python_runtime/python_lib/versions/1/google/appengine/ext/mapreduce/handlers.py,
 line 184, in handle

     entity, input_reader, ctx, tstate)

   File
 /base/python_runtime/python_lib/versions/1/google/appengine/ext/mapreduce/handlers.py,
 line 254, in process_data

     output(ctx)

   File
 /base/python_runtime/python_lib/versions/1/google/appengine/ext/mapreduce/operation/db.py,
 line 87, in __call__

     context.mutation_pool.delete(self.entity)

   File
 /base/python_runtime/python_lib/versions/1/google/appengine/ext/mapreduce/context.py,
 line 171, in delete

     self.__flush_deletes()

   File
 /base/python_runtime/python_lib/versions/1/google/appengine/ext/mapreduce/context.py,
 line 189, in __flush_deletes

     datastore.Delete(self.deletes.items, rpc=self.__create_rpc())

   File
 /base/python_runtime/python_lib/versions/1/google/appengine/api/datastore.py,
 line 550, in Delete

     return DeleteAsync(keys, **kwargs).get_result()

   File
 /base/python_runtime/python_lib/versions/1/google/appengine/api/apiproxy_stub_map.py,
 line 592, in get_result

     return self.__get_result_hook(self)

   File
 /base/python_runtime/python_lib/versions/1/google/appengine/datastore/datastore_rpc.py,
 line 1464, in __delete_hook

     self.check_rpc_success(rpc)

   File
 /base/python_runtime/python_lib/versions/1/google/appengine/datastore/datastore_rpc.py,
 line 1072, in check_rpc_success

     rpc.check_success()

   File
 /base/python_runtime/python_lib/versions/1/google/appengine/api/apiproxy_stub_map.py,
 line 558, in check_success

     self.__rpc.CheckSuccess()

   File
 /base/python_runtime/python_lib/versions/1/google/appengine/api/apiproxy_rpc.py,
 line 133, in CheckSuccess

     raise self.exception

 DeadlineExceededError: The API call datastore_v3.Delete() took too long to
 respond and was cancelled.



 2011-07-24 13:20:37.282 /_ah/mapreduce/worker_callback 500 24793ms 879cpu_ms
 529api_cpu_ms 1kb AppEngine-Google; (+http://code.google.com/appengine)

 0.1.0.2 - - [24/Jul/2011:13:20:37 -0700] POST
 /_ah/mapreduce/worker_callback HTTP/1.1 500 1996
 http://1-dot-latest-dot-locationplay.appspot.com/_ah/mapreduce/worker_callback;
 AppEngine-Google; (+http://code.google.com/appengine)
 1-dot-latest-dot-locationplay.appspot.com ms=24793 cpu_ms=879
 api_cpu_ms=529 cpm_usd=0.024815 queue_name=default
 task_name=appengine-mrshard-15868331507823D8715B7-28-8 pending_ms=9561
 instance=00c61b117cbc93cdc72672caefcebc6c52198bc4

 E 2011-07-24 13:20:37.274

 Traceback (most recent call last):
   File
 /base/python_runtime/python_lib/versions/1/google/appengine/ext/webapp/__init__.py,
 line 702, in __call__
 handler.post(*groups)
   File
 /base/python_runtime/python_lib/versions/1/google/appengine/ext/mapreduce/base_handler.py,
 line 81, in post
 self.handle()
   File
 /base/python_runtime/python_lib/versions/1/google/appengine/ext/mapreduce/handlers.py,
 line 184, in handle
 entity, input_reader, ctx, tstate)
   File
 /base/python_runtime/python_lib/versions/1/google/appengine/ext/mapreduce/handlers.py,
 line 254, in process_data
 output(ctx)
   File
 /base/python_runtime/python_lib/versions/1/google/appengine/ext/mapreduce/operation/db.py,
 line 87, in __call__
 context.mutation_pool.delete(self.entity)
   File
 

Re: [google-appengine] Re: FAQ for out of preview pricing changes

[Gregory D'alesandre]

Hi Gijsbert, both of these are indeed still accurate.

Greg

On Mon, Jul 4, 2011 at 6:28 AM, Gijsbert gijsbert.de.h...@gmail.com wrote:

 Greg, can you confirm if a channel in the new pricing scheme is still
 available for 2 hours and can be reused for multiple clients (as long
 as they don't overlap in time)?

 On Jul 3, 1:39 pm, ksafez216 mell...@gmail.com wrote:
  You say that the new Channel API rates is what users pay today.
 
  First of all, in the original model you get over 8,000 free channels a
  day to 100/day in the new model.
 
  And please tell me if 1000's of users are exchanging messages on
  individual channels, does that mean that there will be have to 1000's
  of instances running at the same time?
 
  Things should get cheaper and simpler, not more expensive and more
  complicated!
 
  On May 17, 9:49 pm, Gregory D'alesandre gr...@google.com wrote:
 
 
 
 
 
 
 
   Hello All!
 
   As you've likely heard, when Google App Engine leaves Preview in the
 second
   half of 2011, the pricing model will change.  Prices are listed here:
 http://www.google.com/enterprise/appengine/appengine_pricing.html.  But
 that
   leaves a lot of questions unanswered, this FAQ is intended to help
 answer
   some of the frequently asked questions about the new model.  We are
   interested in hearing additional thoughts and comments you have based
 on
   this.  Once it is relatively stable I'll add it to our official docs.
  If
   you find there is something you want to know but it is not yet
 answered,
   just ask and I'll try to answer it as clearly as possible.  We've made
 some
   changes based on the feedback we've gotten (from this group in
 particular),
   they are bolded below but not updated on the external pages yet.  There
 are
   still blanks to fill in and I will be sending that information to this
 group
   first in order as it is available.  Finally, thank you for your
 questions
   and bearing with us as we are ironing out details, I and the whole App
   Engine team very much appreciate it.
 
   Greg D'Alesandre
   Senior Product Manager, Google App Engine
 
   ---
 
   *Definitions*
   *Instance*: A small virtual environment to run your code with a
 reserved
   amount of CPU and Memory.
   *Frontend Instance*: An Instance running your code and scaling
 dynamically
   based on the incoming requests but limited in how long a request can
 run.
   *Backend Instance*: An Instance running your code with limited scaling
 based
   on your settings and potentially starting and stopping based on your
   actions.
   *Scheduler*: Part of the App Engine infrastructure that determines
 which
   Instance should serve a request including whether or not a new Instance
 is
   needed.
 
   *Serving Infrastructure*
   Q: What’s an Instance?
   A: When App Engine starts running your code it creates a small virtual
   environment to run your code with a reserved amount of CPU and Memory.
  For
   example if you are running a Java app, we will start a new JVM for you
 and
   load your code into it.
 
   Q: Is an App Engine Instance similar to a VM from infrastructure
 providers?
   A: Yes and no, they both have a set amount of CPU and Memory allocated
 to
   them, but GAE instances don’t have the overhead of operating systems or
   other applications running, so a much larger percentage of the CPU and
   memory is considered “usable.” They also operate against high-level
 APIs and
   not down through layers of code to virtual device drivers, so it’s more
   efficient, and allows all the services to be fully managed.
 
   Q: How does GAE determine the number of Frontend Instances to run?
   A: For each new request, the Scheduler decides whether there is an
 available
   Instance for the request, the request should wait, or a new Instance
 should
   be created to service the request.  It looks at the number of
 Instances, the
   throughput of the Instances, and the number of requests waiting.  Based
 on
   that it predicts how long it will take before it can serve the request
 (aka
   the Pending Latency).  If it predicts the delay will be over 1 second,
 a new
   Instance is created.  If it looks like an Instance is no longer needed,
 it
   will take that Instance down.
 
   Q: Should I assume I will be charged for the number of Instances
 currently
   being shown in the Admin console?
   A: No, we are working to change the Scheduler to optimize the
 utilization of
   instances, so that number should go down somewhat.  If you are using
 Java,
   you can also make your app threadsafe and take advantage of handling
   concurrent requests.  You can look at that as an upper bound on how
 many
   Instances you will be charged for.
 
   Q: How can I control the number of instances running?
   A: With the new Scheduler you’ll have the ability to choose a set of
   parameters that will help you specify how many instances are spun up to
   serve your traffic.  More information about the specific parameters and
 

Re: [google-appengine] Fork-Join queue implementation question regarding overlapping joins for same index/batch

[Robert Kluin]

Hi,
  No, they won't be summed twice.  The update to the aggregate occurs
within a transaction, so one of the writes will fail and retry.  Even
if some of the update work units are refetched the marker entities
stored within the aggregates entity group have a revision number that
allows you to skip stale updates.



Robert


On Mon, Jul 25, 2011 at 04:39, Bert robertbcur...@gmail.com wrote:
 Hi,

 Based on Brett's talk
 http://www.google.com/url?sa=Dq=http://www.google.com/events/io/2010/sessions/high-throughput-data-pipelines-appengine.html,
 is it possible to have a scenario where you get 2 join tasks happening
 closer than 1 second(or given batch time duration) apart?

 A scenario where I think it may happen is as follows:
 Batch time 1 second
 Stall timeout is 30 seconds
 The first insert is added at hh:mm:29.9 (multiple inserts follow)
 Task_name is e.g. sumName-1-1237847
 The task will only execute at about hh:mm:30.9

 At hh:mm:30.0 a new task is inserted but the index remains the same
 (the task sumName-1-1237847 hasn't incremented index yet).
 The task_name is sumName-2-1237847 and it will execute at hh:mm:31.0
 (only 0.1 seconds after the previous task for the same index).

 In most cases this may be OK (because 2 updates in a second for an
 entity group is probably OK), but what happens if there are many
 records for index 1237847 (more than task can finish in 0.1 seconds)?
 Potentially task sumName-2-1237847 could start executing before task
 sumName-1-1237847 has had time to finish executing, in which case it
 will also retrieve the work items for index 1237847 and may sum the
 same values twice.

 Here's the code (from the slides):

 now = time.time()
 stallTimeout =30
 try:
  taskqueue.add(
  name='%s-%d-%d' % (
  sum_name, int(now / stallTimeout ), index),
  url='/work',
  eta=datetime.datetime.utcfromtimestamp(now) +
  datetime.timedelta(seconds=1))
 except taskqueue.TaskAlreadyExistsError:
  pass


 def join(sum_name, index):
  # force new writers to use the next index
  memcache.incr('index-' + sum_name)
  lock = '%s-lock-%d' % (sum_name, index)
  memcache.decr(lock, 2**15) # You missed the boat
  # busy wait for writers
  for i in xrange(20): # timeout after 5s
    counter = memcache.get(lock)
    if counter is None or int(counter) = 2**15:
      break
    time.sleep(0.250)
  #get work for index
  #sum or whatever you want
  #delete work

 If what I've said is valid, is a workaround to check the lock at join
 time to see if its already been decremented by 2**15 before starting
 the join as that would indicate it's already been run?

 def join(sum_name,index):
  lock = '%s-lock-%d' % (sum_name, index)
  counter =memcache.get(lock)
  if int(counter)  2**16:
    #the task for index has already started running, return to prevent
 overlap
    return False

   #rest of code here...

 Thanks
 Rob

 --
 You received this message because you are subscribed to the Google Groups 
 Google App Engine group.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to 
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at 
 http://groups.google.com/group/google-appengine?hl=en.



-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] TransactionFailedError: Is the transaction committed or not?

[Robert Kluin]

Hi Pol,
  Generally you will probably want to execute it again.


Robert





On Mon, Jul 25, 2011 at 03:53, Pol i...@pol-online.net wrote:
 Hi,

 The exception says: TransactionFailedError: The transaction could not
 be committed. Please try again.

 The doc at 
 http://code.google.com/appengine/docs/python/datastore/transactions.html
 says: You can receive Timeout, TransactionFailedError, or
 InternalError exceptions in cases where transactions have been
 committed and eventually will be applied successfully.

 So if you get TransactionFailedError, to you need to execute the
 transaction again or will it automatically be applied later?

 - Pol

 --
 You received this message because you are subscribed to the Google Groups 
 Google App Engine group.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to 
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at 
 http://groups.google.com/group/google-appengine?hl=en.



-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Impossible to update backend

[Robert Kluin]

Did you get this fixed?

If not, I'd suggest submitting a production issue.

http://code.google.com/p/googleappengine/issues/entry?template=Production%20issue


Robert



On Saturday, July 16, 2011, Eduardo Garcia egar...@stream18.com wrote:
 Since last maintenance I have been experiencing problems with backends:
 - dynamic backend instances dissapeared after some time executing (I have had 
 to sitch my two Backends to static/Resident now)
 - one of my resident backends' instances still continue dissapearing.

 Worst of all, when I try to update this backend (B2, 1 instance, resident), 
 the appcfg tool always respond with the Unable to update app: Version not 
 ready message, after some minutes trying to upload.
 I have tried to delete the backend, with no change.

 This backend should be running as part of an application that is now life and 
 in Marketplace (we are paying for it), is there any way to fix this ASAP?


 --
 You received this message because you are subscribed to the Google Groups 
 Google App Engine group.
 To view this discussion on the web visit 
 https://groups.google.com/d/msg/google-appengine/-/L41G3Oru0-QJ.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to 
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at 
 http://groups.google.com/group/google-appengine?hl=en.


-- 
--
Robert Kluin
Ezox Systems, LLC

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Unable to create Socket in IE7 from second time

[Robert Kluin]

If you're still having the issue you might search the bug tracker for
similar issues and submit a new one if needed.

http://code.google.com/p/googleappengine/issues/entry?template=Other%20defect



Robert




On Thursday, July 14, 2011, ramesh chiluverirame...@gmail.com wrote:
 Hi,

 I am having problem while connecting with Google server to establish
 channel and create socket with that channel for second time in IE7
 browser.

 I am able to create connection and socket in all other browser like
 Chrome,firefox ,IE8  IE9.

 In IE7 browser , i am able to create channel and socket only one time.
 Once i closed the socket,  if i try to create one more socket, it is
 not allowing me to create socket.

 Please help me .

 Thanks
 Ramesh

 --
 You received this message because you are subscribed to the Google Groups 
 Google App Engine group.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to 
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at 
 http://groups.google.com/group/google-appengine?hl=en.



-- 
--
Robert Kluin
Ezox Systems, LLC

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] HOW TO DISPLAY CHANGING INFORMATION IN GOOGLE APP

[Robert Kluin]

Store your data in the datastore:
http://code.google.com/appengine/docs/python/datastore/

You can require login to access the pages that allow changes to be made:
http://code.google.com/appengine/docs/python/config/appconfig.html#Requiring_Login_or_Administrator_Status



Robert



On Wednesday, July 13, 2011, tioluwa tolucoll...@gmail.com wrote:
 I'm new to google app, and i wan't to design an app that shows a
 certain number of pictures, and a map that places markers at each
 pictures geographic location. This data containing the images and
 their maps can be changed periodically.

 What method can i use to store these data on the google app engine, in
 such a way that they can be changed, by administrative permission,
 while others can only view the particular image data uploaded.

 --
 You received this message because you are subscribed to the Google Groups 
 Google App Engine group.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to 
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at 
 http://groups.google.com/group/google-appengine?hl=en.



-- 
--
Robert Kluin
Ezox Systems, LLC

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.