Re: [google-appengine] Trouble reading a pile of stuff from the datastore
You haven't said if you running this in a task or front end request, If it is a frontend request it will never run to completion as you have a limit of 60 seconds. Also look at your default batch size. You may find that a bigger batch size will speed things up too. T On Wednesday, October 8, 2014 1:50:44 AM UTC+8, Joshua Smith wrote: Yup, cursors are a good workaround. Here’s my fix… hcur = None keepGoing = True while keepGoing: count = 0 hall = HitModel.all() if hcur: hall.with_cursor(start_cursor=hcur) keepGoing = False for h in hall: ...my processing stuff... count += 1 if count == 2: hcur = hall.cursor() keepGoing = True break On Oct 7, 2014, at 11:38 AM, PK p...@gae123.com javascript: wrote: I have seen this error as well and had to change my code. My situation is python 2.7/HRD/db.Query() in a module with manual scaling. q = db.Query(…) … for ent in q.run(): do stuff The iteration goes well for a large number of entities and then gives up in a similar way. Something seems to timeout for long lived queries. Needless to say it works fine on the dev server but there of course I do not have so much data. Breaking it down with cursors or equivalent works fine and this is what I am doing as a work around. I did not even bother to see if there is an issue for it but I would happily star it if there is one. PK http://www.gae123.com On October 7, 2014 at 8:26:11 AM, Joshua Smith (mrjoshu...@gmail.com javascript:) wrote: I have an app that once a day does a big data processing task. Every now and then it would throw a datastore timeout error. But now it’s throwing them constantly. I thought maybe my data had tripped over some limit on how much you can read, but I just added some instrumentation and it’s only reading less than half of the entities. If I was tripping over an undocumented limit, I’d think it would read all most all of them (since only a few get added each day). Basically, the code is simply: for h in HitModel.all(): (do collect up info about h) and there are about 85K HitModel objects in the database. It’s dying after reading 35,000 of them (which takes about a minute). It’s on HR data store. Still on Python 2.5. App ID is “kaon-log” The error I’m getting is: 2014-10-07 11:16:46.925 The datastore operation timed out, or the data was temporarily unavailable. Traceback (most recent call last): File /base/data/home/runtimes/python/python_lib/versions/1/google/appengine/ext/webapp/_webapp25.py, line 714, in __call__ handler.get(*groups) File /base/data/home/apps/s~kaon-log/33.379217403803985923/main.py, line 648, in get for h in HitModel.all(): File /base/data/home/runtimes/python/python_lib/versions/1/google/appengine/ext/db/__init__.py, line 2326, in next return self.__model_class.from_entity(self.__iterator.next()) File /base/data/home/runtimes/python/python_lib/versions/1/google/appengine/datastore/datastore_query.py, line 3091, in next next_batch = self.__batcher.next_batch(Batcher.AT_LEAST_OFFSET) File /base/data/home/runtimes/python/python_lib/versions/1/google/appengine/datastore/datastore_query.py, line 2977, in next_batch batch = self.__next_batch.get_result() File /base/data/home/runtimes/python/python_lib/versions/1/google/appengine/api/apiproxy_stub_map.py, line 612, in get_result return self.__get_result_hook(self) File /base/data/home/runtimes/python/python_lib/versions/1/google/appengine/datastore/datastore_query.py, line 2710, in __query_result_hook self._batch_shared.conn.check_rpc_success(rpc) File /base/data/home/runtimes/python/python_lib/versions/1/google/appengine/datastore/datastore_rpc.py, line 1333, in check_rpc_success raise _ToDatastoreError(err) Timeout: The datastore operation timed out, or the data was temporarily unavailable. Any ideas? (Breaking this up into multiple tasks would be really hard.) -Joshua -- You received this message because you are subscribed to the Google Groups Google App Engine group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com javascript:. To post to this group, send email to google-a...@googlegroups.com javascript:. Visit this group at http://groups.google.com/group/google-appengine. For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine. For more options, visit
Re: [google-appengine] Trouble reading a pile of stuff from the datastore
Well I did say: I have an app that once a day does a big data processing task. So, obviously, it's a cron, right? Yeah, this isn't the time limit. It's apparently a bug in the database infrastructure, as PK confirmed. (If it was the time limit, then my work-around wouldn't have worked!) -Joshua On Oct 8, 2014, at 6:28 AM, timh zutes...@gmail.com wrote: You haven't said if you running this in a task or front end request, If it is a frontend request it will never run to completion as you have a limit of 60 seconds. Also look at your default batch size. You may find that a bigger batch size will speed things up too. T On Wednesday, October 8, 2014 1:50:44 AM UTC+8, Joshua Smith wrote: Yup, cursors are a good workaround. Here's my fix... hcur = None keepGoing = True while keepGoing: count = 0 hall = HitModel.all() if hcur: hall.with_cursor(start_cursor=hcur) keepGoing = False for h in hall: ...my processing stuff... count += 1 if count == 2: hcur = hall.cursor() keepGoing = True break On Oct 7, 2014, at 11:38 AM, PK p...@gae123.com wrote: I have seen this error as well and had to change my code. My situation is python 2.7/HRD/db.Query() in a module with manual scaling. q = db.Query(...) ... for ent in q.run(): do stuff The iteration goes well for a large number of entities and then gives up in a similar way. Something seems to timeout for long lived queries. Needless to say it works fine on the dev server but there of course I do not have so much data. Breaking it down with cursors or equivalent works fine and this is what I am doing as a work around. I did not even bother to see if there is an issue for it but I would happily star it if there is one. PK http://www.gae123.com On October 7, 2014 at 8:26:11 AM, Joshua Smith (mrjoshu...@gmail.com) wrote: I have an app that once a day does a big data processing task. Every now and then it would throw a datastore timeout error. But now it's throwing them constantly. I thought maybe my data had tripped over some limit on how much you can read, but I just added some instrumentation and it's only reading less than half of the entities. If I was tripping over an undocumented limit, I'd think it would read all most all of them (since only a few get added each day). Basically, the code is simply: for h in HitModel.all(): (do collect up info about h) and there are about 85K HitModel objects in the database. It's dying after reading 35,000 of them (which takes about a minute). It's on HR data store. Still on Python 2.5. App ID is kaon-log The error I'm getting is: 2014-10-07 11:16:46.925 The datastore operation timed out, or the data was temporarily unavailable. Traceback (most recent call last): File /base/data/home/runtimes/python/python_lib/versions/1/google/appengine/ext/webapp/_webapp25.py, line 714, in __call__ handler.get(*groups) File /base/data/home/apps/s~kaon-log/33.379217403803985923/main.py, line 648, in get for h in HitModel.all(): File /base/data/home/runtimes/python/python_lib/versions/1/google/appengine/ext/db/__init__.py, line 2326, in next return self.__model_class.from_entity(self.__iterator.next()) File /base/data/home/runtimes/python/python_lib/versions/1/google/appengine/datastore/datastore_query.py, line 3091, in next next_batch = self.__batcher.next_batch(Batcher.AT_LEAST_OFFSET) File /base/data/home/runtimes/python/python_lib/versions/1/google/appengine/datastore/datastore_query.py, line 2977, in next_batch batch = self.__next_batch.get_result() File /base/data/home/runtimes/python/python_lib/versions/1/google/appengine/api/apiproxy_stub_map.py, line 612, in get_result return self.__get_result_hook(self) File /base/data/home/runtimes/python/python_lib/versions/1/google/appengine/datastore/datastore_query.py, line 2710, in __query_result_hook self._batch_shared.conn.check_rpc_success(rpc) File /base/data/home/runtimes/python/python_lib/versions/1/google/appengine/datastore/datastore_rpc.py, line 1333, in check_rpc_success raise _ToDatastoreError(err) Timeout: The datastore operation timed out, or the data was temporarily unavailable. Any ideas? (Breaking this up into multiple tasks would be really hard.) -Joshua -- You received this message because you are subscribed to the Google Groups Google App Engine group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-a...@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine. For more options, visit https://groups.google.com/d/optout. -- You received this message because you are
[google-appengine] Re: Help Needed in handling Forms (GAE PHP)
I think Vinny P is on the right track. Only thing I can see (as stated earlier) is the PHP trying to upload as static files. Put the files back where they were then replace: - url: /scripts static_dir: static_website/scripts with - url: /(.+\.php)$ script: \1 https://cloud.google.com/appengine/docs/php/config/appconfig#PHP_app_yaml_About_app_yaml And try the upload again. On Thursday, October 2, 2014 7:34:31 AM UTC-4, Anmol Parashar wrote: I'm trying to set up my new website using *Google App Engine (PHP)*. I have bought the domain and my website is working fine. Well, kind of. I have two forms on my website, one asks for just an email and other a contact form. Both of them accepts input but can't send the input to where they are supposed to. A message is displayed, PHP scripts/files are either not configured properly or are missing. Contact your host. *Here's the thing*, I made my website using Adobe Muse and their forms uses PHP. I have provided all my static html files to Google App Engine (which I'm using as a host) using the Deploy function but whenever I hit deploy, most of the files are read and uploaded except the 8 .php files. The error says, *Mimetype can't be recognized for *.php. Using application octet stream instead.* *How do I upload my .php files?* (What to write in my app.yaml) I can use the *script:* operative for lets say *index.php* but that page only requires two more php files to run the form and we can't use *script:* 3 times for a single page. *If you have any knowledge of HTML or HTML5, can you tell me how to make a form that doesn't use PHP?* I'm not using a database. The only thing the form has to do is to send the input to an email. My website is Subscriber Shop http://subscribershop.com. My app.yaml currently, application: sub version: 1 runtime: php api_version: 1 threadsafe: yes handlers: - url: /favicon\.ico static_files: favicon.ico upload: favicon\.ico - url: / static_files: static_website/index.html upload: static_website/index.html - url: /index.html static_files: static_website/index.html upload: static_website/index.html #specific html pages: - url: /about-us.html static_files: static_website/about-us.html upload: static_website/about-us.html - url: /faqs.html static_files: static_website/faqs.html upload: static_website/faqs.html - url: /drop-a-word.html static_files: static_website/drop-a-word.html upload: static_website/drop-a-word.html - url: /unsubscribe script: static_website/jl3sxnhv7vv.php - url: /css static_dir: static_website/css - url: /images static_dir: static_website/images - url: /scripts static_dir: static_website/scripts - url: /phone static_dir: static_website/phone - url: /.* static_files: static_website/what-what.html upload: static_website/what-what.html inbound_services: - mail - xmpp_subscribe My index page, contact page and another are supposed to have these 3 forms and they use a total of 8 .php files that are in *static_website/scripts* Can anyone please help? *PS:* The support at Adobe Muse says that PHP is not configured. Contact the server admin. How do I tell Google To enable php? -- You received this message because you are subscribed to the Google Groups Google App Engine group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine. For more options, visit https://groups.google.com/d/optout.
[google-appengine] Domain wide delegation to an App Engine Application...?
I'd like to grant Domain Level access for the Admin SDK (specifically the User API) to my Google App Engine Application. Per this page: https://developers.google.com/admin-sdk/directory/v1/guides/delegation I should be able to grant the access here: Admin Console Security Advanced settings Manage API client access However, no matter what I put in for the Client Name, I can't get it to work. I've tried the following: * Application Identifier * Service Account Name * Application Default Version URL * Application Identifier Alias I pulled all of these from these from the Applications Settings of the My Applications page. I assume that I could set up a Service Account, but it seems to me that I should be able to grant access directly to the application. Can anyone help? Thanks! William -- You received this message because you are subscribed to the Google Groups Google App Engine group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine. For more options, visit https://groups.google.com/d/optout.
[google-appengine] Re: Configuring Cloud SQL and JPA with Eclipse
These log is local o remote ? On Saturday, October 4, 2014 1:53:14 PM UTC-5, Nick wrote: However I continue to get this error: Exception [EclipseLink-4021] (Eclipse Persistence Services - 2.5.2.v20140319-9ad6abd): org.eclipse.persistence.exceptions.DatabaseException Exception Description: Unable to acquire a connection from driver [null], user [null] and URL [null]. Verify that you have set the expected driver class and URL. Check your login, persistence.xml or sessions.xml resource. The jdbc.driver property should be set to a class that is compatible with your database platform Any insight on this problem would be great as I would like to get back to developing my app. Below is the relevant code: *persistence.xml* ?xml version=1.0 encoding=UTF-8 ?persistence xmlns=http://java.sun.com/xml/ns/persistence; xmlns:xsi=http://www.w3.org/2001/XMLSchema-instance; xsi:schemaLocation=http://java.sun.com/xml/ns/persistence http://java.sun.com/xml/ns/persistence/persistence_1_0.xsd; version=1.0 persistence-unit name=test_pu providerorg.eclipse.persistence.jpa.PersistenceProvider/provider properties property name=javax.persistence.jdbc.user value=root / property name=eclipselink.ddl-generation value=create-tables / /properties/persistence-unit *appengine-web.xml* *?xml version=1.0 encoding=utf-8? appengine-web-app xmlns=http://appengine.google.com/ns/1.0 http://appengine.google.com/ns/1.0 applicationredacted/application version1/version threadsafetrue/threadsafe use-google-connector-jtrue/use-google-connector-j static-files include path=** / include path=**.nocache.* expiration=0s / include path=**.cache.* expiration=365d / exclude path=**.gwt.rpc / /static-files system-properties property name=java.util.logging.config.file value=WEB-INF/logging.properties/ property name=cloudsql.url.dev value=jdbc:mysql://127.0.0.1:3306/TestDB?user=root http://127.0.0.1:3306/TestDB?user=root/ property name=cloudsql.url value=jdbc:google:mysql://redacted:main/TestDB?user=root/ /system-properties /appengine-web-app* *project structure* http://i.stack.imgur.com/BSIIp.png https://lh6.googleusercontent.com/-qAaKLQ1uC1I/VDBAVkAL8cI/ANY/VgWP00lIVXQ/s1600/jpa%2Bproperties.png http://i.stack.imgur.com/BSIIp.png -- You received this message because you are subscribed to the Google Groups Google App Engine group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine. For more options, visit https://groups.google.com/d/optout.
[google-appengine] ndb.TextProperty() encoding
Hi, I am completely new at appEngine and I don't have experience on web development. I am trying to create an app which the user can write a text in any language. The problem is that the site is not showing special characters as: ç, á, â, 我, στ ... However, when I run the site locally on my computer, it shows perfectly... with any special characters that I enter... I am using *ndb.TextProperty()*: class Opiniao(ndb.Model): autor = ndb.TextProperty( ) chave = ndb.StringProperty() resposta = ndb.TextProperty( ) foto = ndb.BlobProperty() data = ndb.DateTimeProperty(auto_now_add=True) dia = ndb.DateProperty(auto_now_add=True) I don't understand why it doesn't store utf-8 characters. Thanks in advance! -- You received this message because you are subscribed to the Google Groups Google App Engine group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine. For more options, visit https://groups.google.com/d/optout.
