Re: [google-appengine] get_num_instances always raises InvalidVersionError
Sorry, I meant if I put in the names of the module I want, 'default', or even if I omit all parameters, the error is raised. It was my impression that if the parameters were not supplied, it should pick up the current module/version and return that information to me. On Sun, Mar 16, 2014 at 11:32 AM, Vinny P vinny...@gmail.com wrote: On Wed, Mar 12, 2014 at 3:14 PM, someone1 someo...@gmail.com wrote: I am trying to leverage this new API functionality but no matter what inputs I choose, the modules.get_num_instances function always raises an error. I know I have basic scaling setup for the module and version I'm trying to look this information up for (in the YAML file). Calls to get modules.get_versions works just fine. In my dev server I am passing in my module YAML files as part of my startup script. When you say no matter what inputs I choose can you expand further on that? get_num_instances requires the name of the module to collect information about, are you supplying anything different to the function? - -Vinny P Technology Media Advisor Chicago, IL App Engine Code Samples: http://www.learntogoogleit.com -- You received this message because you are subscribed to a topic in the Google Groups Google App Engine group. To unsubscribe from this topic, visit https://groups.google.com/d/topic/google-appengine/PuUMW8aLknQ/unsubscribe . To unsubscribe from this group and all its topics, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine. For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine. For more options, visit https://groups.google.com/d/optout.
[google-appengine] Question regarding MapReduce library
Hello, I am trying to understand running a map-reduce job a little better. I know a lot of datastore operations are required to keep the state of the job, but how does the mapreduce library keep track of yielded data? I am running a job to process over 13 million entities and normalize some data to dump into Google Storage (approximately 15GB of data). When I use the FileOutputWriter, where does it keep track of each line I've yielded? How do I end up with only 1 large file written to Google Storage? I looked at the bucket during a map-reduce operation and I don't see anything until the job is done and there's one large file ready for me to use. Does each Shard aggregate the data into a blobstore object before a final step merges all the shards' data and writes it to GCS? How is the library able to do this with F1 instances and memory constraints? I was not able to easily follow the code behind all this so I was hoping someone who is more familiar with the process can shed some light. I have other use cases in which I need to process a lot of data and would like to end up with a single large output file, but my method isn't the most stable of processes and does not fit into a map-reduce job. If I knew the general logic behind aggregating the data and placing it into Google Storage this would be of great benefit to me. Any and all insight would be greatly appreciated! Thank you, Prateek -- You received this message because you are subscribed to the Google Groups Google App Engine group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine. For more options, visit https://groups.google.com/groups/opt_out.
Re: [google-appengine] Re: 1.8.9 Pre-Release SDKs are now available.
A reflective code sample: import pytz import datetime str(pytz.utc.localize(datetime.datetime.now())) This part of our code has been working fine for months with no change, though looking at it now I think the change occurred on one of the service API's we use from a 3rd party. My apologies on jumping the gun there. Thank you, Prateek Malhotra Project Manager/Lead Web Applications Developer ReferABuyer.com - Developer of the Automatic$eller Software! 1973 Washington Valley Road Martinsville, NJ 08836 Phone: 732-868-4421 Fax: 732-356-5875 http://www.referabuyer.com prat...@referabuyer.com On Wed, Jan 15, 2014 at 3:43 PM, R Samuel Klatchko r.samuel.klatc...@gmail.com wrote: Hi Prateek- I'm unable to reproduce your results. If I use: str(datetime.datetime.now()) str(datetime.datetime.today()) I do not see the timezone offset in either 1.8.8 or 1.8.9. On the other hand, if I use: class faketz(datetime.tzinfo): def utcoffset(self, _): return datetime.timedelta(0) def dst(self, _): return datetime.timedelta(0) datetime.datetime.now(faketz()) I see the timezone offset in both 1.8.8 and 1.8.9. Can you provide more detailed reproduction steps. samuel On Tuesday, January 14, 2014 3:26:28 PM UTC-8, Prateek Malhotra wrote: Today all of the sudden in my Python app the code `str(datetime.datetime)` started producing results with a time zone offset (e.g. +) at the end where it wasn't done before. The only difference I can see to my app is the change from 1.8.8 to 1.8.9. It broke some code that was expecting datetime information in a certain format which we have since fixed but it was a nasty surprise none-the-less. Was this an intended side effect of the SDK update? On Friday, January 10, 2014 7:26:59 PM UTC-5, Richmond Manzana wrote: Happy New Year! We wanted to inform you that the pre-release SDKs for Python, PHP and Java are now available and are found here: http://code.google.com/p/googleappengine/downloads/list Please see the pre-release notes below. Cheers, Richmond Manzana Technical Program Manager Google App Engine App Engine SDK - Pre-Release Notes Version 1.8.9 Python PHP == - Renamed appcfg start to appcfg start_module_version and stop to stop_module_version. - Fixed an issue with ID allocation collisions in the Datastore. https://code.google.com/p/googleappengine/issues/detail?id=10134 - Fixed an issue with Cloud Storage calls failing in the dev_appserver. https://code.google.com/p/googleappengine/issues/detail?id=10181 https://code.google.com/p/googleappengine/issues/detail?id=10185 - Fixed an issue with the dev_appserver not starting on Windows when using versions of Python earlier than 2.7.2. https://code.google.com/p/googleappengine/issues/detail?id=10363 Python == - Deprecated start_module, start_module_async, stop_module, stop_module_async get_modules_async, get_versions_async, get_default_version_async, get_num_instances_async, get_hostname_async and added start_version, start_version_async, stop_version, stop_version_async as replacements in the Modules API. - Added verbose_name support for ComputedProperty in NDB. https://code.google.com/p/appengine-ndb-experiment/ issues/detail?id=239 - Fixed an issue with deffered.defer not waiting for async operations. https://code.google.com/p/appengine-ndb-experiment/ issues/detail?id=238 - Fixed an issue with LocalStructureProperty not handling None value in NDB. https://code.google.com/p/appengine-ndb-experiment/ issues/detail?id=233 - Fixed an issue NDB not working with sqlite. https://code.google.com/p/googleappengine/issues/detail?id=8381 - Fixed an issue with debug not working with PyDev in 1.8.8. https://code.google.com/p/googleappengine/issues/detail?id=10390 - Fixed an issue with object properties not being retrieved in NDB. https://code.google.com/p/googleappengine/issues/detail?id=10467 PHP = - Renamed startModule, stopModule to startVersion, stopVersion in the Modules API. - Added support for the ftp extension. - Added support for the zip extension. - Added support for the gethostname() function. - A proper billing required message is now given when users attempt to use Sockets API on free apps. - Fixed an issue with header keys and values having whitespaces preventing responses from being compressed. - Fixed an issue with content-type and metadata not being updated when renaming Google Cloud Storage objects. - Fixed an issue with Cloud Storage rename requests failing when an object has a space in its name. Java == - Renamed appcfg start to appcfg start_module_version and stop to stop_module_version. - The following methods are deprecated in the Modules API and will be removed in an upcoming release
[google-appengine] Re: 1.8.9 Pre-Release SDKs are now available.
Today all of the sudden in my Python app the code `str(datetime.datetime)` started producing results with a time zone offset (e.g. +) at the end where it wasn't done before. The only difference I can see to my app is the change from 1.8.8 to 1.8.9. It broke some code that was expecting datetime information in a certain format which we have since fixed but it was a nasty surprise none-the-less. Was this an intended side effect of the SDK update? On Friday, January 10, 2014 7:26:59 PM UTC-5, Richmond Manzana wrote: Happy New Year! We wanted to inform you that the pre-release SDKs for Python, PHP and Java are now available and are found here: http://code.google.com/p/googleappengine/downloads/list Please see the pre-release notes below. Cheers, Richmond Manzana Technical Program Manager Google App Engine App Engine SDK - Pre-Release Notes Version 1.8.9 Python PHP == - Renamed appcfg start to appcfg start_module_version and stop to stop_module_version. - Fixed an issue with ID allocation collisions in the Datastore. https://code.google.com/p/googleappengine/issues/detail?id=10134 - Fixed an issue with Cloud Storage calls failing in the dev_appserver. https://code.google.com/p/googleappengine/issues/detail?id=10181 https://code.google.com/p/googleappengine/issues/detail?id=10185 - Fixed an issue with the dev_appserver not starting on Windows when using versions of Python earlier than 2.7.2. https://code.google.com/p/googleappengine/issues/detail?id=10363 Python == - Deprecated start_module, start_module_async, stop_module, stop_module_async get_modules_async, get_versions_async, get_default_version_async, get_num_instances_async, get_hostname_async and added start_version, start_version_async, stop_version, stop_version_async as replacements in the Modules API. - Added verbose_name support for ComputedProperty in NDB. https://code.google.com/p/appengine-ndb-experiment/issues/detail?id=239 - Fixed an issue with deffered.defer not waiting for async operations. https://code.google.com/p/appengine-ndb-experiment/issues/detail?id=238 - Fixed an issue with LocalStructureProperty not handling None value in NDB. https://code.google.com/p/appengine-ndb-experiment/issues/detail?id=233 - Fixed an issue NDB not working with sqlite. https://code.google.com/p/googleappengine/issues/detail?id=8381 - Fixed an issue with debug not working with PyDev in 1.8.8. https://code.google.com/p/googleappengine/issues/detail?id=10390 - Fixed an issue with object properties not being retrieved in NDB. https://code.google.com/p/googleappengine/issues/detail?id=10467 PHP = - Renamed startModule, stopModule to startVersion, stopVersion in the Modules API. - Added support for the ftp extension. - Added support for the zip extension. - Added support for the gethostname() function. - A proper billing required message is now given when users attempt to use Sockets API on free apps. - Fixed an issue with header keys and values having whitespaces preventing responses from being compressed. - Fixed an issue with content-type and metadata not being updated when renaming Google Cloud Storage objects. - Fixed an issue with Cloud Storage rename requests failing when an object has a space in its name. Java == - Renamed appcfg start to appcfg start_module_version and stop to stop_module_version. - The following methods are deprecated in the Modules API and will be removed in an upcoming release: getModulesAsync getVersionsAsync getDefaultVersionAsync getNumInstancesAsync getVersionHostnameAsync getInstanceHostnameAsync startModule (replaced by startVersion) startModuleAsync (replaced by startVersionAsync) stopModule (replaced by stopVersion) stopModuleAsync (replaced by stopVersionAsync) getModuleHostname (replaced by getVersionHostname and getInstanceHostname) getModuleHostnameAsync - The following exceptions have been deprecated in the Modules API and will be removed in a following release: InvalidInstanceException InvalidModuloeException InvalidVersionException - The following exceptions have been removed in the Modules API: ModuleAlreadyStartedException ModuleAlreadyStoppedException - Modules are now supported in the dev_appserver Development Console. - Fixed an issue with ID allocation collisions in the Datastore. https://code.google.com/p/googleappengine/issues/detail?id=10134 -- You received this message because you are subscribed to the Google Groups Google App Engine group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to
[google-appengine] Admin section of non-default version not working
I keep getting Server 500 errors when I try to access the admin section of my application on the non-default version. It works fine locally, it was working fine last week (no modifications since), and it works fine on the default version (only from my custom domain, NOT from default_version.appid.appspot.com). The logs just show 301/302 responses, but no errors. I've tried to upload the new version again, but it still doesn't work. I need to feel comfortable rolling out the new version of my app, but I can't get into the administrator backend to run tests! This is a major problem for me that seems to be a production issue and out of my control. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
[google-appengine] Re: Admin section of non-default version not working
The URL the 500 error appears on is as follows: http://app_version.app_id.appspot.com/_ah/conflogin On Tuesday, May 28, 2013 4:53:19 PM UTC-4, Prateek Malhotra wrote: I keep getting Server 500 errors when I try to access the admin section of my application on the non-default version. It works fine locally, it was working fine last week (no modifications since), and it works fine on the default version (only from my custom domain, NOT from default_version.appid.appspot.com). The logs just show 301/302 responses, but no errors. I've tried to upload the new version again, but it still doesn't work. I need to feel comfortable rolling out the new version of my app, but I can't get into the administrator backend to run tests! This is a major problem for me that seems to be a production issue and out of my control. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
[google-appengine] TaskQueue Issues
My tasks are increasing in retry count but I have no logs showing they ever executed. My backends are shutting down pre-maturely (that process these tasks) and again, no errors or anything in the logs explaining why. Not sure if this is a backend or taskqueue issue. Here is the latency graph for task queues: https://lh6.googleusercontent.com/-WUcDcWegm70/UVsHDGm6YcI/ABQ/C09Ysu8CarE/s1600/chart.png -- You received this message because you are subscribed to the Google Groups Google App Engine group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
Re: [google-appengine] Re: TaskQueue Issues
I am using HRD. No changes to code, we've been doing the same thing for months. I remember this happening in the past, tasks just failing randomly with no logs. They eventually go through, but the instances usually restart before the tasks finish. CPU/Memory usage look fine the entire life of the process I run. I use python so there's no log4j but I use the default settings, so all logging messages, including debug, make it into my log. The problem seems to have gone away, tasks are executing fine again and my backends aren't shutting down. It may have been a small issue with the task queue service. My tasks are assigned to specific back end instances so the fact that they kept failing and the backend instances shutdown and come back shows that the task was having an error connecting to the backend which caused it to just shutdown due to inactivity, and finally when the task queue started connecting again, the instances came back up. Prateek Malhotra Project Manager/Lead Web Applications Developer ReferABuyer.com - Developer of the Automatic$eller Software! 1973 Washington Valley Road Martinsville, NJ 08836 Phone: 732-868-4421 Fax: 732-356-5875 http://www.referabuyer.com prat...@referabuyer.com On Tue, Apr 2, 2013 at 1:42 PM, Vinny P vinny...@gmail.com wrote: Hello Prateek, Are you running on Master/Slave or HRD? I recall similar occurrences happening for me on M/S before I migrated. If you're running on HRD, a few other questions: 1. Can you provide a screenshot of your logs? I realize that you said your tasks are showing no logs, but there could be a diamond in the rough that you're missing. 2. What logging level have you specified in log4j? 3. Have you changed your application code recently? If so, can we see source code? My first thought is that your backend is the problem here. The fact that the retry count is increasing means that AppEngine is (probably!) executing tasks just fine, but your backend is failing and responding with error codes. What is your backend doing? Does it depend on any non-GAE APIs? - -Vinny P Technology Media Advisor Chicago, IL @GOV on AppDotNet: https://alpha.app.net/gov On Tuesday, April 2, 2013 11:29:53 AM UTC-5, Prateek Malhotra wrote: My tasks are increasing in retry count but I have no logs showing they ever executed. My backends are shutting down pre-maturely (that process these tasks) and again, no errors or anything in the logs explaining why. Not sure if this is a backend or taskqueue issue. Here is the latency graph for task queues: https://lh6.googleusercontent.com/-WUcDcWegm70/UVsHDGm6YcI/ABQ/C09Ysu8CarE/s1600/chart.png -- You received this message because you are subscribed to a topic in the Google Groups Google App Engine group. To unsubscribe from this topic, visit https://groups.google.com/d/topic/google-appengine/4CYIHFua1Kw/unsubscribe?hl=en . To unsubscribe from this group and all its topics, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine?hl=en. For more options, visit https://groups.google.com/groups/opt_out. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
[google-appengine] Re: TaskQueue Issues
(Reposting without signature) I am using HRD. No changes to code, we've been doing the same thing for months. I remember this happening in the past, tasks just failing randomly with no logs. They eventually go through, but the instances usually restart before the tasks finish. CPU/Memory usage look fine the entire life of the process I run. I use python so there's no log4j but I use the default settings, so all logging messages, including debug, make it into my log. The problem seems to have gone away, tasks are executing fine again and my backends aren't shutting down. It may have been a small issue with the task queue service. My tasks are assigned to specific back end instances so the fact that they kept failing and the backend instances shutdown and come back shows that the task was having an error connecting to the backend which caused it to just shutdown due to inactivity, and finally when the task queue started connecting again, the instances came back up. Thank you, Prateek On Tuesday, April 2, 2013 1:42:21 PM UTC-4, Vinny P wrote: Hello Prateek, Are you running on Master/Slave or HRD? I recall similar occurrences happening for me on M/S before I migrated. If you're running on HRD, a few other questions: 1. Can you provide a screenshot of your logs? I realize that you said your tasks are showing no logs, but there could be a diamond in the rough that you're missing. 2. What logging level have you specified in log4j? 3. Have you changed your application code recently? If so, can we see source code? My first thought is that your backend is the problem here. The fact that the retry count is increasing means that AppEngine is (probably!) executing tasks just fine, but your backend is failing and responding with error codes. What is your backend doing? Does it depend on any non-GAE APIs? - -Vinny P Technology Media Advisor Chicago, IL @GOV on AppDotNet: https://alpha.app.net/gov On Tuesday, April 2, 2013 11:29:53 AM UTC-5, Prateek Malhotra wrote: My tasks are increasing in retry count but I have no logs showing they ever executed. My backends are shutting down pre-maturely (that process these tasks) and again, no errors or anything in the logs explaining why. Not sure if this is a backend or taskqueue issue. Here is the latency graph for task queues: https://lh6.googleusercontent.com/-WUcDcWegm70/UVsHDGm6YcI/ABQ/C09Ysu8CarE/s1600/chart.png -- You received this message because you are subscribed to the Google Groups Google App Engine group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
[google-appengine] Strange issues on production
We have a process that splits a job between backend instances and synchronizes via memcache (not the best method, I know, but it has been working for us). Earlier in the day, we started seeing weird behavior. Its one or more of the following: 1. Backends are acting funky 2. Memcache latency/issues 3. URL Fetch latency/issues 4. Google Cloud Storage latency/issues 5. Task Queues are scheduling weirdly From what I can see, tasks are being executed fine. The expected completion time of the tasks, however, are taking a lot longer than anticipated (5-10x longer). I also see weird behavior of the countdown mechanism that synchronizes parts of the job (based on memcache). Sporadic updates and random resets down to '0' and back up to a more believable number. I wonder if atomic incr/decr calls to memcache are not finding previous values properly? If I had to guess, I think the issue is with either 1, 2, or 4 listed above. Other parts of my app hit the same URL endpoints this process hits, and they are running fine. As aforementioned, memcache values seem to be acting the weirdest, so I think the issue might reside there. Has anyone else noticed weird behavior with memcache in recent hours? Thank you, Prateek -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
[google-appengine] Latency Issues across Production APIs
The status dashboard keeps switching between Elevated and Normal and these graphs show an obvious spike in latency: Memcache Set: https://lh4.googleusercontent.com/-1MEvD3r645s/UQLsMBFp-CI/AAw/G0mcAtUj-xI/s1600/memcache_put.png HR Datastore Queries: https://lh6.googleusercontent.com/-qvPkxzW-ZvM/UQLsS5qY1MI/AA4/Xboj3RcZ9rE/s1600/Queries.png My application is experiencing issues that's effecting my business. Is this being investigated? (Status dashboard does not indicate so) My app that's effected uses backends, task queues, memcache, HRD, google storage, and url fetch (I think task queues/memcache are what's really effected) Thank you, Prateek -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
[google-appengine] Re: Latency Issues across Production APIs
Sorry, forgot to mention: Memcache and Task Queues seem to be the most effected. Tasks that usually take 10-20 seconds on my backends are taking almost 2 minutes. Some tasks seem to not even fully complete. Also, I use memcache as a synchronizing mechanism and use incr/decr to keep track of how many tasks are left (uses channel API to display # to the user), and it sporadically jumps down to 0 when it really shouldn't be. My backend jobs (the full generation of the report) take anywhere between 2-80 minutes, with 95% taking below 4-5 minutes. We are getting about a 20% success rate on the reports and that's taking 15 minutes or more. Any help would be greatly appreciated! -Prateek On Friday, January 25, 2013 3:36:26 PM UTC-5, Prateek Malhotra wrote: The status dashboard keeps switching between Elevated and Normal and these graphs show an obvious spike in latency: Memcache Set: https://lh4.googleusercontent.com/-1MEvD3r645s/UQLsMBFp-CI/AAw/G0mcAtUj-xI/s1600/memcache_put.png HR Datastore Queries: https://lh6.googleusercontent.com/-qvPkxzW-ZvM/UQLsS5qY1MI/AA4/Xboj3RcZ9rE/s1600/Queries.png My application is experiencing issues that's effecting my business. Is this being investigated? (Status dashboard does not indicate so) My app that's effected uses backends, task queues, memcache, HRD, google storage, and url fetch (I think task queues/memcache are what's really effected) Thank you, Prateek -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
