[google-appengine] Re: Many cold start problems recently.
Be careful how you analyze the situation with your instances. When I was playing with resident instances, I assumed that they represented a "floor" you wouldn't drop below in times of low traffic, but that's not quite what they do. Rather, they represent a pool of *deliberately idle* resources that can absorb spikes. When working according to Google's intent, resident instances will absorb incoming traffic, but as soon as they become busy (with even one request) dynamic instances will spin up to offset the loss of idle instances. Since this happens very close together, it's easy to assume that the dynamic instance was spun up to handle a given request and that the resident instance got nothing. But in my particular case, what happened was that the resident instance got the first request, concurrently with the warmup for the dynamic request, and then all subsequent requests went to the dynamic instance. Once I understood what resident instances do, I realized they do not - by design - solve the problem I wanted to solve, and I stopped using them altogether. I never actually produced evidence to demonstrate that traffic was going to a cold instance when a resident instance was available. I believe others when they say they have such evidence, but I also strongly suspect that Google lumps all such claims into the same "not understanding what resident instances are supposed to do" bucket and ignores them. - Kris On Wednesday, March 6, 2013 5:45:30 PM UTC-8, Cesium wrote: > > Tapir, > > This is an ongoing issue we have suffered with. > > Get used to it. There will be no (useful) response from G. > > David > > On Tuesday, March 5, 2013 6:52:28 PM UTC-7, Tapir wrote: > > There is a resident instance obviously there. But the GAE scheduler > often ignores it totally and always creates a new instance then the let the > new instacne to handle the new request. > > What is the functionality of the resident instance? If the scheduler > always creates a new instance, the why put a resident instance there? > > > > It really make the user experience very bad. > > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
[google-appengine] How much space in memcache
Hello, I noticed in Google App engine the quota is 1mb value for menarche, but does not specify a TOTAL limit of how much I can use. How much memcache space do I have? Thanks -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
[google-appengine] Re: How much space in memcache
Nobody knows except maybe Google, and I doubt even they know. The reason is that memcache exists in the memory of the particular machine your application is on (as opposed to the datastore/cloud sql, which can exist on another machine or span multiple machines). This means that memcache is blazingly fast, but is subject to ever-changing space constraints - i.e. the machine your app is running on might suddenly decide that it needs to allocate RAM away from memcache to running another application, or another application might be putting data into memcache, etc. Memcache is fast, but unreliable. Let me emphasize that last part - IT IS UNRELIABLE. You could put an object into memcache and have it live for days; or it might be unavailable even 1 second later. The smartest thing to do is to simply put everything into memcache, and when you need the object, pull it out. If the pull fails, then query the datastore/cloud sql. But none of the above directly answers your question, so I'll say from personal experience that I find the memcache maintains roughly 500 MB - 1GB of data on good days. -Vinny On Thursday, March 7, 2013 12:44:42 PM UTC-6, victo...@gmail.com wrote: > > Hello, I noticed in Google App engine the quota is 1mb value for menarche, > but does not specify a TOTAL limit of how much I can use. How much memcache > space do I have? > > Thanks > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
[google-appengine] Re: How much space in memcache
On Thursday, March 7, 2013 1:13:35 PM UTC-6, Vinny P wrote: > The smartest thing to do is to simply put everything into memcache, and > when you need the object, pull it out. If the pull fails, then query the > datastore/cloud sql. > Or if you like, you could use Objectify which manages the Memcache as a Datastore cache automatically... -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
[google-appengine] Re: Alternate to Conversion API
A lot of this can be done in Python and since you need this in functionality in Java you can set up a Python version or backend wrapped in a RESTful API that uses xhtml2pdf and then consume that API from your Java code. For HTML/image to PDF I use the Python library http://www.xhtml2pdf.com/ which uses Reportlab, pyPdf, and html5lib running on GAE. I have been using it to generate very nice article PDFs with embedded images and once I figured out how to get the page size correct I have found this to be a very good library. For PDF to image I have been using ImageMagick on an EC2 and for extracting text I have used Apache PDFBox (Java) and pdfminer (Python) on GAE. I have to convert CMYK PDFs into RGB JPGs with watermarks and ImageMagick makes this nice and easy but of course cannot be run on GAE so I wrapped it in an API and consume from GAE. I mostly use PDFBox to extract text for my search index and have no experience trying to get a nicely formatted text version from a PDF but I know pdfminer will give you a formatted HTML version of a PDF. - Bryce On Tuesday, August 21, 2012 4:00:57 AM UTC-7, aswath wrote: > > Hello, > We were deeply involved in utilizing the conversion api for the HTML to > PDF conversion. Suddenly, I got the email from Google about the plan for > decommissioning from Nov 2012. > > Does anyone has suggestions for doing the HTML to PDF conversion that is > compatible with Google Appengine for Java. > > > Regards > -Aswath > www.AccountingGuru.in > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
[google-appengine] Re: How much space in memcache
On Thursday, March 7, 2013 2:38:39 PM UTC-6, Chad Vincent wrote: > > Or if you like, you could use Objectify which manages the Memcache as a > Datastore cache automatically... There are many data storage abstraction layers and frameworks available for Java and Python on GAE, of which Objectify is a good example. There are more listed on this page: https://developers.google.com/appengine/docs/java/datastore/overview -Vinny P -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
[google-appengine] Re: Can anyone confirm or deny that round about 1st March, GoogleAppEngine trippled the length of Blobstore filenames from around 300 chars up to around 900 chars?
Can you also please fix issue 5236 since files.blobstore.create() still fails 1-2% of the time due to timeout (it hangs in google coded as explained in issue)? https://code.google.com/p/googleappengine/issues/detail?id=5236 On Tuesday, March 5, 2013 9:38:10 PM UTC-5, Ryan Huebsch wrote: > > We've filed an issue > https://code.google.com/p/googleappengine/issues/detail?id=8932 to track > the issue and have begun investigating. > > On Tuesday, March 5, 2013 6:39:44 AM UTC-8, Andrew Bindon wrote: >> >> Can anyone confirm or deny that round about 1st March, GoogleAppEngine >> trippled the length of Blobstore filenames from around 300 chars up to >> around 900 chars? >> > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
Re: [google-appengine] gae mapreduce problem
You should probably create a pipeline that first runs your mapreduce, then runs a pipeline stage that only prepends a header to the header-less csv created from the mapreduce. On Tue, Mar 5, 2013 at 3:45 PM, Jonas Heyden wrote: > I am fiddling around with GAE mapreduce and have one question: > > Is it possible to change a variable only for a certain job in mapreduce? > > The reason I am asking is: > > The input csv and output csv of my mapreduce job are supposed to have the > same header row - however, the header row is somewhere in the output csv, > but never at the top. To get the right header row, I inserted a counter into > my reduce function that checks the current iteration of the reduce job and > if it is 0, it will pass the hard-coded header-row to the pipeline. The > counter gets reset when the output csv gets stored in the blobstore. > > The problem: More often than not the counter resets itself randomly, > probably because I had to define it as global variable "reduce_counter = 0" > outside of the function. > > Is there any method to chain a variable/parameter to a job or is there any > better way to get the header_row? > > I don't think that I can work with the DictReader or csv module as the > output is stored in the blobstore and blobstore objects cannot be altered as > far as I know. > > You can find my code on www.github.com/jvdheyden/ste in the main.py > document. > > Thanks! > > > > -- > Jonas von der Heyden > +49 163 2464010 > http://de.linkedin.com/in/jvheyden > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to google-appengine+unsubscr...@googlegroups.com. > To post to this group, send email to google-appengine@googlegroups.com. > Visit this group at http://groups.google.com/group/google-appengine?hl=en. > For more options, visit https://groups.google.com/groups/opt_out. > > -- Jim -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
[google-appengine] Re: How much space in memcache
Thanks for all the replies. Yes, the reason I ask is because all the data I aniticapte will be about 500mb big and was wondering to store it all in memcache and non in the datastore to avoid front end instance hours, but it seems as this is not a good approach since it is unreliable. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
[google-appengine] Re: Many cold start problems recently.
I believe most who suffer from this problem understand the intended operation of the scheduler as stated in the docs. Some just imply that the design is sub-optimal. A much louder complaint is the lack of consistent behavior, whether it is understood or not. David On Thursday, March 7, 2013 1:13:46 AM UTC-7, Kristopher Giesing wrote: > Be careful how you analyze the situation with your instances. When I was > playing with resident instances, I assumed that they represented a "floor" > you wouldn't drop below in times of low traffic, but that's not quite what > they do. Rather, they represent a pool of *deliberately idle* resources that > can absorb spikes. > > > When working according to Google's intent, resident instances will absorb > incoming traffic, but as soon as they become busy (with even one request) > dynamic instances will spin up to offset the loss of idle instances. Since > this happens very close together, it's easy to assume that the dynamic > instance was spun up to handle a given request and that the resident instance > got nothing. But in my particular case, what happened was that the resident > instance got the first request, concurrently with the warmup for the dynamic > request, and then all subsequent requests went to the dynamic instance. > > > Once I understood what resident instances do, I realized they do not - by > design - solve the problem I wanted to solve, and I stopped using them > altogether. I never actually produced evidence to demonstrate that traffic > was going to a cold instance when a resident instance was available. I > believe others when they say they have such evidence, but I also strongly > suspect that Google lumps all such claims into the same "not understanding > what resident instances are supposed to do" bucket and ignores them. > > > - Kris > > On Wednesday, March 6, 2013 5:45:30 PM UTC-8, Cesium wrote:Tapir, > > > > This is an ongoing issue we have suffered with. > > > > Get used to it. There will be no (useful) response from G. > > > > David > > > > On Tuesday, March 5, 2013 6:52:28 PM UTC-7, Tapir wrote: > > > There is a resident instance obviously there. But the GAE scheduler often > > ignores it totally and always creates a new instance then the let the new > > instacne to handle the new request. > > > What is the functionality of the resident instance? If the scheduler always > > creates a new instance, the why put a resident instance there? > > > > > > It really make the user experience very bad. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
[google-appengine] Re: Many cold start problems recently.
This has been an issue since last November. The only consistency I see on GAE/J is 15s+ user facing requests. This is a absolute killer on interactive webpages. It is trivial to identify which requests are "using facing" cold instance starts in the logs. Google however, was un-inpressed. So I ported our mission critical stuff to AWS. The rest of our GAE/Java stuff is mostly fluff, and I am converting that to Go. I hope within a week or so, I can permanently retire all our GAE/Java code. The result of all this is that I feel like a burn victim being sold Icy Hot under the generic brand "Soothing Cream". On the upside, I can now cancel my subscription to "Unicorns Weekly". On Thursday, March 7, 2013 6:38:50 PM UTC-8, Cesium wrote: > > I believe most who suffer from this problem understand the intended > operation of the scheduler as stated in the docs. > > Some just imply that the design is sub-optimal. > > A much louder complaint is the lack of consistent behavior, whether it is > understood or not. > > David > > > > > On Thursday, March 7, 2013 1:13:46 AM UTC-7, Kristopher Giesing wrote: > > Be careful how you analyze the situation with your instances. When I > was playing with resident instances, I assumed that they represented a > "floor" you wouldn't drop below in times of low traffic, but that's not > quite what they do. Rather, they represent a pool of *deliberately idle* > resources that can absorb spikes. > > > > > > When working according to Google's intent, resident instances will > absorb incoming traffic, but as soon as they become busy (with even one > request) dynamic instances will spin up to offset the loss of idle > instances. Since this happens very close together, it's easy to assume > that the dynamic instance was spun up to handle a given request and that > the resident instance got nothing. But in my particular case, what > happened was that the resident instance got the first request, concurrently > with the warmup for the dynamic request, and then all subsequent requests > went to the dynamic instance. > > > > > > Once I understood what resident instances do, I realized they do not - > by design - solve the problem I wanted to solve, and I stopped using them > altogether. I never actually produced evidence to demonstrate that traffic > was going to a cold instance when a resident instance was available. I > believe others when they say they have such evidence, but I also strongly > suspect that Google lumps all such claims into the same "not understanding > what resident instances are supposed to do" bucket and ignores them. > > > > > > - Kris > > > > On Wednesday, March 6, 2013 5:45:30 PM UTC-8, Cesium wrote:Tapir, > > > > > > > > This is an ongoing issue we have suffered with. > > > > > > > > Get used to it. There will be no (useful) response from G. > > > > > > > > David > > > > > > > > On Tuesday, March 5, 2013 6:52:28 PM UTC-7, Tapir wrote: > > > > > There is a resident instance obviously there. But the GAE scheduler > often ignores it totally and always creates a new instance then the let the > new instacne to handle the new request. > > > > > What is the functionality of the resident instance? If the scheduler > always creates a new instance, the why put a resident instance there? > > > > > > > > > > It really make the user experience very bad. > > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
[google-appengine] Datastore writes are ununsually high (15-20 time higher)
Hello, I have noticed that today, my datastore writes were unusually very high. It is about 15-20 time higher. How can I resolve this and figure out what happened. Thanks, -Aswath -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
