Re: [google-appengine] Re: Programmatic access the logs
On Tue, Feb 16, 2010 at 2:14 PM, peterk peter.ke...@gmail.com wrote: There's some talk about it in issue 76, 'log admin enhancements', but maybe it should have its own dedicated one... Could I ask a couple of other related questions here? 1) I read that the raw request logs are kept for 90 days. Is this invariant with the number of requests? Even if I have a huge number of requests, the past 90 days worth will always be available? Can you point me to where you see this? I'll follow up with the team. 2) Is this access guaranteed going forward? I've a webapp in mind where log access would be a dependency. It would be an app breaker if one day an update removed this access. We wouldn't yank it away without a warning. I don't think we're going to take this away, but if we did, we'd provide a migration path. 3) Is it guaranteed that the current level of log access and storage will remain free? If, for example, punitive charges were enforced on log downloads, it could upset the economics of an app I have in mind and again render it inviable. I know that's unlikely to happen but I'm a worrier :P No, we can't guarantee anything will stay fixed - it's similar to how we lowered the free quota a few months after release. Generally speaking, the trend is that we lower costs, not raise them. 4) Is there any limit on how many times I can download the logs in a given period? Or any other limit on download access? Not that I know of. I'm not sure if this counts against your download quota or not - that may be an oversight on our part if it is. 5) Is there any way - or any chance in the future - of getting a little more info in the raw request logs? Like for example, the length in ms it took to fulfill the request and cpu time used too? The logs in the admin console are embellished with this info, but the raw logs when downloaded don't have that. Possibly. It depends on how we calculate these numbers. Thanks for any answers to these questions :) On Feb 16, 9:24 pm, Ikai L (Google) ika...@google.com wrote: I'm pretty sure this specific issue isn't in the issue tracker yet: http://code.google.com/p/googleappengine/issues/list?can=2q=programm... On Mon, Feb 15, 2010 at 3:44 PM, peterk peter.ke...@gmail.com wrote: Came to ask about this and found your post. This would be exceptionally useful. A common use case at the moment seems to be this - a request comes in with some data, dump the data to memcache, then later on a task might come along, aggregate data from memcache, and persist it to datastore in batches, or aggregated results in batches. Saving the datastore being accessed for every request. Problem with this, though, is that memcache is not perfectly reliable. You might lose data. However...if you had access to the request logs, along with data sent with a request, you could do a number of things. You could sanity check your data periodically to check the above approach is 'OK'. Or, more directly, you could skip the memcaching altogether. You could just set up a task to process your logs every so often. The data is all there (at least if you pass your data using GET parameters), and the logs are, I presume, reliable. So your requests would just immediately return, and then later you periodically process the logs and do whatever aggregating and persisting you need to do. For this it would be handy to be able to programmatically 'regex' the logs you want to access also... This saves us processing time of course. And thus could mean less money for Google. However, I believe Google has to date encouraged and provided tools for efficient use of GAE. It seems really wasteful to put data in memcache for later processing if it's already being recorded more reliably somewhere else by default...as appears to be the case with the logs. Cheers, On Feb 11, 7:41 am, $ÂÑЀ€P sandeep.kod...@gmail.com wrote: Is Programmatic access thelogsavailable by now. last post states it will be available in future. its long time after that was posted i need to access the user ips,requesttime and url accessed for all requests, to aggregate user requests. my question is, do i have to maintain my own records in datastore/ memcache. or can i have access the data that is already present inlogs. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com . To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.comgoogle-appengine%2bunsubscr...@googlegroups.comgoogle-appengine%2Bunsubscrib e...@googlegroups.com . For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Ikai Lan Developer Programs Engineer, Google App Enginehttp://
[google-appengine] Re: Programmatic access the logs
Thanks for your replies Ikai. Mine are below: Can you point me to where you see this? I'll follow up with the team. I actually initially read it in Dan Sanderson's 'Programming Google App Engine' book. But as it turns out it's also mentioned here in the billing FAQ: http://code.google.com/appengine/kb/billing.html#dos We wouldn't yank it away without a warning. I don't think we're going to take this away, but if we did, we'd provide a migration path. OK, that's somewhat encouraging :) I think I'm over worrying on this myself - I see no reason why Google would take away log access since they'll always have the logs themselves. Not that I know of. I'm not sure if this counts against your download quota or not - that may be an oversight on our part if it is. I wouldn't entirely mind if it did count toward outgoing bandwidth, for example, but as long as I can pull them down as often as I need them I'm a happy chappy. For some cases that might be every few minutes. Possibly. It depends on how we calculate these numbers. Cool cool. And thanks for your replies again :) If you get anything back about how long logs are kept for, and if this is invariant with the amount of requests your app gets, it'd be great to hear about it. Cheers, -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: Programmatic access the logs
I'm pretty sure this specific issue isn't in the issue tracker yet: http://code.google.com/p/googleappengine/issues/list?can=2q=programmatic+log+access On Mon, Feb 15, 2010 at 3:44 PM, peterk peter.ke...@gmail.com wrote: Came to ask about this and found your post. This would be exceptionally useful. A common use case at the moment seems to be this - a request comes in with some data, dump the data to memcache, then later on a task might come along, aggregate data from memcache, and persist it to datastore in batches, or aggregated results in batches. Saving the datastore being accessed for every request. Problem with this, though, is that memcache is not perfectly reliable. You might lose data. However...if you had access to the request logs, along with data sent with a request, you could do a number of things. You could sanity check your data periodically to check the above approach is 'OK'. Or, more directly, you could skip the memcaching altogether. You could just set up a task to process your logs every so often. The data is all there (at least if you pass your data using GET parameters), and the logs are, I presume, reliable. So your requests would just immediately return, and then later you periodically process the logs and do whatever aggregating and persisting you need to do. For this it would be handy to be able to programmatically 'regex' the logs you want to access also... This saves us processing time of course. And thus could mean less money for Google. However, I believe Google has to date encouraged and provided tools for efficient use of GAE. It seems really wasteful to put data in memcache for later processing if it's already being recorded more reliably somewhere else by default...as appears to be the case with the logs. Cheers, On Feb 11, 7:41 am, $ÂÑЀ€P sandeep.kod...@gmail.com wrote: Is Programmatic access thelogsavailable by now. last post states it will be available in future. its long time after that was posted i need to access the user ips,requesttime and url accessed for all requests, to aggregate user requests. my question is, do i have to maintain my own records in datastore/ memcache. or can i have access the data that is already present inlogs. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.comgoogle-appengine%2bunsubscr...@googlegroups.com . For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Ikai Lan Developer Programs Engineer, Google App Engine http://googleappengine.blogspot.com | http://twitter.com/app_engine -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: Programmatic access the logs
There's some talk about it in issue 76, 'log admin enhancements', but maybe it should have its own dedicated one... Could I ask a couple of other related questions here? 1) I read that the raw request logs are kept for 90 days. Is this invariant with the number of requests? Even if I have a huge number of requests, the past 90 days worth will always be available? 2) Is this access guaranteed going forward? I've a webapp in mind where log access would be a dependency. It would be an app breaker if one day an update removed this access. 3) Is it guaranteed that the current level of log access and storage will remain free? If, for example, punitive charges were enforced on log downloads, it could upset the economics of an app I have in mind and again render it inviable. I know that's unlikely to happen but I'm a worrier :P 4) Is there any limit on how many times I can download the logs in a given period? Or any other limit on download access? 5) Is there any way - or any chance in the future - of getting a little more info in the raw request logs? Like for example, the length in ms it took to fulfill the request and cpu time used too? The logs in the admin console are embellished with this info, but the raw logs when downloaded don't have that. Thanks for any answers to these questions :) On Feb 16, 9:24 pm, Ikai L (Google) ika...@google.com wrote: I'm pretty sure this specific issue isn't in the issue tracker yet: http://code.google.com/p/googleappengine/issues/list?can=2q=programm... On Mon, Feb 15, 2010 at 3:44 PM, peterk peter.ke...@gmail.com wrote: Came to ask about this and found your post. This would be exceptionally useful. A common use case at the moment seems to be this - a request comes in with some data, dump the data to memcache, then later on a task might come along, aggregate data from memcache, and persist it to datastore in batches, or aggregated results in batches. Saving the datastore being accessed for every request. Problem with this, though, is that memcache is not perfectly reliable. You might lose data. However...if you had access to the request logs, along with data sent with a request, you could do a number of things. You could sanity check your data periodically to check the above approach is 'OK'. Or, more directly, you could skip the memcaching altogether. You could just set up a task to process your logs every so often. The data is all there (at least if you pass your data using GET parameters), and the logs are, I presume, reliable. So your requests would just immediately return, and then later you periodically process the logs and do whatever aggregating and persisting you need to do. For this it would be handy to be able to programmatically 'regex' the logs you want to access also... This saves us processing time of course. And thus could mean less money for Google. However, I believe Google has to date encouraged and provided tools for efficient use of GAE. It seems really wasteful to put data in memcache for later processing if it's already being recorded more reliably somewhere else by default...as appears to be the case with the logs. Cheers, On Feb 11, 7:41 am, $ÂÑЀ€P sandeep.kod...@gmail.com wrote: Is Programmatic access thelogsavailable by now. last post states it will be available in future. its long time after that was posted i need to access the user ips,requesttime and url accessed for all requests, to aggregate user requests. my question is, do i have to maintain my own records in datastore/ memcache. or can i have access the data that is already present inlogs. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.comgoogle-appengine%2Bunsubscrib e...@googlegroups.com . For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Ikai Lan Developer Programs Engineer, Google App Enginehttp://googleappengine.blogspot.com|http://twitter.com/app_engine -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: Programmatic access the logs
Came to ask about this and found your post. This would be exceptionally useful. A common use case at the moment seems to be this - a request comes in with some data, dump the data to memcache, then later on a task might come along, aggregate data from memcache, and persist it to datastore in batches, or aggregated results in batches. Saving the datastore being accessed for every request. Problem with this, though, is that memcache is not perfectly reliable. You might lose data. However...if you had access to the request logs, along with data sent with a request, you could do a number of things. You could sanity check your data periodically to check the above approach is 'OK'. Or, more directly, you could skip the memcaching altogether. You could just set up a task to process your logs every so often. The data is all there (at least if you pass your data using GET parameters), and the logs are, I presume, reliable. So your requests would just immediately return, and then later you periodically process the logs and do whatever aggregating and persisting you need to do. For this it would be handy to be able to programmatically 'regex' the logs you want to access also... This saves us processing time of course. And thus could mean less money for Google. However, I believe Google has to date encouraged and provided tools for efficient use of GAE. It seems really wasteful to put data in memcache for later processing if it's already being recorded more reliably somewhere else by default...as appears to be the case with the logs. Cheers, On Feb 11, 7:41 am, $ÂÑЀ€P sandeep.kod...@gmail.com wrote: Is Programmatic access thelogsavailable by now. last post states it will be available in future. its long time after that was posted i need to access the user ips,requesttime and url accessed for all requests, to aggregate user requests. my question is, do i have to maintain my own records in datastore/ memcache. or can i have access the data that is already present inlogs. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
