Thanks for your help Don. Just one more question (hopefully :)). I think just the request logs will be good enough for my purposes, and I won't actually have to use the diagnostic logs. How much past data is stored for the request logs?
On Mar 11, 8:16 am, Don Schwarz <schwa...@google.com> wrote: > On Wed, Mar 10, 2010 at 8:05 PM, Spines <kwste...@gmail.com> wrote: > > Sorry, I don't think I really understood the task queue approach. To > > ensure no loss of data, would the task have to update in the datastore > > every time? And the benefit over just doing it directly in the > > servlet handler would be the faster response time to the user? > > Basically, yes. > > > I think the memcache solution may be my best bet. I can tolerate some > > chance of loss of data. I'm wondering how likely loss of data would > > be? How often does the data get booted out? If I persist from memcache > > to datastore every 10 seconds would data loss be super rare? > > You would have to experiment with this as we don't make any guarantees, but > I believe that memcache data generally survives much longer than 10 seconds > of inactivity, yes. > > On Mar 10, 4:11 pm, Spines <kwste...@gmail.com> wrote: > > > > > > Thanks Don, > > > I thought about the task queue, but that caps at being able to execute > > > like 5 tasks per second right? > > > > So, as long as the log data doesn't get full before I download it then > > > it would be fine? > > > > On Mar 10, 3:42 pm, Don Schwarz <schwa...@google.com> wrote: > > > > > Yeah, those are diagnostic logs. They effectively go into a ring > > buffer per > > > > logging level, so the maximum data stored at any given time is capped. > > The > > > > more you log, the more frequently you would have to download the logs > > to > > > > avoid missing any. You would also be competing with log space with any > > > > other log messages generated by your application. > > > > > What I would suggest instead is either to increment counters in > > memcache, > > > > and flush them to the datastore periodically if you need durability > > (I'm > > > > assuming you can tolerate some chance of data loss here). If you > > cannot > > > > tolerate any loss of data, then I would suggest enqueueing tasks to a > > task > > > > queue for each request that maintains a summary in memcache and/or the > > > > datastore. > > > > > On Wed, Mar 10, 2010 at 4:59 PM, Spines <kwste...@gmail.com> wrote: > > > > > I'm talking about the logs that get written when I call > > > > > Logger.info("something"). > > > > > > Basically this is what I'm thinking: I have certain data that needs > > to > > > > > get written very often, but hardly ever needs to be read (stuff like > > > > > what users view what pages of my site). The datastore is optimized > > > > > for read efficiency. So, I want to output this data to the logs. I > > > > > will have an offsite computer download these logs, do calculations on > > > > > them, and upload the result of the calculations to the datastore. > > > > > > On Mar 10, 2:03 pm, Don Schwarz <schwa...@google.com> wrote: > > > > > > Are you talking about request logs or diagnostic logs? Although we > > > > > conflate > > > > > > them a bit in both the Admin Console viewer and the appcfg command, > > but > > > > > they > > > > > > are stored and tracked separately. > > > > > > > On Wed, Mar 10, 2010 at 4:01 PM, Spines <kwste...@gmail.com> > > wrote: > > > > > > > Hmm, that is my biggest concern, log reliability. Can someone > > from > > > > > > > Google confirm whether or not I can rely on the logs having all > > of the > > > > > > > log data? Or might certain entries just disappear? > > > > > > > > On Mar 10, 1:24 pm, thierry Le conniat <thlec...@euriware.fr> > > wrote: > > > > > > > > Hello, > > > > > > > > I think google log are stored in file. > > > > > > > > My experience about log reliability is that when the app is > > very > > > > > > > > strong working, not all the log are stored. > > > > > > > > It's confusing, but i can't not explain it. > > > > > > > > > Bye > > > > > > > > > On 10 mar, 22:04, Spines <kwste...@gmail.com> wrote: > > > > > > > > > > Where does Google store the logs when you do a Logging > > statement? > > > > > > > > > Logging statements seem to be pretty fast, so it doesn't seem > > like > > > > > > > > > they are stored in the datastore. > > > > > > > > > > How reliable are the logs? If I do a logging statement and it > > > > > > > > > succeeds, is it pretty much guaranteed that it will show up > > in the > > > > > > > > > logs? > > > > > > > > > > How much past history of logs is stored? > > > > > > > > > > The reason I'm interested in this is because I'm making a > > question > > > > > and > > > > > > > > > answer website, and I want to keep track of views by each > > unique > > > > > > > > > logged in user to each question, and display the view count > > on the > > > > > > > > > question page. So if 10 different users visit the question > > page 100 > > > > > > > > > times, it still only counts as 10 unique views. > > > > > > > > > > I have an offsite computer that does background processing > > for my > > > > > app. > > > > > > > > > I'm planning to have this offsite computer download the logs > > about > > > > > > > > > every 30 minutes, and calculate what the view count should be > > for > > > > > each > > > > > > > > > question based off of the logs. By doing this, I don't have > > to > > > > > create > > > > > > > > > a datastore entity for each different question each user > > views. > > > > > > > > > > What do you guys think? Does anyone see any problems with > > this? > > > > > > > > -- > > > > > > > You received this message because you are subscribed to the > > Google > > > > > Groups > > > > > > > "Google App Engine for Java" group. > > > > > > > To post to this group, send email to > > > > > > > google-appengine-j...@googlegroups.com. > > > > > > > To unsubscribe from this group, send email to > > > > > > > google-appengine-java+unsubscr...@googlegroups.com<google-appengine-java%2B > > > > > > > unsubscr...@googlegroups.com><google-appengine-java%2B > > unsubscr...@googlegroups.com><google-appengine-java%2B > > > > > unsubscr...@googlegroups.com> > > > > > > > . > > > > > > > For more options, visit this group at > > > > > > >http://groups.google.com/group/google-appengine-java?hl=en. > > > > > > -- > > > > > You received this message because you are subscribed to the Google > > Groups > > > > > "Google App Engine for Java" group. > > > > > To post to this group, send email to > > > > > google-appengine-j...@googlegroups.com. > > > > > To unsubscribe from this group, send email to > > > > > google-appengine-java+unsubscr...@googlegroups.com<google-appengine-java%2B > > > > > unsubscr...@googlegroups.com><google-appengine-java%2B > > unsubscr...@googlegroups.com> > > > > > . > > > > > For more options, visit this group at > > > > >http://groups.google.com/group/google-appengine-java?hl=en. > > > -- > > You received this message because you are subscribed to the Google Groups > > "Google App Engine for Java" group. > > To post to this group, send email to > > google-appengine-j...@googlegroups.com. > > To unsubscribe from this group, send email to > > google-appengine-java+unsubscr...@googlegroups.com<google-appengine-java%2B > > unsubscr...@googlegroups.com> > > . > > For more options, visit this group at > >http://groups.google.com/group/google-appengine-java?hl=en. -- You received this message because you are subscribed to the Google Groups "Google App Engine for Java" group. To post to this group, send email to google-appengine-j...@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.