Thanks for your help Don. Just one more question (hopefully :)).
I think just the request logs will be good enough for my purposes, and
I won't actually have to use the diagnostic logs. How much past data
is stored for the request logs?
On Mar 11, 8:16 am, Don Schwarz <schwa...@google.com> wrote:
> On Wed, Mar 10, 2010 at 8:05 PM, Spines <kwste...@gmail.com> wrote:
> > Sorry, I don't think I really understood the task queue approach. To
> > ensure no loss of data, would the task have to update in the datastore
> > every time? And the benefit over just doing it directly in the
> > servlet handler would be the faster response time to the user?
>
> Basically, yes.
>
> > I think the memcache solution may be my best bet. I can tolerate some
> > chance of loss of data. I'm wondering how likely loss of data would
> > be? How often does the data get booted out? If I persist from memcache
> > to datastore every 10 seconds would data loss be super rare?
>
> You would have to experiment with this as we don't make any guarantees, but
> I believe that memcache data generally survives much longer than 10 seconds
> of inactivity, yes.
>
> On Mar 10, 4:11 pm, Spines <kwste...@gmail.com> wrote:
>
>
>
> > > Thanks Don,
> > > I thought about the task queue, but that caps at being able to execute
> > > like 5 tasks per second right?
>
> > > So, as long as the log data doesn't get full before I download it then
> > > it would be fine?
>
> > > On Mar 10, 3:42 pm, Don Schwarz <schwa...@google.com> wrote:
>
> > > > Yeah, those are diagnostic logs. They effectively go into a ring
> > buffer per
> > > > logging level, so the maximum data stored at any given time is capped.
> > The
> > > > more you log, the more frequently you would have to download the logs
> > to
> > > > avoid missing any. You would also be competing with log space with any
> > > > other log messages generated by your application.
>
> > > > What I would suggest instead is either to increment counters in
> > memcache,
> > > > and flush them to the datastore periodically if you need durability
> > (I'm
> > > > assuming you can tolerate some chance of data loss here). If you
> > cannot
> > > > tolerate any loss of data, then I would suggest enqueueing tasks to a
> > task
> > > > queue for each request that maintains a summary in memcache and/or the
> > > > datastore.
>
> > > > On Wed, Mar 10, 2010 at 4:59 PM, Spines <kwste...@gmail.com> wrote:
> > > > > I'm talking about the logs that get written when I call
> > > > > Logger.info("something").
>
> > > > > Basically this is what I'm thinking: I have certain data that needs
> > to
> > > > > get written very often, but hardly ever needs to be read (stuff like
> > > > > what users view what pages of my site). The datastore is optimized
> > > > > for read efficiency. So, I want to output this data to the logs. I
> > > > > will have an offsite computer download these logs, do calculations on
> > > > > them, and upload the result of the calculations to the datastore.
>
> > > > > On Mar 10, 2:03 pm, Don Schwarz <schwa...@google.com> wrote:
> > > > > > Are you talking about request logs or diagnostic logs? Although we
> > > > > conflate
> > > > > > them a bit in both the Admin Console viewer and the appcfg command,
> > but
> > > > > they
> > > > > > are stored and tracked separately.
>
> > > > > > On Wed, Mar 10, 2010 at 4:01 PM, Spines <kwste...@gmail.com>
> > wrote:
> > > > > > > Hmm, that is my biggest concern, log reliability. Can someone
> > from
> > > > > > > Google confirm whether or not I can rely on the logs having all
> > of the
> > > > > > > log data? Or might certain entries just disappear?
>
> > > > > > > On Mar 10, 1:24 pm, thierry Le conniat <thlec...@euriware.fr>
> > wrote:
> > > > > > > > Hello,
> > > > > > > > I think google log are stored in file.
> > > > > > > > My experience about log reliability is that when the app is
> > very
> > > > > > > > strong working, not all the log are stored.
> > > > > > > > It's confusing, but i can't not explain it.
>
> > > > > > > > Bye
>
> > > > > > > > On 10 mar, 22:04, Spines <kwste...@gmail.com> wrote:
>
> > > > > > > > > Where does Google store the logs when you do a Logging
> > statement?
> > > > > > > > > Logging statements seem to be pretty fast, so it doesn't seem
> > like
> > > > > > > > > they are stored in the datastore.
>
> > > > > > > > > How reliable are the logs? If I do a logging statement and it
> > > > > > > > > succeeds, is it pretty much guaranteed that it will show up
> > in the
> > > > > > > > > logs?
>
> > > > > > > > > How much past history of logs is stored?
>
> > > > > > > > > The reason I'm interested in this is because I'm making a
> > question
> > > > > and
> > > > > > > > > answer website, and I want to keep track of views by each
> > unique
> > > > > > > > > logged in user to each question, and display the view count
> > on the
> > > > > > > > > question page. So if 10 different users visit the question
> > page 100
> > > > > > > > > times, it still only counts as 10 unique views.
>
> > > > > > > > > I have an offsite computer that does background processing
> > for my
> > > > > app.
> > > > > > > > > I'm planning to have this offsite computer download the logs
> > about
> > > > > > > > > every 30 minutes, and calculate what the view count should be
> > for
> > > > > each
> > > > > > > > > question based off of the logs. By doing this, I don't have
> > to
> > > > > create
> > > > > > > > > a datastore entity for each different question each user
> > views.
>
> > > > > > > > > What do you guys think? Does anyone see any problems with
> > this?
>
> > > > > > > --
> > > > > > > You received this message because you are subscribed to the
> > Google
> > > > > Groups
> > > > > > > "Google App Engine for Java" group.
> > > > > > > To post to this group, send email to
> > > > > > > google-appengine-j...@googlegroups.com.
> > > > > > > To unsubscribe from this group, send email to
> > > > > > > google-appengine-java+unsubscr...@googlegroups.com<google-appengine-java%2B
> > > > > > > unsubscr...@googlegroups.com><google-appengine-java%2B
> > unsubscr...@googlegroups.com><google-appengine-java%2B
> > > > > unsubscr...@googlegroups.com>
> > > > > > > .
> > > > > > > For more options, visit this group at
> > > > > > >http://groups.google.com/group/google-appengine-java?hl=en.
>
> > > > > --
> > > > > You received this message because you are subscribed to the Google
> > Groups
> > > > > "Google App Engine for Java" group.
> > > > > To post to this group, send email to
> > > > > google-appengine-j...@googlegroups.com.
> > > > > To unsubscribe from this group, send email to
> > > > > google-appengine-java+unsubscr...@googlegroups.com<google-appengine-java%2B
> > > > > unsubscr...@googlegroups.com><google-appengine-java%2B
> > unsubscr...@googlegroups.com>
> > > > > .
> > > > > For more options, visit this group at
> > > > >http://groups.google.com/group/google-appengine-java?hl=en.
>
> > --
> > You received this message because you are subscribed to the Google Groups
> > "Google App Engine for Java" group.
> > To post to this group, send email to
> > google-appengine-j...@googlegroups.com.
> > To unsubscribe from this group, send email to
> > google-appengine-java+unsubscr...@googlegroups.com<google-appengine-java%2B
> > unsubscr...@googlegroups.com>
> > .
> > For more options, visit this group at
> >http://groups.google.com/group/google-appengine-java?hl=en.
--
You received this message because you are subscribed to the Google Groups
"Google App Engine for Java" group.
To post to this group, send email to google-appengine-j...@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine-java+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine-java?hl=en.
