Hi Scott, My first thought: summarize the data and export the summaries. For example, summarize each add by the hour then just export the hourly summaries to the other app.
Robert On Fri, Jul 23, 2010 at 11:03 AM, Scott Newman <snewma...@gmail.com> wrote: > Hello, > > I have an ad-serving application launching soon that we expect to > receive about 2 million requests/day. Each page serves a random ad, > and I log which ad was served in a Log model. > > I need to get these log records back to my billing system each day for > archival, statistical and reconciliation purposes. The problem is that > I haven't come up with a good way to get this data out of AppEngine. > Even if I tried to get the data out hourly, I'd still end up with > about 85k records - with a 1000 record query limit, I'd have to > execute an offset 85 times every hour. (unless I'm missing something) > > I can't see a way that the data exporter can be configured to give me > a date range; it appears to be an all-or-nothing approach. I *could* > export all the data nightly, filter out only the records in the date > range I care about, then go back to AppEngine and purge the records in > that date range. That would mean at most I'd be exporting slightly > more than one day's records. > > Can anyone think of a better way to accomplish what I'm trying to do? > I'd appreciate any thoughts. > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to google-appeng...@googlegroups.com. > To unsubscribe from this group, send email to > google-appengine+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
