During the course of an appengine request, we record a bunch of stats in the logs, such as number/type of datastore operations, profiling info, URLfetch stats, etc. Before the logs can go into BigQuery, we want to extract that info from the logs and place them in particular columns in our BigQuery tables.
Currently our GAE instances do this transformation but ideally this transformation would be done in dataflow which would be way more efficient, less code to write, cheaper, more stable and managed! On Wed, Apr 29, 2015 at 12:08 PM Jason Collins <jason.a.coll...@gmail.com> wrote: > we need to process logs before sending to bigquery > > > Aleem, can you tell me more about what processing you do on your logs > before sending them to BigQuery? > > -- > You received this message because you are subscribed to a topic in the > Google Groups "Google App Engine" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/google-appengine/GK_5qVwBIuQ/unsubscribe > . > To unsubscribe from this group and all its topics, send an email to > google-appengine+unsubscr...@googlegroups.com. > To post to this group, send email to google-appengine@googlegroups.com. > Visit this group at http://groups.google.com/group/google-appengine. > To view this discussion on the web visit > https://groups.google.com/d/msgid/google-appengine/91564e30-0a50-460d-8e7f-680871272c34%40googlegroups.com > <https://groups.google.com/d/msgid/google-appengine/91564e30-0a50-460d-8e7f-680871272c34%40googlegroups.com?utm_medium=email&utm_source=footer> > . > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine. To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/CADBYpykRkhBFCzQhT3Wj5ap2gpp1DyeiBxObygL1zeWjtdwJ-w%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
