Hi Khaled, To answer your questions
1. Your sink will hold all logs from the day it was initialized. Which means 31 days from now, it will hold the 31st day of the log rather than deleting the log from the 1st day. This also means that any logs prior to the export sink will not be retained and will be removed after 30 days. 2. In regards to your second inquiry, using export sink is the only way to retain logs over 30 days. That said, as per the following document [1] these are the current formats provided by Stackdriver. I hope this helps. [1]: https://cloud.google.com/solutions/design-patterns-for-exporting-stackdriver-logging#logging_formats On Thursday, September 5, 2019 at 11:55:50 AM UTC-4, Khaled Wagdy wrote: > > Hi All, > > Is there a best practice for keeping logs for more than 30 days yet in an > easily readable format? > > Recently we had an issue and wanted to check the logs of the previous > month. Since the logs are only kept for 30 days in *Log Viewer *we couldn't > find it there. We have previously setup a sink to BigQuery, but although > the logs seems to be there, we couldn't find the logs we're looking for! > Not sure if this is lack of experience with BigQuery or is it that only the > request logs coming from actual users are the ones kept. We are looking for > a particular monthly CRON job logs, which we couldn't find in the backed up > logs on BigQuery for some reason. > > We thought we would setup another sink to Cloud storage for the future, > but it seems to be saved in JSON format, which includes too much > information that makes it hard to read in critical cases where we want to > see normal log flow as it appears in *Log Viewer. * > > We used to use instructions in the Download logs page > <https://cloud.google.com/appengine/docs/standard/python/tools/downloading-logs> > using > the command *appcfg.py request_logs *to download logs periodically, but > we stopped doing that a while ago. The problem with this method is that > instead of showing the date and time as they appear on the *Log Viewer * > page*, *they appear in what I assume to be an Epoch format which still > makes it hard to read, example: > *1:1565112559.564261** log line...* > *3:1565112559.564692 log line...* > *...* > furthermore this method seems to be on its way to be deprecated as > mentioned on the download logs page and the new *gcloud* tool does not > have a download option > > *Warning:* The following describes using the appcfg tool to download >> logs. This tool is now deprecated >> <https://cloud.google.com/appengine/docs/standard/python/sdk-gcloud-migration>. >> >> Currently the replacement, Cloud SDK <https://cloud.google.com/sdk/docs/>, >> does not support this download capability; however, you can view your logs >> using the gcloud app log commands >> <https://cloud.google.com/sdk/gcloud/reference/app/logs/>. > > > I am not sure why it is not easy to export log viewer as it is, it's > really very convenient as is, we just a way to download it straight > forwardly. > > Resources*:* > > 1. Export logs documentation > <https://cloud.google.com/logging/docs/export/?hl=en_US> > 2. Download logs documentation > > <https://cloud.google.com/appengine/docs/standard/python/tools/downloading-logs> > > *Questions:* > > 1. Knowing that we have a sink setup to BigQuery, is there a way to > fetch the logs of the previous month CORN job? Was it exported as part of > the sink? How can we filter through BigQuery to fetch those logs? > 2. Is there a way to export logs to keep for more than 30 days, > without changing the log viewer format? > > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/1765d8bb-7889-42ec-8314-cf7fcd024be3%40googlegroups.com.
